airbnb/aerosolve

This release introduces a comprehensive Gradle build system with Travis CI integration, replacing the previous build configuration. It significantly expands the machine learning capabilities by adding support for image feature extraction, XGBoost pipelines, and a wide variety of new model types including linear, tree-based, and kernel models. The update also brings a generic ML pipeline framework for training and scoring, alongside new demo workflows for image impressionism, income prediction, and text classification. Finally, the release includes extensive unit test coverage for the new features and utilities.

The project now includes a Gradle wrapper (gradlew, gradlew.bat) and a .travis.yml configuration to automate builds and publish artifacts to Bintray. The build script installs the Thrift compiler and dependencies, runs checks on non-tag builds, and triggers a Slack notification upon successful tag builds. Additionally, the .gitignore is updated to exclude Gradle and IDE files, and the license header is updated to 2015.

(repo-wide) · high confidence

The image\_impressionism demo is now available, providing a complete workflow for training a regression model on an input image to recreate it. The demo includes a README with prerequisites and step-by-step instructions for running the training, model training, and impression generation tasks using Spark and Gradle. The implementation features a configuration file (image\_impressionism.conf) that defines the data pipeline, including a multiscale grid quantize transform for pixel locations and a cross transform combining color channels. Users can execute jobs like MakeTraining, TrainModel, and MakeImpression via a shell script, with the model learning to predict pixel colors based on location and channel features.

_demo/image\_impressionism, demo/income\_prediction, demo/income\prediction/src/main/scala · high confidence

Added a new Spark-based data conversion job that transforms internal Thrift \Example\ objects into PhotonML Avro format. This includes a new \ConvertExamplesToPhotonMLAvroJob\ entry point, utility classes for schema generation and feature mapping, and a metadata extractor implementation, enabling users to export training data in the format required by Photon ML.

training/src/main/scala/com/airbnb/aerosolve/training/photon · high confidence

A new Twenty News demo is introduced, providing a complete end-to-end workflow for training and evaluating a text classification model on the 20 Newsgroups dataset. The change adds two Scala files: JobRunner, which serves as the entry point to execute specific pipeline stages (debugging, data preparation, training, and evaluation), and TwentyNewsPipeline, which implements the core logic for each stage. Users can now run the full lifecycle—preprocessing raw text into examples, training a linear ranker model, and evaluating its performance—using the provided configuration-driven commands.

_demo/twenty\news/src/main/scala · high confidence

A new Twenty News demo has been added, providing a complete workflow for training a multiclass model on the 20 Newsgroups dataset. The change includes a Python script (convert\_to\_aerosolve.py) to convert the raw dataset into a flat text format, a shell script (job\_runner.sh) to execute the Spark job, and a README.md with setup instructions and example output.

_demo/twenty\news · high confidence

A new configuration file (twenty\_news.conf) is added to the Twenty News demo, defining the pipeline for training and evaluating a full-rank linear model. The config specifies the training data source, model output paths, and a combined transform pipeline that includes tokenization, normalization, and string deletion. It also sets the loss function to 'softmax' and the solver to 'sparse\_boost' for the model training process.

_demo/twenty\news/src/main/resources · high confidence

Users can now run end-to-end training and scoring workflows using new demo configuration files for binary and multiclass classification, along with a Bash script to submit jobs to Spark. The package includes a conversion utility to transform Example data into PhotonML Avro format, providing ready-to-use templates for common machine learning pipelines.

training/src/main/resources · high confidence

Introduced a new image processing module in the core library that enables extraction of visual features from images. The update adds a base \ImageFeature\ class and an \ImageFeatureExtractor\ orchestrator that computes histograms for RGB, HOG (Histogram of Oriented Gradients), LBP (Local Binary Patterns), and HSV (Hue, Saturation, Value) color spaces. These features are aggregated into dense feature vectors for downstream machine learning tasks.

core/src/main/java/com/airbnb/aerosolve/core/images · high confidence

Added a new XGBoost pipeline that enables training, evaluation, and scoring of XGBoost models within a Spark and Hive environment. The update includes a Monte Carlo parameter search to optimize model hyperparameters, along with scripts and configuration templates to manage the full lifecycle from data preparation to model output. This provides a structured way to handle XGBoost workloads, including saving models and results to HDFS and Hive.

airlearner/airlearner-xgboost · high confidence

Added a new binary regression strategy model implementation, including configuration classes (BaseSearchConfig, EvalConfig, TrainingConfig, TrainingOptions), data models (BaseBinarySample, BinaryTrainingSample, BinaryScoringSample), parameter handling (BaseParam, StrategyParams), and the core training logic (BinaryRegressionTrainer, BaseBinaryRegressionTrainer). This enables training and evaluating binary regression models with configurable hyperparameters and evaluation metrics.

airlearner/airlearner-strategy · high confidence

A new generic pipeline framework is introduced to standardize the workflow for generating, training, evaluating, and scoring machine learning models. This includes a main orchestrator (JobRunner) that dispatches to specific pipeline stages such as NDTreePipeline for building feature maps, GenericPipeline for training and evaluation, and utilities for model debugging and result reporting. The change adds support for multiclass and regression tasks, configurable sampling and shuffling, and integration with Hive for data ingestion and model dumping.

training/src/main/scala/com/airbnb/aerosolve/training/pipeline · high confidence

Added new classes to the core features package to support structured feature handling and model scoring. This includes the \FeatureFamily\ hierarchy (\StringFamily\, \FloatFamily\) for organizing string and float features, along with \FeatureGen\, \FeatureMapping\, \FeatureVectorGen\, and \Features\ to manage feature generation and conversion. Additionally, new \ModelConfig\ and \ModelScorer\ classes were added to the scoring package to handle model loading and scoring operations.

core/src/main/java/com/airbnb/aerosolve/core/features · high confidence

Added new Thrift schema files to define data structures for machine learning models and spatial indexing. MLSchema.thrift introduces definitions for FeatureVector, Example, ModelRecord, and related structures to support various model types (e.g., spline, linear, RBF, MLP) and evaluation metrics. KDTree.thrift defines the schema for K-D tree nodes, enabling storage and retrieval of spatial data structures.

core/src/main/thrift · high confidence

Added a suite of new feature engineering transforms to the core library, including ApproximatePercentileTransform, BucketFloatTransform, CapFloatTransform, CoalesceFloatTransform, ConvertStringCaseTransform, CrossTransform, CustomLinearLogQuantizeTransform, CustomMultiscaleQuantizeTransform, CustomRangeQuantizeTransform, CutFloatTransform, DateDiffTransform, DateValTransform, DecisionTreeTransform, DefaultStringTokenizerTransform, DeleteFloatFeatureFamilyTransform, DeleteFloatFeatureTransform, DeleteStringFeatureColumnTransform, DeleteStringFeatureFamilyTransform, DeleteStringFeatureTransform, DivideTransform, FloatCrossFloatTransform, FloatFamilyCrossToTwoDDenseTransform, FloatLabelTransform, FloatToDenseTransform, and FloatToStringTransform. These additions provide capabilities for quantization, date handling, string tokenization, feature deletion, and cross-product generation.

core/src/main/java/com/airbnb/aerosolve/core/transforms · high confidence

The core library now supports a wider variety of machine learning models, including linear models (full-rank, low-rank, and standard), tree-based models (decision trees, random forests, and boosted stumps), kernel machines, and KD-trees. This update introduces the \AbstractModel\ base class to standardize model interfaces, enabling consistent scoring, serialization, and debug scoring across all model types. Users can now train and serve these new model architectures, which offer different trade-offs in terms of generalization, speed, and interpretability.

core/src/main/java/com/airbnb/aerosolve/core/models · high confidence

The training module now includes new implementations for several machine learning models: an AdditiveModelTrainer that uses backfitting with spline and linear functions, a BoostedForestTrainer for gradient boosted trees, a BoostedStumpsTrainer for simple tree priors, a DecisionTreeTrainer supporting classification, regression, and multiclass tasks, a ForestTrainer for random forests, and a FullRankLinearTrainer supporting softmax and hinge losses with rprop and sparse boost solvers. These additions expand the range of models users can train and evaluate within the Aerosolve framework.

training/src/main/scala/com/airbnb/aerosolve/training · high confidence

Added JsonParser, a utility for parsing JSON using Jackson, and Sort, which provides a quick-select algorithm with O(n) average time complexity for finding the nth element in an unsorted sequence. These new utilities are now available in the training package for use in training workflows.

training/src/main/scala/com/airbnb/aerosolve/training/utils · high confidence

This change introduces a suite of new utility classes to the \airlearner-utils\ library. A Java-based configuration loader (\AirCon\) and its associated macro executor are added to handle complex configuration loading and processing. Scala utilities are added for managing HDFS paths, parsing dates, and handling Hive table operations, including saving RDDs to Hive and updating partitions. Additionally, a generic \Job\ and \JobRunner\ framework is introduced to standardize the execution of Spark-based pipelines, while helper objects for sorting, random sampling, and logging are also included.

airlearner/airlearner-utils/src/main · high confidence

Added several new utility classes to the core library to support feature engineering and model debugging. The changes introduce a FloatVector class for efficient dense feature storage and manipulation, alongside helper classes like FeatureVectorUtil for computing kernel similarities. Additionally, new classes including Debug, DateUtil, Distance, and Weibull have been added to support debugging workflows, date range processing, geographic distance calculations, and statistical distributions respectively.

core/src/main/java/com/airbnb/aerosolve/core/util · high confidence

The income prediction demo configuration has been updated to support an additive model trained with dynamic buckets, allowing the model to adapt feature binning during training. The configuration also introduces learning rate decay and a minimum learning rate threshold, providing more stable convergence. Additionally, the config includes parameters for spline and additive models, with specific settings for evaluation and training data paths, model output locations, and hyperparameters such as iterations, subsample rate, and dropout.

_demo/income\prediction/src/main/resources · medium confidence

The function evaluation logic has been restructured by introducing an abstract base class \AbstractFunction\ and a \Function\ interface, providing a common foundation for all function types. New concrete implementations include \Linear\, \Point\, \Zero\, and \MultiDimensionSpline\, each handling specific feature representations (e.g., one-hot, constant, or multi-dimensional splines). The \FunctionUtil\ class was added to support polynomial fitting and smoothing operations. This refactoring standardizes how models are saved, loaded, and evaluated, enabling easier extension and more consistent behavior across different feature types.

core/src/main/java/com/airbnb/aerosolve/core/function · high confidence

The project now uses Gradle 4.0, as specified in the newly added gradle-wrapper.properties file. This update brings the build system to the 4.0 release, which may include new features, performance improvements, and compatibility changes for the build process.

gradle · high confidence

Added comprehensive unit tests for training algorithms; Added test logging configuration for unit tests; Added test resources for income prediction and logging configuration; Added unit tests and test utilities for ML utility classes; Added unit tests for JSON parsing and sorting utilities; Added unit tests for ModelScorer; Added unit tests for PhotonML data conversion utilities; Added unit tests for core feature transforms; Added unit tests for core function classes; Added unit tests for core utility classes; Added unit tests for feature generation and mapping logic; Added unit tests for image feature extractors and core ML models; Added unit tests for training pipeline components.

Added build configuration files for the Aerosolve project, including the root build.gradle, settings.gradle, and subproject build files for core, training, and demo modules. The configuration establishes a Gradle-based build system with dependencies on Apache Spark (version 3.1.1), Scala 2.12, and various utility libraries like Joda Time and SLF4J, while also configuring Bintray for artifact publishing.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 48 → 56 (+8.3)
• Rubric changed (rubric-2026.08.15 → rubric-2026.09.15) — scores are not directly comparable.

• Code Health 94 → 91 (-3.3)
• Architecture 94 → 97 (+3.3)
• Maturity 50 → 50 (-0.4)
• Readiness 28 → 46 (+18.0)
• Security 71 → 71 (+0.0)

• Coverage not measured — test suite did not build
• Dimension evaluation failed
• Duplicated block (152 lines × 2) (core/src/main/java/com/airbnb/aerosolve/core/util/FloatVector.java)
• Duplicated block (187 lines × 2) (core/src/main/java/com/airbnb/aerosolve/core/function/MultiDimensionSpline.java)
• Duplicated block (5 lines × 2) (core/src/main/java/com/airbnb/aerosolve/core/models/AdditiveModel.java)
• Duplicated block (5 lines × 2) (core/src/main/java/com/airbnb/aerosolve/core/models/SplineModel.java)
• Duplicated block (5 lines × 2) (core/src/main/java/com/airbnb/aerosolve/core/transforms/LinearLogQuantizeTransform.java)
• Duplicated block (6 lines × 2) (core/src/main/java/com/airbnb/aerosolve/core/models/LinearModel.java)
• Duplicated block (6 lines × 2) (core/src/main/java/com/airbnb/aerosolve/core/models/MaxoutModel.java)
• Duplicated block (6 lines × 2) (core/src/main/java/com/airbnb/aerosolve/core/models/MaxoutModel.java)
• Duplicated block (6 lines × 2) (core/src/main/java/com/airbnb/aerosolve/core/util/Debug.java)
• Duplicated block (7 lines × 2) (core/src/main/java/com/airbnb/aerosolve/core/models/AdditiveModel.java)
• Duplicated block (7 lines × 2) (core/src/main/java/com/airbnb/aerosolve/core/models/AdditiveModel.java)
• Duplicated block (7 lines × 2) (core/src/main/java/com/airbnb/aerosolve/core/models/LinearModel.java)
• Duplicated block (7 lines × 2) (core/src/main/java/com/airbnb/aerosolve/core/models/SplineModel.java)
• Duplicated block (7 lines × 2) (core/src/main/java/com/airbnb/aerosolve/core/util/Debug.java)
• Duplicated block (7 lines × 2) (core/src/main/java/com/airbnb/aerosolve/core/util/Debug.java)
• Duplicated block (8 lines × 2) (core/src/main/java/com/airbnb/aerosolve/core/models/MaxoutModel.java)
• Duplicated block (80 lines × 2) (core/src/main/java/com/airbnb/aerosolve/core/util/Util.java)
• High: security finding (details withheld)
• …and 7 more

• AdditiveModel.debugScoreItem (cognitive 17) (core/src/main/java/com/airbnb/aerosolve/core/models/AdditiveModel.java)
• AdditiveModelTrainer.train (cognitive 24) (training/src/main/scala/com/airbnb/aerosolve/training/AdditiveModelTrainer.scala)
• BinaryRegressionTrainer.train (cognitive 17) (airlearner/airlearner-strategy/src/main/scala/com/airbnb/common/ml/strategy/trainer/BinaryRegressionTrainer.scala)
• BoostedForestTrainer.addNewTree (cognitive 21) (training/src/main/scala/com/airbnb/aerosolve/training/BoostedForestTrainer.scala)
• BoostedForestTrainer.optionalExample (cognitive 17) (training/src/main/scala/com/airbnb/aerosolve/training/BoostedForestTrainer.scala)
• DecisionTreeTrainer.evaluateClassificationSplit (cognitive 29) (training/src/main/scala/com/airbnb/aerosolve/training/DecisionTreeTrainer.scala)
• DecisionTreeTrainer.getBestSplit (cognitive 16) (training/src/main/scala/com/airbnb/aerosolve/training/DecisionTreeTrainer.scala)
• DecisionTreeTrainer.makeLeaf (cognitive 27) (training/src/main/scala/com/airbnb/aerosolve/training/DecisionTreeTrainer.scala)
• Dependency hygiene PARTLY measured — Maven/Gradle declarations read, no dependency graph resolved
• Documentation: no installation or build instructions (README.md)
• Documentation: no licence statement (README.md)
• Documentation: no project overview (README.md)
• Documentation: no usage examples (README.md)
• Dormant codebase
• Duplicated block (10 lines × 2) (airlearner/airlearner-utils/src/main/scala/com/airbnb/common/ml/util/HiveUtil.scala)
• Duplicated block (10 lines × 2) (training/src/main/scala/com/airbnb/aerosolve/training/MaxoutTrainer.scala)
• Duplicated block (11 lines × 2) (training/src/main/scala/com/airbnb/aerosolve/training/LinearRankerTrainer.scala)
• Duplicated block (13 lines × 2) (training/src/main/scala/com/airbnb/aerosolve/training/BoostedForestTrainer.scala)
• Duplicated block (14 lines × 2) (training/src/main/scala/com/airbnb/aerosolve/training/FullRankLinearTrainer.scala)
• Duplicated block (14 lines × 3) (demo/image_impressionism/src/main/scala/JobRunner.scala)
• …and 101 more

• Containers 0 added · 0 removed · contexts 2 added · 0 removed · edges 0 added · 0 removed

• core
• repository

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

airbnb/aerosolve was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.