autodeployai/ai-serving

This system is an AI model serving platform that supports deploying and inferring machine learning models in PMML and ONNX formats. It provides HTTP and gRPC endpoints compliant with the v2 inference protocol, featuring automatic request batching to optimize throughput. The service is containerized for both CPU and GPU environments, leveraging ONNX Runtime for acceleration, and includes example notebooks for training and deployment workflows.

Users can now build and run AI Serving containers for both CPU and GPU environments. The new Dockerfiles use Ubuntu 22.04 and OpenJDK 17, with the CUDA variant leveraging NVIDIA CUDA 12.3.2 and cuDNN 9. The CPU image builds a standard assembly, while the CUDA image enables the ONNX Runtime backend via the \--add-exports\ and \-Donnxruntime.backend=cuda\ flags to support GPU acceleration.

dockerfiles · high confidence

The examples/models directory now includes a new \xgb-iris.pmml\ file, providing a pre-exported PMML representation of an XGBoost classification model trained on the Iris dataset. This file, generated by Nyoka version 5.5.0, contains the complete model definition including data dictionary, standard scaler transformations, and the tree-based mining model structure, allowing users to load and infer using this specific example without needing to train the model locally.

examples/models · high confidence

Added example notebooks demonstrating how to train, convert, and serve machine learning models using AI-Serving. The \IrisXGBoost.ipynb\ notebook shows how to train an XGBoost classifier and convert it to PMML format. The \AIServingIrisXGBoostPMMLModel.ipynb\ notebook provides a tutorial on deploying and making predictions with that PMML model via HTTP. Additionally, the \AIServingMnistOnnxModel.ipynb\ notebook demonstrates deploying and serving a pre-trained MNIST CNN model in ONNX format. These examples also include the generated \ai\_serving\_pb2.py\ Python file required for interacting with the AI-Serving gRPC/HTTP endpoints.

examples · high confidence

The serving layer now supports the v2 inference protocol for both HTTP and gRPC endpoints, exposing standard health, metadata, and inference routes (e.g., /v2/health/live, /v2/models/{name}/infer) alongside the existing v1 APIs. Additionally, the system introduces automatic batching for supported models (such as ONNX and PMML), which aggregates incoming requests to improve throughput based on configurable batch size and delay parameters.

src/main/scala/com · high confidence

The model serving infrastructure in the \ai.autodeploy.serving\ package has been removed. This includes the deletion of the \ModelCache\ and \ModelManager\ classes that handled model versioning and deployment, the removal of specific model runtime implementations for ONNX (\OnnxModel\) and PMML (\PmmlModel\), and the elimination of the gRPC \DeploymentServiceImpl\ along with its associated protobuf conversion utilities (\protobuf/package.scala\) and JSON support (\JsonSupport\, \JsonUtils\).

src/main/scala/ai · high confidence

The AI-Serving gRPC API now supports automatic batching inference via a new DeployConfig message (allowing configuration of request timeout, max batch size/delay, and warmup settings) in the DeploymentService. The service also introduces the v2 inference protocol through a new grpc\_predict\_v2.proto defining standard server/model metadata and inference RPCs. Additionally, model version identifiers have been changed from integers to strings across ModelInfo, ModelMetadata, and ModelSpec, and the internal tensor representation has been updated to a custom TensorProto message replacing the previous ONNX dependency.

src/main/protobuf · high confidence

The application's AI dispatcher has been renamed to 'ai-dispatcher' and its thread pool size is now dynamically set to use all available CPU cores (-1) instead of a fixed size of 16. New configuration sections have been added to support ONNX Runtime backend options (including CPU, CUDA, DNNL, TensorRT, and DirectML) and to control request timing logs. Additionally, the logging format has been updated to include method and line number details, the root logger package has been changed from 'ai.autodeploy.serving' to 'com.autodeployai.serving', and explicit logging levels have been configured for Akka and Netty components.

src/main/resources · high confidence

Added test coverage for v2 inference APIs and batch processing; Added test resources for v2 API and PMML model validation; Removed BaseSpec test helper.

The build manifest has been updated to version 2.2.0, significantly upgrading core libraries including Akka HTTP (10.1.11 to 10.5.3), Akka Streams (2.6.4 to 2.7.0), PMML4s (0.9.5 to 1.5.8), and ONNX Runtime (1.6.0/1.7.0 history to 1.22.0). The project now supports GPU acceleration for ONNX models via a configurable 'gpu' build property that switches the dependency to 'onnxruntime\_gpu'. Additionally, Logback was upgraded to 1.5.13, and the assembly merge strategy was adjusted to handle Java 9+ module info files, while test execution was configured to fork with specific Java exports.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• First survey — no prior run to compare against. CAI 59.

• Code Health 91
• Architecture 95
• Maturity 57
• Readiness 50
• Security 66

• 73 commits — 67 feature/other, 6 fixes

• (root) — 26 commits
• src/main — 15 commits
• src/test — 11 commits
• dockerfiles/scripts — 6 commits
• .github/workflows — 5 commits
• (repo) — 3 commits
• examples/AIServingMnistOnnxModel.ipynb — 3 commits
• dockerfiles/Dockerfile — 2 commits
• examples/AIServingIrisXGBoostPMMLModel.ipynb — 2 commits

• fix: Fix failed UTs for various platforms
• fix: Fix the build error
• fix: Fix the memory leak after deploying
• fix: Fixed warmup failure
• fix: fix build error
• fix: fix build error
• change: Add conf.json
• change: Add more details of Predict Protocol V2 in README.md
• change: Add run options to cancel running requests
• change: Add test cases for v2 APIs
• change: Build image when a tag pushed only
• change: Bump pmml4s to 1.5.8
• change: Correct the link
• change: Correct the outdated link
• change: Create cuda-docker-image.yml
• change: Create docker-image.yml
• change: Handle errors from onnxruntime
• change: Merge branch 'master' of https://github.com/autodeployai/ai-serving
• change: Merge pull request #17 from autodeployai/api-v2
• change: Merge pull request #19 from autodeployai/api-v2-enhance
• …and 53 more

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

autodeployai/ai-serving was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.