google/magika

Magika is a high-performance file content-type detection system that uses machine learning models to identify file formats from byte data. It provides a unified inference engine implemented in Rust, with native bindings for Python, Go, and JavaScript, allowing developers to integrate detection capabilities across multiple languages. The system supports various model configurations and hardware accelerations, offering a robust solution for accurately classifying diverse file types in different computing environments.

29 changes.

The project migrated its core inference engine from Python/ONNX to native Rust via PyO3 bindings, introducing dedicated Go and JavaScript libraries alongside a new Rust CLI. This period also involved standardizing release infrastructure, updating model versions to v3.3, and restructuring the codebase with comprehensive test suites and documentation.

13 changes.

This period focused on releasing the Magika V3 series of file-type detection models, progressively expanding the supported label space and configuration options. The project significantly broadened its integration ecosystem by introducing Python bindings, C API bindings, and JavaScript examples, while also replacing the inference runtime with a faster embedded Tract engine. Additionally, the documentation site was rebuilt using Astro, and a new Rust-based YARA rule evaluation crate was added to enhance detection capabilities.

A new file-type detection model named 'standard\_v2\_1' has been added to the assets. It supports a comprehensive list of 190 content types, ranging from common documents (PDF, DOCX, XLSX) and images (JPEG, PNG) to source code (Python, Java, C++) and archives (ZIP, TAR). The model configuration specifies a block size of 4096, a medium confidence threshold of 0.5, and specific high-confidence thresholds (0.95) for LaTeX and Pascal files.

_assets/models/standard\_v2\_0, assets/models/standard\_v2\1 · high confidence

Added the standard\_v3\_1 model configuration, including its documentation, minimal configuration file (config.min.json), and metadata. The configuration defines a target label space of 181 file types, sets specific confidence thresholds for certain labels (e.g., 0.95 for ignorefile and pascal), and maps random inputs to 'unknown' or 'txt'.

_assets/models/standard\_v3\1 · high confidence

New example projects have been added to demonstrate how to integrate Magika in different JavaScript environments. This includes a browser-based example using ES modules with Playwright for testing, Node.js examples using both CommonJS and ES module syntax, and a TypeScript ES module example. These examples show how to initialize the Magika model and identify file types from byte data in each context.

_js/simple\examples · high confidence

New model configuration files have been added for the 'begonly\_v2\_1' and 'fast\_v2\_1' variants. The 'begonly\_v2\_1' model is configured with a 2048-byte beginning context and no mid/end context, while the 'fast\_v2\_1' model uses a 512-byte beginning and 512-byte ending context. Both variants share the same extensive target label space (covering file types like code, documents, and archives) and specific confidence thresholds for LaTeX and Pascal, providing users with additional options for file type detection based on different context window sizes.

_assets/models/begonly\_v2\_1, assets/models/fast\_v2\1 · high confidence

A new model version, standard\_v3\_3, has been introduced in the assets/models directory. This update includes the model's configuration file (config.min.json), which defines parameters such as input sizes, block size, and a comprehensive list of target file types and confidence thresholds, along with its associated documentation (README.md) and metadata (metadata.json).

_assets/models/standard\_v3\3 · high confidence

Introduces the \magika-cli\ binary, a command-line interface for file content type detection using AI. The CLI supports recursive directory traversal, symbolic link handling, and multiple output formats including text, JSON, JSONL, and custom formats. It features true color support, configurable inference backends (CPU/GPU), and robust error handling for non-regular files, permission errors, and directory cycles. The tool is built with a standalone inference runtime (replacing ONNX Runtime) and includes comprehensive tests for stability.

rust/cli · high confidence

Adds a new Go module (\github.com/google/magika/go/magika\) that allows Go applications to infer file content types using the Magika ONNX model. The library handles feature extraction and delegates inference to the ONNX Runtime via cgo, exposing a \Scanner\ API for scanning byte streams. A CLI tool and example code are included to demonstrate usage, along with a Dockerfile for building a containerized environment with the required ONNX Runtime dependencies.

go · high confidence

This change adds a new PyO3-based Python wrapper for the Magika library, exposing the core file identification functionality to Python users. The \MagikaResult\ class provides detailed output including MIME type, label, score, and status, while the underlying Rust implementation handles path resolution, symlink handling, and feature extraction. This enables Python applications to leverage Magika's content type detection capabilities directly.

rust/pyo3 · high confidence

The new magika-rules crate provides a pure-Rust implementation of a bounded YARA subset, evaluating the first 4 KiB of input files to identify formats. It bundles pre-compiled rules (adapted from libmagic, PRONOM, puremagic, and Tika) that are generated at build time via build.rs, ensuring no native dependencies, unsafe code, or global state. The crate supports a specific subset of YARA constructs (text, hex, regex, integer reads, and logical operators) and categorizes rules into 'full' (zero observed false positives/negatives), 'partial' (some false negatives), and 'notworking' (parked rules requiring future facts stream support). Users can load bundled rules via RuleSet::bundled() and scan inputs, with regexes built on first use for efficiency.

rust/rules · high confidence

The website-ng directory now contains a complete, new static site built with Astro, Starlight, and Svelte. This replaces previous deployment configurations with a modern stack that includes Tailwind CSS for styling and shadcn-svelte for UI components. The site is configured to deploy to Google App Engine (Node.js 22 runtime) and includes a structured documentation sidebar covering introduction, installation, core concepts, CLI bindings, and contributing guidelines.

website-ng · high confidence

A new file-type detection model, standard\_v3\_0, is now available, featuring a comprehensive list of supported content types (including code, archives, documents, and media) and specific confidence thresholds for labels such as LaTeX, Markdown, and SQL. The model configuration defines input parameters like block size and padding, maps ambiguous inputs like random bytes to 'unknown', and is identified as major version 3 with 100 training epochs.

_assets/models/standard\_v3\0 · high confidence

Added the standard\_v3\_2 model assets, including a configuration file (config.min.json) that defines model parameters such as input sizes, block size, and a comprehensive list of target file types, along with specific confidence thresholds and an overwrite map for unknown types. The update also includes a metadata file recording the model epoch and a README documenting the full list of supported content type labels.

_assets/models/standard\_v3\2 · high confidence

The project has significantly expanded its documentation, replacing the minimal README with a comprehensive guide covering installation methods (Homebrew, pipx, cargo, npm), usage examples, and links to the research paper and web demo. A CITATION.cff file has been added to facilitate proper academic referencing. Additionally, the release process for the Rust CLI has been modernized by adopting cargo-dist for generating installers and managing CI, while a new Dockerfile provides a secure, non-root containerized environment for the Python package.

(repo-wide) · high confidence

The Magika Rust library is now exposed as a C library, providing a C99/C++ compatible header and both static and dynamic libraries for integration into C/C++ projects. The API includes functions to initialize a shared inference runtime, create identification sessions, and detect file types from disk paths or in-memory buffers, with support for configuring hardware backends (CPU/GPU) and batch processing. A C example and a comprehensive test suite are included to demonstrate usage and validate behavior across Linux and macOS platforms.

rust/ffi · high confidence

The \python/scripts\ directory now includes a comprehensive suite of automation tools to support the Python package's lifecycle. This includes \build\_wheel.sh\ and \stage\_cli.sh\ to handle native Rust CLI binary staging and wheel packaging, and \run\_phases\_1\_to\_3.sh\ to automate local development checks, wheel inspection, and clean-environment validation. For quality assurance, \check\_copyright.py\, \check\_changelog.sh\, and \check\_documentation.py\ enforce code standards and documentation integrity, while \pre\_release\_check.py\ validates version consistency before publishing. Additionally, \sync.py\ manages model and content-type synchronization, and \run\_quick\_test\_magika\_cli.py\ / \run\_quick\_test\_magika\_module.py\ provide rapid feedback on CLI and module functionality.

python/scripts · high confidence

The \js/src\ directory now includes a comprehensive set of TypeScript definitions and configuration classes that underpin the Magika content-type detection engine. This change introduces the \ContentTypeLabel\ enum and \ContentTypeInfo\ interface to represent detected file types and their properties (such as whether they are text), alongside a large auto-generated \ContentTypesInfos\ registry. It also adds the \MagikaOptions\, \MagikaResult\, and \MagikaPrediction\ interfaces to define the API contract for configuration, results, and prediction scores. Furthermore, it implements the \ModelConfig\ and \ModelConfigNode\ classes to handle loading model parameters (like thresholds and overwrite maps) from URLs or local files, and the \ModelFeatures\ class to prepare input data for the TensorFlow.js model inference.

js/src · high confidence

The website-ng public directory now includes a new favicon and logo SVG, along with the Magika V3.2 file-type detection model. This model, converted to TensorFlow.js, introduces a larger label space of 214 types (up from the previous 1024-byte input size) and is configured with a block size of 4096, a medium confidence threshold of 0.5, and specific thresholds for formats like LaTeX and Markdown.

(repo-wide) · high confidence

This change introduces the foundational structure and tooling for the Rust implementation of Magika. It adds the \cli\, \lib\, \tract-runtime\, and \gen\ crate directories, along with shell scripts (\publish.sh\, \sync.sh\, \test.sh\, \latest.sh\, \changelog.sh\) to automate publishing to crates.io, synchronizing generated files and models, running tests, and managing the \cli-latest\ release. It also includes the \tract-bench\ crate with utilities to convert ONNX models to NNEF, verify model accuracy, and benchmark inference throughput, alongside configuration files for formatting and linting.

rust · high confidence

The main CLI script at magika/cli/magika.py has been deleted. This removes the command-line interface that previously allowed users to identify file types via arguments for options like JSON output, MIME types, labels, and model directory selection.

magika-python-package/magika/cli · high confidence

A change to existing behaviour in tests\_data/previous\_missdetections — 1 commit, 1 file.

_tests\_data/previous\missdetections · low confidence · unverified

A change to existing behaviour in tests\_data/current\_missdetections — 2 commits, 2 files.

_tests\_data/current\missdetections · medium confidence · unverified

The website now serves dynamic redirect pages for the shell and PowerShell installer scripts. Visiting the install.sh or install.ps1 endpoints will no longer serve a static file; instead, users are immediately redirected (302) to the latest versions of the installer scripts hosted in the GitHub releases (cli-latest). This ensures users always download the most recent installer without needing to update their bookmarks or scripts.

website-ng/src/pages · high confidence

The Magika library and CLI now use a new embedded Tract inference runtime instead of ONNX Runtime. This runtime loads the model from a pre-parsed graph (model.graph.json) rather than parsing an NNEF archive at startup, reducing load time from approximately 14 ms to 3 ms. It supports fixed batch sizes (1, 4, 8, 16, 32, 64) and prepares target-specific execution plans once at startup, allowing inference threads to spawn private mutable state from those shared plans. The runtime supports automatic, CPU, or GPU device selection (with Metal on macOS and optional CUDA support), and includes a startup health check on GPU devices to verify plan correctness against CPU references.

rust/tract-runtime · high confidence

The legacy website now redirects users to the new Magika website, but continues to serve the \standard\_v3\_2\ TensorFlow.js model files (including \config.min.json\, \metadata.json\, and \model.json\) from the \public/models/standard\_v3\_2\ directory. This ensures that existing integrations relying on the old model URLs remain functional for backwards compatibility with early Magika versions, even though the main site experience has moved.

website · high confidence

The Magika Python library has been restructured to use a PyO3-based Rust backend, introducing a new \Magika\ class with methods like \identify\_path\, \identify\_paths\, and \identify\_bytes\. The public API now exposes new types such as \MagikaPrediction\, \MagikaResult\, \OverwriteReason\, and \PredictionMode\, replacing previous output structures. The default model has been updated to \standard\_v3\_3\, and the library now supports a \no\_dereference\ option to avoid following symlinks. Internal logging has been simplified to use a custom \SimpleLogger\ writing to stderr, and the package is now marked as typed with \py.typed\.

python/src/magika · high confidence

The Magika JavaScript library has been updated to version 1.1.0, which now requires Node.js 20 or later due to the end-of-life of Node 18. This release also introduces dual CommonJS (CJS) and ECMAScript Module (ESM) support, allowing the library to be imported correctly in both module systems. The package structure now includes separate \dist/cjs\ and \dist/mjs\ directories with corresponding \package.json\ files to facilitate this, alongside a TypeScript refactoring that organizes the codebase into \src/\ and exposes \Magika\ for browser use and \MagikaNode\ for Node.js environments.

js · high confidence

The website now uses a new global stylesheet that imports Tailwind CSS, Shadcn UI, and Animate.css, establishing a consistent design system with light and dark mode color variables. Additionally, list items within the \.sl-container\ class are now styled with standard bullet and decimal markers, while the table of contents navigation list remains unstyled to prevent visual clutter.

website-ng/src/styles · high confidence

The Python package now uses native Rust bindings (via PyO3) for inference, replacing the previous pure-Python ONNX Runtime implementation. This change removes runtime dependencies on \onnxruntime\, \click\, and \numpy\, resulting in significantly faster startup and inference speeds. The public API remains compatible, but the underlying engine is now the optimized Rust \magika\ crate.

python · high confidence

The standard\_v1 model now correctly identifies RAR archives using the official \application/vnd.rar\ MIME type instead of the legacy \application/x-rar\. Additionally, the model's detection thresholds have been significantly raised for numerous file types (such as APK, HTML, PHP, and various Office formats), requiring higher confidence scores for positive identification, which generally reduces false positives at the cost of potentially lower recall for ambiguous files.

_assets/models/standard\v1 · high confidence

The Python API's data models have been restructured into a dedicated \magika.types\ package, introducing new classes such as \MagikaResult\, \MagikaPrediction\, and \ContentTypeInfo\ to better represent scan outcomes and prediction details. This change includes the addition of an \OverwriteReason\ field to \MagikaPrediction\ to explain why the final output may differ from the raw deep learning result, and provides an \asdict()\ method on \MagikaResult\ for easy serialization. Additionally, several legacy properties on \ContentTypeInfo\ (such as \.ct\_label\, \.magic\, and \.score\) are now deprecated or removed, with users directed to use the updated model structure.

python/src/magika/types · high confidence

The \content\_types.py\ module, which defined the \ContentType\ and \ContentTypesManager\ classes along with the \content\_types\_config.json\ path, has been removed. The main \magika.py\ class no longer imports or instantiates \ContentTypesManager\, and the \model\_output\_overwrite\_map.json\ file (which previously mapped 'odex' to 'elf') has also been deleted. This indicates a significant internal restructuring where the previous content type mapping and model output override logic has been stripped out or moved to a different location not visible in this diff.

magika-python-package/magika · medium confidence

Test data files, including mini.protobuf, html.htm, and mini.plist, have been moved or added to the tests\_data/mitra\_candidates directory. This reorganization groups these specific file types (protobuf, HTML, plist) together for candidate testing purposes.

_tests\_data/mitra\candidates · high confidence

The test data directory for Mitra has been reorganized to group files by their expected content type (e.g., pdf, php, svg) within subdirectories. This change introduces new sample files for PDF, PHP, and SVG formats, replacing the previous flat structure to better align with content-type-based testing.

_tests\data/mitra · medium confidence

The Rust code generator has been updated to use the standard\_v3\_3 model, replacing the previous standard\_v3\_0 target. This change updates the symbolic link in \rust/gen/model\ to point to \../../assets/models/standard\_v3\_3\, ensuring that the generated Rust library (\rust/lib/src/model.rs\ and \rust/lib/src/content.rs\) reflects the configuration, labels, and content types defined in the v3\_3 model. The generator script (\rust/gen/src/main.rs\) and its test runner (\rust/gen/test.sh\) have been restructured to support this new model version, including filtering content types based on the model's target labels and generating corresponding Rust enums and constants.

rust/gen · high confidence

The Rust library has been updated to version 2.0.0, introducing significant breaking changes and new capabilities. The underlying inference engine has switched from ONNX Runtime to an embedded Tract runtime, removing the ONNX dependency and enabling GPU inference via a new \cuda\ feature. The public API has been simplified: the \Session\ type is replaced by a \Runtime\ for model preparation and per-thread session creation, async support is removed, and the \Builder\ now constructs a \Runtime\. New features include \ContentType::from\_label()\ for lookups by label, configurable inference backends (CPU/GPU), and batch processing that accepts an iterator of features. Small file reading during feature extraction has also been optimized to occur in a single pass.

rust/lib · high confidence

The Rust library has replaced the previous ONNX Runtime with an embedded Tract inference runtime, enabling CPU and GPU acceleration via the new \Backend\ and \Builder\ APIs. The underlying AI model has been updated to \standard\_v3\_3\, which includes new content type definitions, updated confidence thresholds, and an overwrite map to refine classification results.

rust/lib/src · high confidence

Added comprehensive test suite for Magika JS; Added comprehensive test suite for the Python module; Expanded test dataset with new file format samples; Removal of Python CLI and module test suite; Removed test symlinks to internal package.

This change introduces the foundational dependency manifests for the Go, Python, and Rust parts of the project. The Go module (\go/go.mod\) is initialized with Go 1.22.3 and the \go-cmp\ library. The Python package (\python/pyproject.toml\) is configured for version 2.0.0.dev0, specifying support for Python 3.8 through 3.14 and defining development dependencies for testing and linting (mypy, pytest, ruff, etc.). The Rust components (CLI, library, FFI, rules, etc.) are initialized with their respective \Cargo.toml\ files and lock files, establishing the core dependencies for the AI-based content type detection engine.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 57 → 62 (+4.9)
• Rubric changed (rubric-2026.09.8 → rubric-2026.09.16) — scores are not directly comparable.

• Code Health 65 → 93 (+27.8)
• Architecture 93 → 92 (-0.4)
• Maturity 60 → 61 (+1.1)
• Readiness 69 → 61 (-7.8)
• Security 48 → 57 (+9.0)
• Accessibility 64 → 64 (+0.0)
• Performance 100 (new)

• Critical CVE: [CVE redacted] (js/yarn.lock)
• Critical CVE: [CVE redacted] (website-ng/package-lock.json)
• Critical CVE: [CVE redacted] (website-ng/package-lock.json)
• Documentation: no installation or build instructions (README.md)
• Documentation: no usage examples (README.md)
• Documentation: written for insiders (rust/gen/README.md)
• Duplicated block (5 lines × 2) (rust/tract-bench/src/bin/convert_model.rs)
• Further sole-owners (lower concentration)
• HackComment (python/tests/test_features_extraction_vs_reference.py)
• HackComment (python/tests/test_inference_vs_reference.py)
• High CVE: [CVE redacted] (js/yarn.lock)
• High CVE: [CVE redacted] (js/yarn.lock)
• High CVE: [CVE redacted] (website-ng/package-lock.json)
• High CVE: [CVE redacted] (js/yarn.lock)
• High CVE: [CVE redacted] (website-ng/package-lock.json)
• High CVE: [CVE redacted] (js/yarn.lock)
• High CVE: [CVE redacted] (website-ng/package-lock.json)
• High CVE: [CVE redacted] (website-ng/package-lock.json)
• High CVE: [CVE redacted] (js/yarn.lock)
• High CVE: [CVE redacted] (website-ng/package-lock.json)
• …and 67 more

• Coverage not measured — JavaScript/TypeScript suite
• Duplicated block (11 lines × 2) (python/scripts/sync.py)
• FixmeComment (python/src/magika/magika.py)
• FixmeComment (python/src/magika/magika.py)
• FixmeComment (python/src/magika/magika.py)
• FixmeComment (python/src/magika/magika.py)
• FixmeComment (python/src/magika/magika.py)
• FixmeComment (rust/pyo3/src/lib.rs)
• FixmeComment (rust/pyo3/src/lib.rs)
• FixmeComment (rust/pyo3/src/lib.rs)
• High CVE: [GHSA redacted] (js/yarn.lock)
• High CVE: [GHSA redacted] (js/yarn.lock)
• High CVE: [GHSA redacted] (js/yarn.lock)
• High CVE: [GHSA redacted] (js/yarn.lock)
• High CVE: [GHSA redacted] (js/yarn.lock)
• High CVE: [GHSA redacted] (website-ng/package-lock.json)
• High vulnerability: [GHSA redacted] (rust/pyo3/Cargo.lock)
• High: security finding (details withheld)
• High: security finding (details withheld)
• InferredType exposes both a public property content_type of type String and a method content_type() returning ContentType. This is a direct signature conflict where the same name refers to two different types and access patterns (field vs method).
• …and 12 more

• 36 commits — 33 feature/other, 3 fixes

• .github/workflows — 8 commits
• rust/pyo3 — 4 commits
• python/scripts — 3 commits
• python/src — 3 commits
• rust/lib — 3 commits
• (repo) — 2 commits
• rust/cli — 2 commits
• rust/rules — 2 commits
• rust/tract-runtime — 2 commits
• .github/CODEOWNERS — 1 commit
• js/src — 1 commit
• python/CHANGELOG.md — 1 commit
• python/pyproject.toml — 1 commit
• rust/color.sh — 1 commit
• rust/ffi — 1 commit
• rust/sync.sh — 1 commit

• fix: Fix before-script-linux path in poc-pyo3-wheels workflow
• fix: Fix tract convolution padding and check every GPU batch plan at startup (#1466)
• fix: workflow: fix CLI binary staging path for Linux targets and comment out upstream tract-linalg limitations
• change: Add 62 content types to the knowledge base (#1471)
• change: Add multi-platform test matrix and unify CLI binary staging in workflow
• change: Add the magika-rules crate (#1462)
• change: Comment out retired macos-13 (x86_64-apple-darwin) runner from workflow
• change: Compile the bundled rules when the crate is built (#1492)
• change: Detect recursive directory cycles in the CLI (#1463)
• change: Harden the CLI against bad limits, special files and pipeline errors (#1464)
• change: Implement PyO3 wrapper for Magika Python library POC
• change: Link the C library on macOS (#1470)
• change: Look up content types by label (#1484)
• change: Make ia0 code owner of anything below rust (#1498)
• change: Merge pull request #1478 from google/less-coverall-tests
• change: Merge pull request #1483 from google/pyo3
• change: Port PyO3 wrapper to embedded tract inference runtime and update workflow
• change: Read small files once during feature extraction (#1465)
• change: Remove legacy CLI, Click dependency, models dir, and expose model name via PyO3
• change: Remove legacy Python fallback code, implementation data models, and non-critical tests
• …and 16 more

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

google/magika was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.