deeplethe/utopia

Utopia is a knowledge base system that ingests documents, extracts structured facts, and maintains an ontology-driven graph with rigorous governance and reasoning capabilities. It features an open extraction pipeline with provenance tracking, an R1 reasoning engine for deriving facts and enforcing business rules, and a chat interface backed by a rig agent framework for tool-use. The system includes comprehensive operational tools for backup, security hardening, and performance benchmarking, alongside a web UI for managing the knowledge graph and reviewing governance decisions.

The database schema now supports a comprehensive reasoning and governance layer. Business logic is introduced via \attribute\_rules\ and \attribute\_rule\_conditions\, allowing users to define typed facts and attribute derivations based on entity properties, with results stored in \derived\_facts\. Ontology integrity is enforced through \ontology\_defects\ (for self-contradictory definitions) and \axiom\_violations\ (for data-definition conflicts, including signature and derived contradictions). Data provenance and security are strengthened with a new \audit\_events\ table (append-only), \personal\_tokens\ for MCP agent authentication, and \data\_source\_grants\ to restrict data source access by workspace. The system also tracks pending user confirmations (\pending\_facts\), handles document deletion as a reversible event (\document\_deletions\), and supports multiple query engines (Trino, Databricks, Snowflake, MySQL) via \data\_sources\. Finally, conversation context is preserved via \tool\_exchange\ to improve agent continuity.

migrations · high confidence

The \utopia\ command-line tool is now available, providing operators with \backup\ and \restore\ subcommands to manage system state. The \backup\ command is fully implemented, allowing users to snapshot the Postgres database and optionally include the data directory and secret key into a tarball, with support for dry runs and custom binary paths. The \restore\ command is currently stubbed, accepting arguments like \--from\ and \--force\ but not yet performing the actual restoration logic.

crates/utopia-cli · high confidence

The \utopia-reason\ crate now implements the R1 reasoning layer, which materializes derived facts based on ontology axioms (transitive, symmetric, inverse, and sub-property relationships) and evaluates business rules against entity attributes. This change adds a new derivation engine (\derive.rs\) that computes transitive closures with depth limits and cycle detection, introduces an ontology consistency checker (\ontology.rs\) to detect definition-level defects like symmetric/asymmetric conflicts or subclass cycles, and adds a business rule evaluator (\rules.rs\) that derives new facts from attribute values and entity relationships. The system enforces strict precedence of asserted facts over derived ones, handles temporal validity through interval intersection, and caps derivations per predicate to prevent unbounded expansion.

crates/utopia-reason · high confidence

The extraction module has been restructured to implement an 'open extraction' phase where the model writes statements using the document's own words without referencing the ontology or normalizing dates. This raw output is subsequently processed by new alignment components that bind relation phrases and kind words to ontology properties and classes, respectively, using signature-based caching. The pipeline also includes modules for interpreting time mentions, deriving implicit facts, reviewing extracted facts via an errata agent, and adjudicating entity identity pairs with a multi-step lookup tool.

crates/utopia-extract · high confidence

The \scripts/bench\ directory now provides a comprehensive set of tools to measure the system's performance across several key capabilities. It includes a typed-graph benchmark using Re-DocRED to evaluate entity type alignment, a governance measurement tool (\govern.mjs\) to assess the accuracy of the agent's entity merging decisions, and a recall benchmark (\recall.mjs\) using SEC filings to check for missing quantitative facts. Additionally, it introduces an end-to-end question-answering benchmark (\ask.mjs\) to verify numerical accuracy, a mapping evaluation tool (\mappings.mjs\) for semantic layer proposals, and an audit script (\audit.sql\) for diagnosing extraction quality. The suite also provides scripts to fetch and prepare various corpora, including AI company timelines, Wikipedia history snapshots, and Wikidata-based ground truth.

scripts/bench · high confidence

This change introduces several new backend capabilities and operational tools. First, it adds a new \migrate\ example (\crates/utopia-store/examples/migrate.rs\) that allows developers to manually run database migrations against a target database using the \UTOPIA\_DATABASE\_URL\ environment variable, facilitating local testing and CI setup. Second, it implements a new alerting system (\alerts.rs\) that records system and knowledge-base faults (such as sync failures, LLM rate limits, and governance trips) as immutable rows, with visibility controls based on user roles and support for grouping and purging. Third, it adds an alignment queue (\alignment\_queue.rs\) to hold phrase signatures, kind words, and implication rules for human review when automated alignment is uncertain. Fourth, it introduces business rules storage (\business\_rules.rs\) allowing users to define and manage rules that derive entity types or relations based on conditions. Fifth, it implements an errata review system (\errata.rs\) for an agent to audit typed facts for structural issues (domain/range violations, missing names) and apply corrections. Finally, it adds an execution gate (\execution\_gate.rs\) that evaluates the impact of merges or fact changes on derived facts and chat answers before allowing automatic actions, and tracks mapping exploration runs (\exploration\_runs.rs\) to provide coverage and drop statistics.

crates/utopia-store/src · high confidence

The \utopia-ingest\ crate introduces a new ingestion architecture that structures document parsing into distinct stages: block-level segmentation, table normalization, and provenance anchoring. It adds a \blocks\ module to parse Markdown into semantic units (headings, paragraphs, tables) and rejoin tables split by page breaks, ensuring table headers persist across chunks. A new \table\ module normalizes complex HTML tables (handling colspans, empty columns, and implicit headers) into clean Markdown. Provenance is now tracked via \Origin\ types (Stated, OCR, Transcribed, Described) and \Segment\ structures, allowing the system to preserve source context for scanned documents, audio transcripts, and raw text. The pipeline also includes specialized readers for MinerU OCR output and OpenAI diarized transcripts, and an OWL/RDF ontology projector for semantic vocabularies.

crates/utopia-ingest · high confidence

Added a new benchmarking suite in \scripts/bench/corpora/contracts\ to evaluate how well the system handles temporal changes in contract chains. This includes a manifest of SEC EDGAR filings (primarily Blackbaud lease agreements and amendments), a script to fetch these documents, and a benchmark runner (\lease\_bench.py\) that ingests the data into a knowledge base with a specific ontology and scores the system's ability to answer 'as-of' questions against 18 predefined ground-truth values.

scripts/bench/corpora/contracts · high confidence

A new Node.js script (scripts/star-history.mjs) generates a cumulative star-history SVG chart by fetching data via the GitHub GraphQL API, ensuring monotonic interpolation and handling permission restrictions. Additionally, the default port for local development and smoke tests has been moved from 8080 to 1516, and several scripts have been made executable.

scripts · high confidence

This change introduces static encryption for sensitive credentials (LLM keys, connection strings, tokens) using AES-256-GCM, ensuring that a database leak does not expose secrets; it also adds a dedicated migration database URL to allow the application to run with restricted permissions while migrations use a privileged connection. Additionally, the default server port is moved from 8080 to 1516, and the configuration now treats blank environment variables as unset to prevent deployment failures in containerized environments.

crates/utopia-core · high confidence

The chat conversation loop has been rewritten to use the rig agent framework instead of the previous hand-written loop. This change enforces that a model must call at least one tool before ending a turn, preventing the "I will check" dead-end where the model describes a plan without executing it. It also introduces a single retry for empty model replies to avoid wasting evidence, and adds a new \no\_evidence\_needed\ tool for questions that do not require knowledge base retrieval.

crates/utopia-server · high confidence

The Docker setup now initializes a restricted 'utopia\_app' database role during first-time container startup to enforce least-privilege access, while the application server's default listening port has changed from 8080 to 1516. Additionally, the build process now aligns the Rust build stage with the Debian bookworm runtime to prevent GLIBC version mismatches, includes necessary documentation sources for binary embedding, and installs poppler-utils to support PDF text extraction.

docker · high confidence

The web interface now uses the shadcn/ui component library for its UI elements. This introduces a new design system with standardized tokens for typography, spacing, and colors, enforced by a new style-guard script. Existing dialogs and controls have been replaced with shadcn equivalents (e.g., Dialog, Button, Badge), and the configuration is set up in components.json to use the 'radix-nova' style with Tailwind CSS variables.

web · high confidence

The placeholder documentation comments have been removed from the root library files of the utopia-connectors, utopia-graph, and utopia-mcp crates. These crates currently serve as structural placeholders for future features (external source connectors, graph traversal, and MCP server tools), and this change cleans up the initial roadmap notes while the actual implementations remain in other parts of the codebase.

crates/utopia-connectors, crates/utopia-graph, crates/utopia-mcp · high confidence

The LLM client now distinguishes between transient failures (rate limits, server unavailability, interrupted streams) and permanent errors (account out of credit, rejected requests). Transient errors are now typed and include retry-after information, enabling the system to wait and retry automatically instead of failing silently or treating them as fatal. Permanent errors are clearly separated so that retries are not wasted on issues that will not resolve themselves, such as insufficient funds or invalid API keys. This change improves reliability by ensuring that recoverable network issues do not cause document processing failures, while non-recoverable issues are surfaced correctly for user action.

crates/utopia-llm · high confidence

Added integration tests for store concurrency, job scheduling, and business rule history; Integration tests for store reasoning, materialization, and ingestion.

The web frontend has upgraded to React 19 and added shadcn/ui, Radix UI, and several new UI components, while the Rust backend has restructured its workspace by removing the \utopia-connectors\, \utopia-graph\, and \utopia-mcp\ crates and introducing a new \utopia-cli\ crate. Key library updates include \jsonwebtoken\ 9 to 10 to address an authentication bypass vulnerability, \aws-lc-rs\ to 1.18.1, and \syn\ to 3.0.5, alongside the addition of \aes-gcm\ for credential encryption, \object\_store\ for cloud storage integration, and \tiktoken-rs\ for token-based chunking.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 57 → 55 (-1.9)
• Rubric changed (rubric-2026.09.9 → rubric-2026.09.18) — scores are not directly comparable.

• Code Health 62 → 64 (+2.4)
• Architecture 90 → 79 (-11.7)
• Maturity 61 → 87 (+25.9)
• Readiness 60 → 52 (-7.5)
• Security 66 → 69 (+3.3)
• Domain Modelling 100 → 100 (+0.0)
• Accessibility 50 → 50 (-0.1)
• Performance 60 (new)

• Change coupling: models.rs ↔ mod.rs (crates/utopia-core/src/models.rs)
• Change-coupling hub: zh.ts → models.rs, governance.rs, ingest_sources.rs, governance.rs (web/src/i18n/zh.ts)
• Chat.TurnView (cyclomatic 18) (web/src/pages/Chat.tsx)
• Documentation: no installation or build instructions (README.md)
• Documentation: no usage examples (README.md)
• Duplicated block (10 lines × 2) (crates/utopia-server/src/api/chat.rs)
• Duplicated block (10–11 lines × 2) (crates/utopia-llm/src/lib.rs)
• Duplicated block (12 lines × 2) (crates/utopia-store/src/documents.rs)
• Duplicated block (13 lines × 2) (crates/utopia-store/src/documents.rs)
• Duplicated block (14 lines × 2) (crates/utopia-store/src/ontology.rs)
• Duplicated block (14 lines × 3) (crates/utopia-server/src/extraction.rs)
• Duplicated block (15 lines × 2) (crates/utopia-server/src/query_engine/mysql.rs)
• Duplicated block (15 lines × 2) (crates/utopia-store/src/paths.rs)
• Duplicated block (15 lines × 2) (crates/utopia-store/src/pending.rs)
• Duplicated block (15 lines × 3) (crates/utopia-server/src/extraction.rs)
• Duplicated block (15–16 lines × 3) (crates/utopia-store/src/temporal.rs)
• Duplicated block (16 lines × 3) (crates/utopia-server/src/extraction.rs)
• Duplicated block (19–25 lines × 2) (crates/utopia-server/src/extraction.rs)
• Duplicated block (20 lines × 3) (crates/utopia-server/src/extraction.rs)
• Duplicated block (27 lines × 2) (crates/utopia-server/src/extraction.rs)
• …and 42 more

• Change coupling: AccountShell.tsx ↔ Shell.tsx (web/src/pages/AccountShell.tsx)
• Change coupling: pipeline.rs ↔ api.ts (crates/utopia-server/src/pipeline.rs)
• Change coupling: rdf.rs ↔ graph.rs (crates/utopia-server/src/rdf.rs)
• Change-coupling hub: extraction.rs → lib.rs, kbs.rs, bootstrap_ontology.rs, mappings.rs, jobs.rs (crates/utopia-server/src/extraction.rs)
• Change-coupling hub: zh.ts → models.rs, extraction_open.rs, governance.rs, ingest_sources.rs, extraction_drops.rs, governance.rs, jobs.rs (web/src/i18n/zh.ts)
• ClassTooLong: LlmClient (crates/utopia-llm/src/lib.rs)
• ClassTooLong: Settings (web/src/pages/Settings.tsx)
• DatabricksEngine::run (cognitive 19) (crates/utopia-server/src/query_engine/databricks.rs)
• DocViewer.DocViewer (cognitive 16) (web/src/pages/DocViewer.tsx)
• Duplicated block (10 lines × 2) (crates/utopia-server/src/query_engine/mysql.rs)
• Duplicated block (10 lines × 2) (crates/utopia-store/src/graph.rs)
• Duplicated block (11 lines × 2) (crates/utopia-server/src/api/oidc_routes.rs)
• Duplicated block (11 lines × 2) (crates/utopia-server/src/api/ontology_routes.rs)
• Duplicated block (11–12 lines × 2) (crates/utopia-store/src/export.rs)
• Duplicated block (12 lines × 2) (crates/utopia-extract/src/implication.rs)
• Duplicated block (12 lines × 2) (crates/utopia-store/src/jobs.rs)
• Duplicated block (13 lines × 2) (crates/utopia-server/src/main.rs)
• Duplicated block (13 lines × 2) (crates/utopia-store/src/documents.rs)
• Duplicated block (14 lines × 2) (crates/utopia-server/src/query_engine/mysql.rs)
• Duplicated block (14 lines × 2) (crates/utopia-store/src/graph.rs)
• …and 197 more

• 205 commits — 192 feature/other, 13 fixes

• crates/utopia-server — 59 commits
• crates/utopia-store — 52 commits
• web/src — 29 commits
• docs/decisions — 10 commits
• crates/utopia-extract — 9 commits
• crates/utopia-ingest — 8 commits
• docs/design — 8 commits
• scripts/bench — 7 commits
• (root) — 6 commits
• (repo) — 5 commits
• .github/workflows — 4 commits
• crates/utopia-llm — 4 commits
• .roadmap-proposals/backup-restore.md — 1 commit
• scripts/star-history.mjs — 1 commit
• web/DESIGN.md — 1 commit
• web/index.html — 1 commit

• fix: Fix Databricks schema read fallback and error handling (#702)
• fix: Fix GBK-encoded PDF text extraction (#761)
• fix: fix(auth): equalise login response time across unknown-email and bad-password branches (#867)
• fix: fix(chat): emit one earned terminal per turn (#903)
• fix: fix(export): keep preflight coverage aligned with the schema (#874)
• fix: fix(export): use one snapshot and validate local references (#833)
• fix: fix(rdf): resolve object_value text robustly; emit no rdf:object when literal can't be parsed (#831) (#865)
• fix: fix(server): align worker_concurrency fallbacks with migration 0011's default of 64 (#871)
• fix: fix(server): index mcp.md in the chat-searchable docs corpus (#905)
• fix: fix(store): accept supported expression operands in rule writes (#838)
• fix: fix(store): degenerate record-axis predicates to the single-sided for "now" (#847)
• fix: fix(store): enforce same-KB provenance references (#832)
• fix: fix(store): serialize vector index drops with builds (#908)
• change: (#755)
• change: 0041 records that a name is a claim about an entity (#663)
• change: 0044 makes the ontology a view over what documents say (#710)
• change: 0045 resolves a time mention against its document (#724)
• change: A NUL byte does not fail the whole document (#630)
• change: A backup does not carry the key that opens it (#803)
• change: A bound statement becomes a typed fact (#752)
• …and 185 more

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

deeplethe/utopia was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.