pingcap/tidb

This system is a comprehensive database backup and restore toolkit designed for TiDB, encompassing the BR (Backup/Restore) and Lightning utilities. It provides robust capabilities for full, incremental, and point-in-time recovery, supporting diverse storage backends like S3, GCS, and EBS, while ensuring data integrity through checksums and encryption. The system also includes performance benchmarking tools and a suite of static analysis linters to enforce code quality and security standards across the codebase.

105 changes.

This period focused on integrating Dumpling and the BR tool into the TiDB codebase, establishing their standalone module structures and build systems. It involved extensive refactoring of BR and Dumpling components, alongside the addition of comprehensive integration tests to ensure reliability and compatibility across various storage backends and edge cases.

56 changes.

This period focused on expanding the BR (Backup and Restore) module with new features such as online restore, EBS snapshot support, and log backup checkpointing, alongside comprehensive integration tests for various edge cases. Simultaneously, the project established a robust Bazel-based build infrastructure and a comprehensive Go linter framework to enforce code quality and security standards.

28 changes.

This period focused on restructuring the BR module by extracting distinct components like the import client, log client, and snapshot client into dedicated packages for better maintainability. It also introduced significant features for Point-in-Time Recovery, including checkpoint management, table filtering, and CRR progress tracking, alongside comprehensive integration tests to validate these new capabilities.

The BR Docker build environment is now defined with a multi-stage Dockerfile that uses Go 1.18.9-alpine and includes build tools for go-ycsb and minio client. Additionally, configuration files for PD, TiDB, and TiKV are added to the Docker image, along with environment variable files for GCS and Minio S3 storage, enabling users to run BR operations within a standardized Docker container.

br/docker · high confidence

A new \br/pkg/conn/util\ package has been introduced to centralize utility functions for interacting with the PD client. This includes helpers to retrieve all TiKV stores while handling TiFlash nodes, fetch the current timestamp from PD with retry logic, and retrieve configuration from TiKV stores via HTTP. The package also includes a Bazel build file defining the library and its dependencies.

br/pkg/conn/util · high confidence

The BR package now includes a new \logutil\ library that provides structured logging helpers for BR and Lightning components. This includes context-aware loggers, abbreviated array marshaling for large data sets, and rate tracing for metrics. The package adds support for serializing backup files, rewrite rules, regions, and SST metadata into structured log entries, along with a rate tracer for monitoring operational speeds.

br/pkg/logutil · high confidence

The BR package now includes a new \summary\ package that provides a centralized logging collector for backup and restore operations. This infrastructure tracks metrics such as total key-value pairs, data size, and duration, and outputs a structured summary log at the end of operations. The change introduces the \LogCollector\ interface and its implementation, along with corresponding unit tests to verify the collection and formatting of these summary logs.

br/pkg/summary · high confidence

The BR package now includes a new tracing implementation in \br/pkg/trace\, providing functions to start and finish OpenTracing spans for BR operations. This includes a \TracerStartSpan\ and \TracerFinishSpan\ API that captures trace data and writes it to a file. The change also introduces a \BUILD.bazel\ file to configure the Go library and test targets, and adds corresponding test files (\main\_test.go\, \tracing\_serial\_test.go\) to verify the tracing behavior.

br/pkg/trace · high confidence

The \tools\ directory now includes a comprehensive set of new files to support Bazel-based builds and CI checks. This includes \BUILD.bazel\ files for the \check\, \dashboard-linter\, \fake-oauth\, and \gen-parquet\ subdirectories, along with a suite of new shell scripts in \tools/check/\ (such as \check-bazel-prepare.sh\, \check-errdoc.sh\, and \failpoint-state.sh\) that automate various validation steps like checking file permissions, Go generate outputs, and Bazel file integrity. Additionally, a Go utility \ut.go\ and its associated build rules are introduced to manage unit test execution, while \tools/patch-go\ provides scripts to build a custom Go toolchain with specific runtime patches.

tools · high confidence

The mirror command is now buildable via Bazel, with new BUILD and Go source files defining the binary and library targets. This change enables the mirror tool to be built and run within the Bazel build system, supporting the project's move to Bazel-based builds.

cmd/mirror · high confidence

The BR module now includes a new CRR (Continuous Replication/Recovery) checkpoint service that exposes health and status endpoints (/livez, /readyz, /status) and publishes Prometheus metrics (e.g., service\_live, service\_ready, current\_round, safe\_checkpoint) to monitor the CRR checkpoint advancement loop. The service manages configuration via flags (task-name, retry-interval, calc.poll-interval, calc.meta-read-concurrency) and tracks state transitions (starting, running, degraded, stopped) and phases (idle, waiting upstream, advanced, etc.) for observability.

br/pkg/stream/crr/service · high confidence

The BR module introduces new configuration structures and parsing logic for EBS-based backup and restore operations. This includes the \EBSBasedBRMeta\ type and related structs (\EBSVolume\, \EBSStore\, \ClusterInfo\, \Kubernetes\, etc.) to manage metadata for EBS snapshots and volumes. The \config\ package now provides functions to parse TiKV configuration for import threads, region split size, and log backup settings, enabling the BR tool to handle EBS-specific metadata and configuration requirements.

br/pkg/config · high confidence

A new Go linter has been introduced in the build system to validate the consistency of database bootstrap and upgrade logic. The linter checks that system table definitions in \bootstrap.go\ and \upgrade\_def.go\ follow strict naming conventions (e.g., table IDs must end with 'TableID', SQL constants must start with 'Create' and end with 'Table', and names must match). It also ensures that the \versionedBootstrapSchema\ variable is defined exactly once and that all schema variables are correctly referenced, preventing errors when adding new system tables.

build/linter/bootstrap · high confidence

Introduces a new command-line utility, \benchdb\, designed to benchmark TiDB performance. The tool supports a variety of workload types including creating and truncating tables, inserting and updating rows (randomly or by range), selecting data, running custom SQL queries, and triggering garbage collection. It allows users to configure parameters such as table name, row size, batch size, and PD address to simulate different database operations and measure execution times.

cmd/benchdb · high confidence

A new benchmarking utility named 'benchkv' has been added to the codebase. This tool allows users to perform key-value store benchmarks by connecting to a TiKV cluster, executing concurrent read/write transactions, and exposing Prometheus metrics for monitoring performance.

cmd/benchkv · high confidence

A new Go linter named 'constructor' has been added to the build system. This tool enforces that structs are only instantiated via specific constructor functions, preventing direct struct creation or the use of 'new' in non-constructor contexts. The change includes the linter's implementation (analyzer.go), its Bazel build configuration (BUILD.bazel), and the associated unit tests (analyzer\_test.go).

build/linter/constructor · high confidence

A new Bazel build target and Go entry point have been added for the debug-linter tool. This change introduces the build/debug-linter/BUILD.bazel and main.go files, which configure the Go binary and library for the linter, depending on the bootstrap linter and Go analysis tools. This provides a dedicated build target for the debug linter, making it easier to build and run the linter for debugging purposes.

build/debug-linter · high confidence

BR now supports encrypting backup data using a master key. Users can configure encryption via a plaintext file or a cloud KMS (AWS, GCP) to secure backup files at rest. The new \br/pkg/encryption\ package provides the core logic for managing master keys and decrypting data during restore operations, ensuring that sensitive backup data remains protected.

br/pkg/task · high confidence

The codebase now includes a new gosec-based security linter. This adds a new analyzer that inspects source code for security problems, specifically enabling checks for SQL injection (G104), hardcoded credentials (G103), unsafe template usage (G101), and HTTP request body reading (G201). The linter is configured to run on the project's Go files, reporting issues with low severity and confidence by default, and can be skipped via configuration.

build/linter/gosec · high confidence

The importer tool is now available in the \cmd/importer\ directory, providing a command-line utility for generating and inserting test data into MySQL-compatible databases like TiDB. It supports various data generation rules including range, step, set, and incremental patterns, and can optionally load statistics from a JSON file to guide data distribution. The tool is built with Bazel, includes a configuration file (\config.toml\) and a README with usage examples, and comes with unit tests for its core logic.

cmd/importer · high confidence

The backup and restore (BR) tool now tracks and repairs foreign key constraints when restoring ingest-mode indexes. A new \ingestrec\ package records foreign key relationships and ensures that index repairs during restore correctly handle these constraints, preventing data integrity issues when restoring tables with foreign key dependencies.

br/pkg/restore/ingestrec · high confidence

A new linter has been added to the build system to check that the \AutoSyncInterval\ field is present when configuring the etcd client. This ensures that the \AutoSyncInterval\ is explicitly set, preventing potential issues with client synchronization.

build/linter/etcdconfig · high confidence

A new Go linter has been introduced to ensure that the Recover() function is called directly within a defer statement. This change adds a static analysis tool that flags any usage of Recover() that is not immediately wrapped by defer, helping developers avoid common concurrency and error-handling mistakes.

build/linter/deferrecover · high confidence

A new Go-based linter has been introduced to enforce that Go source files do not have the execute bit set. The checker inspects files during the build process and reports an error if any Go file is found with execute permissions, ensuring consistent file permissions across the codebase.

build/linter/filepermission · high confidence

A new linter has been introduced to detect and report spelling errors in the codebase. This change adds the 'misspell' analyzer, which scans Go source files for common misspellings and reports them as analysis errors. The implementation includes a Bazel build target and a Go analyzer that leverages the 'github.com/golangci/misspell' library to identify incorrect word usage.

build/linter/misspell · high confidence

Added a new \br/pkg/utils/iter\ package that provides a generic, context-aware iterator interface (\TryNextor\) and a suite of combinators for data processing. This includes functions for mapping (\Map\, \Transform\), filtering (\FilterOut\, \MapFilter\), joining (\ConcatAll\, \FlatMap\), and collecting (\CollectAll\, \CollectMany\) data streams. The implementation supports concurrent execution via configurable buffer sizes and worker pools, and includes utilities for enumeration, tapping, and error handling within the iteration pipeline.

br/pkg/utils/iter · high confidence

A new Go linter named 'printexpression' has been added to the build system. This linter detects and prevents the direct printing of expression objects using functions like fmt.Println, fmt.Printf, and fmt.Sprintf, as well as error generation methods. It ensures that types implementing StringWithCtx but not String are not passed directly to these functions, encouraging the use of explicit string conversion methods instead of relying on default formatting.

build/linter/printexpression · high confidence

A new \benchraw\ command has been added to the codebase, providing a benchmarking tool for the TiKV raw API. This utility allows users to run concurrent raw key-value put operations against a PD cluster, with configurable parameters for data count, worker concurrency, value size, and SSL/TLS configuration. The tool is built using Bazel and depends on the TiKV client-go library.

cmd/benchraw · high confidence

The \br/pkg/streamhelper/spans\ package is introduced, providing a \ValuedFull\ data structure backed by a B-tree to manage and merge overlapping key ranges. This implementation supports efficient range queries by value, enabling the BR (Backup/Restore) module to track and process spans with associated values, which is essential for log-backup checkpointing and GC safepoint coordination.

br/pkg/streamhelper/spans · high confidence

The build/linter/unconvert directory now contains the implementation and Bazel build rules for the 'unconvert' Go static analysis tool. This adds a new linting check that identifies and reports unnecessary type conversions in the codebase, helping developers clean up redundant casts.

build/linter/unconvert · high confidence

Added a new set of files in the br/compatibility directory to support testing BR (Backup and Restore) compatibility across different TiDB versions. This includes a Docker Compose configuration (backup\_cluster.yaml) to spin up a test cluster (PD, TiKV, TiDB, MinIO, GCS, and OAuth services), a shell script (prepare\_backup.sh) to orchestrate backup data preparation using YCSB workloads, and helper scripts (get\_last\_tags.sh) to fetch relevant release tags. These changes provide the necessary tooling to run compatibility tests for BR.

br/compatibility · high confidence

The BR tool now includes a dedicated Glue implementation (br/pkg/gluetikv) that allows BR to operate directly against TiKV without requiring a full TiDB server. This new package provides the necessary glue code to manage sessions, domains, and storage for backup and restore tasks, enabling a lighter-weight execution environment for TiKV clusters.

br/pkg/gluetikv · high confidence

New symlinks have been added to the .claude/skills/ directory, pointing to corresponding skills in .agents/skills/. These include tidb-bazel-prepare-gate, tidb-failpoint-test-runner, tidb-integrationtest-recorder, tidb-realtikv-runner, tidb-test-diff-triage, and tidb-verify-profile, making these specific agent capabilities available to the Claude agent.

.claude · high confidence

Introduced a new TiFlash recorder component in the backup and restore (BR) module to track TiFlash replica configurations during restore operations. This component records table IDs and their corresponding replica counts and location labels, then generates the necessary ALTER TABLE DDLs to reset or restore TiFlash replica settings after a restore. The change includes the core implementation and associated unit tests.

br/pkg/restore/tiflashrec · high confidence

The \br/pkg/stream/crr/internal/checkpoint\ package was added, introducing a \Calculator\ that computes a downstream-safe checkpoint for Change Data Replication (CRR). This component tracks the replication progress of backup metadata files, waits for all referenced files to be synced, and advances the \syncedTS\ only after confirming full replication. The implementation includes logic to handle empty metadata files, prune removed stores, and persist state across restarts, with comprehensive unit and randomized integration tests to verify correctness.

br/pkg/stream/crr/internal · high confidence

A new mock ID allocator has been introduced in the br/pkg/mock/mockid package, providing a simple, thread-safe ID generation mechanism for testing purposes. The implementation includes a Go source file defining an IDAllocator struct that uses atomic operations to generate sequential IDs, along with a Bazel build configuration to integrate it into the build system.

br/pkg/mock/mockid · high confidence

A new pre-commit hook has been added to the repository to automatically check Go files for formatting issues using gofmt and spelling errors using goword. This ensures that all committed Go code adheres to standard formatting and spelling conventions before being merged, with the hook aborting the commit if any issues are found.

hooks · high confidence

The build system now includes the \prealloc\ Go linter, which identifies slice declarations that could be preallocated for better performance. Additionally, new utility functions were added to \build/linter/util\ to support linter configuration, specifically enabling the exclusion of files based on \only\_files\ and \exclude\_files\ settings, allowing developers to control which files are analyzed by specific linters.

build/linter/util · high confidence

The restore module now includes a new \rawkv\ package that provides a \RawkvClient\ interface and a \RawKVBatchClient\ for batching raw key-value writes to TiKV. This implementation handles deduplication of entries and metrics collection for batch put operations, supporting the restore workflow with a configurable batch size and column family support.

br/pkg/restore/internal/rawkv · high confidence

A new \storewatch\ package has been added to the BR utility layer, providing a \Watcher\ that monitors the lifetime of TiKV stores. The implementation tracks store registration, disconnection, and reboot events, allowing consumers to register callbacks for these specific lifecycle changes.

br/pkg/utils/storewatch · high confidence

The BR (Backup and Restore) module now manages GC safe points for individual keyspace scopes, not just globally. The \br/pkg/gc\ package introduces a \Manager\ interface with two implementations: \globalManager\, which uses the deprecated \UpdateServiceGCSafePoint\ API for backward compatibility, and \keyspaceManager\, which uses the new \SetGCBarrier\ API. This allows BR to protect data in specific keyspace during backup and restore operations, ensuring that garbage collection does not remove data needed for recovery.

br/pkg/gc · high confidence

The backup implementation has been moved to a new \br/pkg/backup\ package, which now contains the core backup client, schema handling, store interaction, and concurrency limiting logic. This change includes the addition of a \BUILD.bazel\ file for the new package, along with corresponding test files (\client\_test.go\, \limit\_test.go\, \schema\_test.go\, \store\_test.go\, etc.) that verify the backup client's behavior, resource limiting, and schema backup processes.

br/pkg/backup · high confidence

The BR checkpoint implementation has been consolidated into a new \br/pkg/checkpoint\ package, introducing a unified \CheckpointRunner\ that manages backup and restore progress. This refactoring separates the checkpoint logic for backup, snapshot restore, and log restore into distinct files (\backup.go\, \restore.go\, \log\_restore.go\) while sharing core infrastructure like \checkpoint.go\ and \storage.go\. The change also includes a new \manager.go\ that provides interfaces for managing checkpoint metadata in both external storage and temporary database tables, enabling more robust and testable checkpoint handling for TiDB's backup and restore workflows.

br/pkg/checkpoint · high confidence

The BR CLI has been restructured, introducing new subcommands such as 'abort' for canceling restore tasks, 'operator' for managing migrations and checksums, and 'stream' for log backup control. The tool now implements a memory limit calculation to cap BR's memory usage based on available system memory, and includes a leak test to ensure global memory arbitrators are cleaned up properly after tests.

br/cmd · high confidence

The \br debug\ command now supports decoding encrypted backup metadata and statistics files into readable JSON format. This new functionality, implemented in the \br/pkg/metautil\ package, allows users to inspect the contents of backup files for debugging and verification purposes. The update includes new \DecodeMetaFile\ and \DecodeStatsFile\ functions that handle decryption, checksum validation, and conversion of protobuf-based backup metadata and statistics into JSON files, along with comprehensive unit tests for these operations.

br/pkg/metautil · high confidence

The BR tool now uses a new \br/pkg/glue\ package to manage console output, progress bars, and interactive prompts. This change introduces a \ConsoleOperations\ interface and associated implementations that provide colorful, formatted text output, a \Progress\ interface backed by the \mpb\ library for better progress visualization, and utilities for handling terminal interactivity. This refactors how BR communicates status and collects user input, replacing previous ad-hoc printing and progress handling with a structured, testable glue layer.

br/pkg/glue · high confidence

The BR tool now utilizes a new \gluetidb\ package to manage the TiDB domain and sessions, introducing a \brInfoSchemaFilter\ to control which databases are loaded during backup and restore operations. This change isolates BR's internal state from the main TiDB server, ensuring that BR operations do not interfere with the server's schema state. A corresponding mock implementation (\gluetidb/mock\) is also added to support testing of BR's glue interface.

br/pkg/gluetidb · high confidence

The BR directory now includes a comprehensive set of new configuration and documentation files, including .editorconfig, .gitattributes, .gitignore, .golangci.yml, and revive.toml, alongside standard project files like LICENSE.md, CONTRIBUTING.md, and README.md. This establishes the foundational structure for the BR module, providing guidelines for code style, linting, and contribution workflows.

br · high confidence

The BR restore process now accepts an 'online' flag that, when enabled, skips switching TiKV to import mode. This change introduces the \ImportModeSwitcher\ component and its associated tests, allowing users to perform restores without disrupting the cluster's normal operation, which is essential for online restore scenarios.

br/pkg/restore · high confidence

The BR stream package has been restructured to introduce a new \br/pkg/operation\ package that provides an \OperationContext\ for tracking BR command executions with unique IDs, timestamps, and hint fields. Additionally, the \br/pkg/stream/backupmetas\ package was added to parse tagged and legacy backupmeta file names, extracting metadata such as flush timestamps, store IDs, and time ranges. The \br/pkg/stream\ package now includes refactored logic for rewriting metadata keys and values, decoding KV entries, and searching log data files, all supported by new Bazel build files and comprehensive unit tests.

br/pkg/stream · high confidence

The BR (Backup & Restore) utility functions have been consolidated into the new \br/pkg/utils\ package. This refactoring introduces a structured set of Go files including \backoff.go\ for retry strategies, \error\_handling.go\ for standardized error classification, \db.go\ for database configuration management, and \filter.go\ for table filtering logic. The change also adds corresponding unit tests for these utilities, improving code organization and maintainability for the BR tool.

br/pkg/utils · high confidence

The Dumpling command-line interface is now built as a Go binary via Bazel, with a new \main.go\ entry point that parses flags, registers Prometheus metrics collectors, and invokes the export logic. A \versions.go\ file exposes version details (release version, git hash, build timestamp, Go version) for both console output and structured logging. Corresponding \BUILD.bazel\ files define the \cli\ and \dumpling\ Go targets, enabling Bazel-based builds for the Dumpling tool.

dumpling/cmd · high confidence

The Dumpling project has been merged into the TiDB repository as a distinct module. This change introduces the core Dumpling codebase, including its README, an \OWNERS\ file for project governance, a \revive.toml\ for Go linting configuration, and an \install.sh\ script that supports both amd64 and arm64 architectures for downloading dependencies like MinIO and Zstd.

dumpling · high confidence

The BR module now includes a new \br/pkg/aws\ package that implements EBS snapshot creation and management using the AWS SDK for Go v2, enabling the backup and restore workflows to interact with AWS EC2 and CloudWatch services. Additionally, a \br/pkg/common\ package has been added to hold shared constants, specifically \MaxStoreConcurrency\, which defines the limit for TiKV store connections. These changes provide the underlying infrastructure for EBS-based backup and restore operations.

br/pkg/aws · high confidence

The BR (Backup/Restore) module now includes a dedicated \br/pkg/errors\ package that defines a comprehensive set of typed errors for backup, restore, PD, KV, and PiTR operations. This change introduces helper functions like \Is\ and \IsContextCanceled\ to standardize error checking and context handling within the BR tooling.

br/pkg/errors · high confidence

The build system now includes a new Bazel-based build configuration (build/BUILD.bazel) that defines a 'nogo' target integrating a wide array of Go static analysis tools, including staticcheck, revive, errcheck, and many others. Additionally, the 'build/linter' directory now contains the Go implementations for these linters, each structured as a separate Go library with its own BUILD.bazel file. This change establishes the foundation for enforcing code quality and style rules across the codebase via the Bazel build system.

build · high confidence

The \br/pkg/streamhelper/daemon\ package now provides a generic \OwnerDaemon\ wrapper that manages the lifecycle of background tasks in a TiDB cluster. It automatically handles etcd leader election, ensuring that the daemon's \OnStart\, \OnBecomeOwner\, and periodic \OnTick\ hooks are executed only when the current node is the owner. This refactors the previous log-backup implementation into a reusable pattern for other owner-dependent services.

br/pkg/streamhelper/daemon · high confidence

The \br/pkg/streamhelper\ package has been restructured to support 'checkpoint v3', a new mechanism for advancing the global checkpoint of log backup tasks. This change introduces a \CheckpointAdvancer\ that scans regions to collect checkpoints from TiKV, uploads the global checkpoint to PD, and manages subscribers for flush events. The implementation includes new files such as \advancer.go\, \advancer\_cliext.go\, \advancer\_daemon.go\, and \collector.go\, along with a dedicated \config\ subpackage to manage tick intervals, backoff times, and checkpoint lag limits. This refactoring stabilizes the log backup process by decoupling lock resolution from the GC worker and providing a more robust way to track and advance the global checkpoint.

br/pkg/streamhelper · high confidence

The tidb-server command now includes a new external workload controller framework, enabling integration with external workload management systems. Additionally, a FIPS-compliant build target is now available for the tidb-server binary, allowing users to build a version of TiDB that adheres to FIPS (Federal Information Processing Standards) requirements. The build system (BUILD.bazel) and associated test files have been updated to support these new capabilities.

cmd/tidb-server · high confidence

Added a new \br/pkg/backup/prepare\_snap\ package that implements the logic for preparing snapshot backups. This includes the \Preparer\ state machine, environment interfaces for connecting to stores, and stream handling for lease management. The addition introduces a structured approach to managing the pre-snapshot state, including retry logic, connection handling, and event-driven processing of region states.

_br/pkg/backup/prepare\snap · high confidence

Added the \cmd/pluginpkg\ package, a new command-line tool that automates the process of building TiDB plugins. The tool reads a \manifest.toml\ file from a plugin's source directory, generates Go code using a template, and compiles the plugin into a \.so\ shared object file. It supports optional Profile-Guided Optimization (PGO) via a \--pgo-file\ flag and can include the \nextgen\ build tag via a \--next-gen\ flag.

cmd/pluginpkg · high confidence

The build system now includes a new Go library at build/linter/revive that wraps the revive static analysis tool. This adds a stricter, configurable linter to the codebase, enabling checks such as unused parameters, unnecessary statements, and context key types. The integration is defined in a new BUILD.bazel file and an analyzer.go implementation that configures and runs specific revive rules, reporting results as JSON-formatted failures.

build/linter/revive · high confidence

Adds a new 'registry' package to the BR tool that provides persistent tracking of restore tasks via a 'tidb\_restore\_registry' table in the 'mysql' database. This system supports managing task lifecycles (running, paused, resetting) and includes a 'HeartbeatManager' that periodically updates task status to help identify and handle stalled or stale tasks.

br/pkg/registry · high confidence

The BR operator package now includes new commands for managing storage and backup operations. Users can now test external storage connectivity and performance with the new \test-storage\ command. Additionally, the \checksum-table\ and \force-flush\ commands allow for table checksumming and log flushing, while \list-migration\ and \migrate-to\ commands enable managing and executing log backup migrations. These changes provide new capabilities for storage validation, data integrity verification, and migration management within the BR toolset.

br/pkg/task/operator · high confidence

The \br/pkg/restore/utils\ package was introduced to centralize utilities for the backup and restore process. It includes \common.go\ defining the \CreatedTable\ struct, \merge.go\ implementing \MergeAndRewriteFileRanges\ to consolidate small backup ranges and reduce region splits during restore, and \rewrite\_rule.go\ providing \RewriteRules\ for mapping old table/index IDs to new ones. The addition of these files, along with their corresponding tests (\merge\_test.go\, \misc\_test.go\, \rewrite\_rule\_test.go\), enables more efficient handling of compacted SST files and table rewrites during the restore operation.

br/pkg/restore/utils · high confidence

Added new CI build images (base, centos7\_jenkins, parser\_test) and a Bazel CI configuration file to the build/image directory. The base image now includes Go 1.25.12, Rust, Node.js 18, Java 17, Gradle 7.4.2, Bazel 7.7.1, and Oras 1.2.0. The centos7\_jenkins image is based on a custom Go 1.25 image and includes Jenkins user setup. The parser\_test image is based on Rocky Linux 9 with Go 1.25.12 and MySQL 8.x dependencies.

build/image · high confidence

The BR restore process now pre-allocates table IDs before restoring tables, which improves restore performance by batching ID allocation. The new \br/pkg/restore/internal/prealloc\_table\_id\ package handles collecting table and partition IDs, validating them against a threshold, and managing the allocation state, including support for reusing IDs from a previous checkpoint.

_br/pkg/restore/internal/prealloc\_table\id · high confidence

The database now supports MySQL dual passwords, allowing users to manage password rotation and legacy password states. This change affects the executor, privilege, and test modules to implement the new behavior.

(repo-wide) · high confidence

The \column\ package, which previously handled column metadata and value casting (including \CastValue\ and type description generation), has been removed from the codebase. This change eliminates the legacy column handling logic, likely as part of a broader refactoring to a new type system.

column · high confidence

The legacy \domain\ package, which previously managed the storage space, information schema, and DDL instance, has been removed from the codebase. This change eliminates the old \Domain\ struct and its associated test suite, reflecting a broader architectural shift in how the database core manages state and initialization.

domain · high confidence

The legacy \table\ package, which defined the \Table\ interface and the \Ident\ struct, has been removed. Additionally, the \table/tables\ package, which provided the concrete \Table\ struct and its test suite, has been deleted. This change eliminates the old table abstraction layer in favor of a newer, unified table implementation.

table · high confidence

The \util/charset\, \util/codec\, \util/errors\, \util/errors2\, \util/format\, \util/mock\, and \util/prefix\_helper\ packages have been removed from the codebase. This cleanup eliminates legacy code for character set management, key encoding, error handling, and mock contexts, reflecting a broader refactoring to consolidate or replace these utilities with newer implementations.

util · high confidence

The local store engine interface and its BoltDB and GoLevelDB implementations have been removed from the codebase. This eliminates the ability to use local file-based storage backends for transactions and snapshots, as the associated driver, storage, and test files are deleted.

store · high confidence

The field package, which previously provided types and functions for parsing column names and managing result fields, has been removed from the codebase. This includes the deletion of field.go, result\_field.go, and their associated test files, eliminating the ability to interact with the field module for SQL parsing and column metadata handling.

field · high confidence

The rset layer, which previously provided an abstract recordset interface and various plan implementations (such as for distinct, fields, group by, having, join, limit, and order by), has been removed. This change eliminates the redundant rset abstraction layer, simplifying the codebase by removing these specific plan and recordset files and their associated tests.

rset · high confidence

The \interpreter\ directory, which contained the standalone TiDB command-line interface (CLI) for executing SQL statements interactively, has been removed. This includes the deletion of \interpreter/main.go\ and \interpreter/.gitignore\. Users relying on this specific CLI tool will need to use alternative methods to connect to TiDB, as this standalone entry point is no longer available.

interpreter · high confidence

The coldef package, which previously defined types such as ReferenceDef and related column definition structures, has been removed from the parser. This change eliminates the package and its associated code, simplifying the parser's internal structure and dependencies.

parser · high confidence

The \sessionctx/db\ package, which provided helper functions to bind and retrieve the current database schema from the context, has been removed. This cleanup eliminates legacy code that is no longer required for the current session management architecture.

sessionctx · high confidence

The \kv\ package has removed the legacy in-memory key-value store (\memkv\), the \IndexIter\ and \UnionStore\ implementations, and related iterator utilities. This cleanup eliminates the use of the \github.com/syndtr/goleveldb\ library for in-memory storage and simplifies the KV interface by removing the \UnionStore\ and \IndexIterator\ abstractions that were previously used for buffering and merging dirty and snapshot data.

kv · high confidence

The legacy \InfoSchema\ interface and its associated \Handle\ struct in the \infoschema\ package have been removed. This change eliminates the old in-memory schema cache implementation and its corresponding unit tests, reflecting a broader architectural shift away from the previous schema management approach.

infoschema · high confidence

The legacy expression implementation files have been removed from the codebase, including \between.go\, \binop.go\, \builtin.go\, \builtin\_control.go\, \builtin\_groupby.go\, \builtin\_math.go\, \builtin\_string.go\, \builtin\_time.go\, and their corresponding test files. This cleanup removes the old expression evaluation logic, replacing it with a newer, unified expression framework.

expression · high confidence

The \meta\ and \meta/autoid\ packages have been removed from the codebase. This eliminates the legacy auto-increment ID generation and metadata key helpers that were previously used for schema and table ID allocation. Users relying on these internal utilities will need to use the current \autoid\ or \meta\ implementations for ID generation and metadata operations.

meta · high confidence

The \mysqldef\ package and all its associated files (including \charset.go\, \const.go\, \decimal.go\, \errcode.go\, \errname.go\, \error.go\, \error\_test.go\, \fsp.go\, and \state.go\) have been removed from the codebase. This eliminates the legacy MySQL definition and error handling utilities in favor of the current implementation.

mysqldef · high confidence

The legacy plan implementation, including the \plan\ package and the \plan/plans\ subpackage, has been removed. This includes the old \Plan\ interface and all associated plan types such as \TableDefaultPlan\, \GroupByDefaultPlan\, and \HavingPlan\. Users relying on the old plan structure will need to use the new plan implementation.

plan · high confidence

The \stmt\ package and the \stmts\ sub-package, which contained the old statement execution layer (including \stmt.go\ and files for DDL, DML, and other SQL statements), have been removed. This change eliminates the previous execution model, requiring users to rely on the new plan and executor architecture for all SQL operations.

stmt · high confidence

The model package, which previously defined core schema structures such as ColumnInfo, TableInfo, IndexInfo, and DBInfo, has been removed from the codebase. This change eliminates the local definitions for database, table, and index metadata, likely as part of a broader refactoring to consolidate or migrate these types to a different package or repository.

model · high confidence

The legacy DDL implementation files (alter.go, ddl.go, ddl\_test.go) have been removed from the codebase. These files contained the old, unused DDL logic and its associated tests, which are no longer needed.

ddl · high confidence

The BR connection management implementation (conn.go), its tests (conn\_test.go, main\_test.go), and the associated Bazel build file (BUILD.bazel) have been moved to the br/pkg/conn directory. This relocation organizes the codebase by placing BR-specific connection handling logic into its own package, making the structure clearer for users and developers working with backup and restore operations.

br/pkg/conn · high confidence

The BR log restore logic has been extracted from the main br/pkg/restore package into a new br/pkg/restore/log\_client package. This refactoring introduces a dedicated LogClient and supporting components (such as LogRestoreManager, SstRestoreManager, and various processors) to handle log backup and restore operations. The change improves code organization and maintainability by separating log-specific restore logic from the broader restore package.

_br/pkg/restore/log\client · high confidence

The BR snapshot restore client has been moved into a new \br/pkg/restore/snap\_client\ package, separating snapshot restore logic from the main restore client. This change includes the new \SnapClient\ implementation, associated test files, and a Bazel build configuration, providing a cleaner separation of concerns for snapshot-based restore operations.

_br/pkg/restore/snap\client · high confidence

The region split and scatter logic for BR restore has been extracted into a new, shared \br/pkg/restore/split\ package. This refactoring unifies the split and scatter handling for both BR and Lightning, allowing the two components to share the same core splitting, waiting, and scattering logic. The new package includes the \SplitClient\ interface, the \RegionSplitter\ executor, and associated test files, providing a common foundation for region management during restore operations.

br/pkg/restore/split · high confidence

The dumpling/export package has been restructured and integrated into the TiDB codebase, moving from a standalone tool to a core package under pkg/. This includes a new Bazel build file (BUILD.bazel) defining the export library and its tests, along with the associated source files (config.go, conn.go, consistency.go, dump.go, etc.). The refactoring includes moving related packages (like objstore and dumpformat) to the pkg/ directory, updating import paths, and ensuring the export module is properly wired into the TiDB build system.

dumpling/export · high confidence

The BR restore logic for importing data to TiKV has been refactored by extracting the import client into a new internal package (br/pkg/restore/internal/import\_client). This change isolates the gRPC client implementation and its associated tests, improving code organization and maintainability for the backup and restore workflow.

_br/pkg/restore/internal/import\client · high confidence

The repository now includes license files for the QL, STRUTIL, and Unicode-DFS-2016 dependencies. This ensures proper attribution and compliance with the licensing terms for these third-party components.

LICENSES · high confidence

The BR checksum executor now properly handles context cancellation, ensuring that checksum operations are aborted if the context is done, preventing incomplete or stale results. Additionally, the executor builder tracks the request source and resource group name, enabling better observability and resource management for checksum operations.

br/pkg/checksum · high confidence

The BR restore client has been separated into a new \br/pkg/restore/internal/prealloc\_db\ package. This change introduces a dedicated \DB\ struct and associated methods (such as \ExecDDL\, \CreateDatabase\, and \CreatePlacementPolicy\) to handle database and table creation during restore operations. A corresponding test suite (\db\_test.go\) has been added to verify the behavior of these restore operations, including auto-increment ID handling and sequence restoration.

_br/pkg/restore/internal/prealloc\db · high confidence

The BR restore data module has been refactored to separate the restore client and adjust the file directory structure. This change introduces a new \data\ package under \br/pkg/restore/data\ containing core recovery logic, including functions for sorting recover regions, checking consistency and valid peers, and selecting region leaders. The \data.go\ file implements the main \RecoverData\ function which orchestrates the recovery process, while \recover.go\ provides helper functions for region selection and validation. The \key.go\ file contains utility functions for key comparison and prefixing. Tests have been added to verify the new recovery logic.

br/pkg/restore/data · medium confidence

The BR module now enforces stricter version compatibility checks for TiKV and TiFlash nodes during backup and restore operations. Specifically, TiFlash versions are validated against minimum compatible versions (3.1.0 for major version 3, 4.0.0 for major version 4), and TiKV versions are checked against minimum requirements for features like Point-in-Time Recovery (PiTR) and DDL execution. The \CheckClusterVersion\ function in \br/pkg/version\ now iterates through all cluster stores, distinguishing between TiKV and TiFlash nodes to apply the appropriate version constraints, ensuring that incompatible node versions are rejected before operations proceed.

br/pkg/version · high confidence

The BR tool now uses a dedicated build package to manage and log version details (release version, git hash, build time, Go version, race status, and kernel type). This change ensures that both the BR and Lightning binaries log consistent version information at startup, and adds tests to verify the output format.

br/pkg/version/build · high confidence

The br/pkg/pdutil package has been restructured, moving source files (pd.go, utils.go) and tests (main\_test.go, pd\_serial\_test.go) into the new pkg directory. The test infrastructure has been migrated from the pingcap/check framework to testify, and a Bazel build file (BUILD.bazel) has been added to support the new package layout.

br/pkg/pdutil · high confidence

The Dumpling logging subsystem now performs pre-checks for log file permissions, returning a clear error if the log file cannot be created due to permission issues or if a directory is used as a log file. Additionally, the system will automatically create any missing parent directories for the log file path. These changes are supported by new tests that verify error handling for permission-denied scenarios and directory creation.

dumpling/log · high confidence

This release delivers a broad set of fixes and improvements to the query optimizer and execution engine. Key changes include fixing several optimizer bugs that could lead to incorrect results or panics, such as issues with outer-to-inner join conversion, subquery handling, and plan cache correctness. The planner has been refactored to support the new Cascades optimizer framework, improving performance for complex queries. Additionally, various stability issues in the executor, including data races in hash joins and CTEs, have been resolved. The statistics module has also been updated to handle DDL events and async loading more robustly, reducing the chance of 'Information schema is out of date' errors.

pkg · medium confidence

A new static analysis check, 'toomanytests', has been added to the build system to prevent packages from containing an excessive number of test cases. The linter enforces a default limit of 50 tests per package, with specific exceptions allowing 210 tests for 'pkg/planner/core' and 90 for 'pkg/util/topsql/reporter'. This change helps maintain test suite manageability by flagging packages that exceed these thresholds.

build/linter/toomanytests · high confidence

The build system now runs the \revive\ linter across all Go files via a new \allrevive\ analyzer. This replaces the previous linting approach with a stricter, configurable tool that enforces a specific set of rules (such as \early-return\, \range-val-in-closure\, and \unconditional-recursion\) and outputs results in JSON format. This change improves code quality by catching more issues and providing more detailed, structured feedback to developers.

build/linter/allrevive · high confidence

The TiDB Lightning tool and its checkpoint management logic have been moved out of the \br/\ directory into a new \lightning/\ directory. This change includes the \tidb-lightning\ and \tidb-lightning-ctl\ command-line tools, their associated Go source files, and the \checkpoints\ package which manages import progress. The move is accompanied by updated build files (\BUILD.bazel\), test files, and a new \.gitignore\ for the directory. This reorganization separates Lightning's codebase from the backup/restore (BR) module, reflecting its status as a distinct component within the TiDB ecosystem.

lightning · high confidence

The project removed several vendored third-party Go packages from the Godeps/\_workspace directory, specifically github.com/djimenez/iconv-go and github.com/docopt/docopt-go, along with their associated test files, examples, and metadata. This reduces the local dependency footprint by eliminating these specific libraries from the vendored workspace.

_Godeps/\workspace · high confidence

The build system was updated to support Bazel by adding a BUILD file for patches and modifying the gRPC-ecosystem/grpc-gateway patch to align with Bazel build rules. Additionally, the errcheck tool was patched to support an 'excludes' flag for specifying excluded functions, and the gRPC dependency was upgraded.

build/patches · medium confidence

Add BR PITR integration tests for index repair and data integrity; Add BR debug meta test suite for backup/restore validation; Add BR full backup and restore integration test; Add DDL test coverage for PITR online filter; Add consistency test for Dumpling data export; Add integration test for full cluster restore with privilege and system table handling; Add integration tests for Dumpling TLS support; Add mock local backend for testing; Add multi-valued index backup and restore test; Add test for dumping specified tables and views; Add test suite for handling special characters in Dumpling; Add tests for the BR \SHOW BACKUP METADATA\ command; Added BR TTL compatibility test; Added BR charset GBK test suite; Added BR incremental restore and index tests; Added BR integration test for foreign key constraints; Added BR integration tests for TiDB configuration and edge cases; Added BR integration tests for backup and restore scenarios; Added BR key-locked test harness; Added BR region label rule tests; Added BR split region failure test; Added BR table partitioning test suite; Added BR test for GC safe point handling; Added BR test scripts for case sensitivity, clustered indexes, online restore, and range-based backups; Added BR test suites for backup and restore scenarios; Added DDL integration tests for column and index operations; Added EBS-based BR test suite; Added Go-based integration test for BR transactional backup and restore; Added S3 integration test suite for Dumpling; Added automated tests for PITR table filter functionality; Added backup version validation test; Added chaos testing for Dumpling; Added end-to-end test for foreign key handling; Added end-to-end tests for Dumpling CSV export and Lightning import; Added end-to-end tests for Dumpling's Parquet and compression features; Added external storage test for Dumpling; Added integration test for Azure Blob Storage backup and restore; Added integration test for BR Point-in-Time Recovery checkpoint handling; Added integration test for BR S3 backup and restore; Added integration test for BR backup lock file and PD config reset; Added integration test for BR encryption; Added integration test for BR file corruption scenarios; Added integration test for BR log backup task lifecycle; Added integration test for BR restore with TiFlash synchronization; Added integration test for BR system table handling; Added integration test for Dumpling row-based splitting; Added integration test for Dumpling's --sql flag; Added integration test for PITR and accelerate indexing compatibility; Added integration test for extreme integer values and progress logging; Added integration test for incremental DDL restore; Added integration test for partitioned table index addition; Added integration test harness and documentation for Dumpling; Added integration tests for BR backup and restore operations; Added integration tests for BR backup and restore scenarios; Added integration tests for BR backup during TiKV outages; Added integration tests for BR encryption and decryption; Added integration tests for BR raw key-value store backup and restore; Added integration tests for BR shuffle leader and region schedulers; Added integration tests for BR with Google Cloud Storage; Added integration tests for Dumpling; Added integration tests for Dumpling placement policy support; Added integration tests for Dumpling with generated columns; Added integration tests for Dumpling's core dump scenarios; Added integration tests for TiDB placement policy handling in BR; Added integration tests for empty database and null unique index scenarios; Added integration tests for full DDL and index restore scenarios; Added mock infrastructure for BR and Lightning tests; Added test coverage for BR cached table backup and restore; Added test coverage for BR duplicate table handling; Added test coverage for BR replica read functionality; Added test coverage for partitioned tables in Dumpling; Added test coverage for quoted identifiers and view handling in Dumpling; Added test coverage for sequence dump functionality; Added test data for the constructor linter; Added test for BR restore of NONCLUSTERED AUTO\_ID\_CACHE=1 tables; Added test for Dumpling connection timeout handling; Added test for long-running schema loading during PITR; Added test for non-existent table error handling; Added test script for small table backup and restore operations; Added test utilities for log backup and CRR simulation; Added test utility scripts for Dumpling; Added tests for BR restore behavior with new collation settings; Migrate BR rtree test infrastructure to testify; Stabilize integration tests and fix flaky test cases.

Updated Go dependencies, including golang.org/x/mod, golang.org/x/tools, github.com/prometheus/prometheus, github.com/golangci/golangci-lint, github.com/prometheus/client\_golang, pebble, golang.org/x/crypto, golang.org/x/net, and golang.org/x/oauth2.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• First survey — no prior run to compare against. CAI 66.

• Code Health 61
• Architecture 99
• Maturity 84
• Readiness 73
• Security 66
• Domain Modelling 64
• Event-Driven 100

• 300 commits — 261 feature/other, 39 fixes

• pkg/planner — 33 commits
• br/pkg — 32 commits
• pkg/executor — 30 commits
• pkg/ddl — 24 commits
• (root) — 21 commits
• pkg/dxf — 21 commits
• pkg/server — 17 commits
• pkg/util — 13 commits
• tests/realtikvtest — 10 commits
• pkg/objstore — 8 commits
• pkg/session — 8 commits
• pkg/domain — 6 commits
• pkg/parser — 6 commits
• tests/integrationtest — 6 commits
• pkg/expression — 5 commits
• pkg/statistics — 5 commits
• pkg/sessionctx — 4 commits
• pkg/store — 4 commits
• dumpling/export — 3 commits
• lightning/pkg — 3 commits

• fix: *: bump go-m1cpu to v0.2.1 to fix SIGSEGV on newer Apple Silicon (#68896)
• fix: *: fix parser parentheses depth dos (#70190)
• fix: *: fix uncompress mem dos (#70199)
• fix: br: fix PiTR meta KV OOM by copying entries and deduplicating mDB keys in ReadFilteredEntriesFromFiles (#67268)
• fix: br: pre-lease log-backup storage and readLock leak fixes (#67850)
• fix: ddl: fix alter index visibility suffix collision (#70056)
• fix: ddl: fix missing streaming for multi-schema add-index (#68558)
• fix: executor, store/copr: fix ANALYZE failure against an old peer during upgrade (#69300)
• fix: executor: fix RU v2 executor accounting (#68578)
• fix: executor: fix slow log push down bug (#68765)
• fix: expression: fix NO_BACKSLASH_ESCAPES in generated columns (#67743)
• fix: expression: fix isBinCollation to recognize utf8mb4_0900_bin (#68846)
• fix: expression: fix unary minus flen inference (#69088)
• fix: importer: fix table_import to look at real table size instead of compressed size (#68652)
• fix: importinto: fix cancel race before DXF task commit (#69968)
• fix: importinto: fix conflict row identity and MVI deduplication (#70076)
• fix: lightning: fix write limit test timing flake (#68916)
• fix: metrics: fix resource control failed query instance filter (#69103)
• fix: pkg/ddl, planner: revert "add TiKV space precheck for DXF add-index" (#69658)
• fix: pkg/executor: fix concurrent SessionVars.systems map access during ANALYZE (#68465)
• …and 280 more

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

pingcap/tidb was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.