Evil0ctal/Douyin_TikTok_Download_API

This system is a self-hosted data extraction and archival tool for Douyin and TikTok, providing a unified API, CLI, and web console for managing content retrieval. It handles identity lifecycle management through browser-based minting and native signing, while isolating heavy media downloads into a separate Go-based service. The platform supports structured data storage with watchlists and collections, offering operators comprehensive tools for backup, diagnostics, and multi-agent integration via MCP.

The API now uses a structured application factory that mounts middleware for CORS, body limits, and security headers, and serves the built console SPA with language negotiation and proper cache headers. A uniform response envelope standardizes success and error payloads, while authentication dependencies enforce role-based access and scope checks, including a new demo role that restricts write operations to a specific allowlist. Public endpoints can be opened by operators without credentials, but admin and auth routes remain protected. Caller-supplied egress proxies are refused by default and only accepted when explicitly enabled with validation against private addresses.

src/dtk/api · high confidence

The console has been rebuilt for version 5.0.0, introducing a new shell layout with a collapsible sidebar and mobile drawer, and adding a suite of new pages including Library, Watchlist, Scheduler, Endpoint Access, Tools, Diagnose, Backup, and About. Key user-facing capabilities include a new Downloads page with batch parsing and author/author-feed modes, a Library for browsing stored media, and a Diagnose page for self-healing. The UI now supports full i18n with dynamic document titles, a demo role for public instances, and an 'About' page with an easter egg.

web/src/pages · high confidence

The database schema is extended with new tables and columns to support several new capabilities. A new \watchlist\ table allows users to schedule periodic re-collection of specific content, while \collections\ and \collection\_items\ tables enable manual grouping of archived posts. The \media\_downloads\ table gains a unique constraint to prevent concurrent downloads of the same post, and a new \availability\_checked\_at\ column is added to \archived\_contents\ to track when post existence was last verified. Additionally, the \users\ table now supports a 'demo' role, and the \tasks\ table includes an \is\_demo\ flag to manage retention for demo users, while \users\ and \api\_keys\ tables store encrypted demo credentials for public display.

src/dtk/db/migrations/versions · high confidence

This change introduces the core identity management infrastructure in src/dtk/identity. It adds robust cookie importing that automatically detects and parses four different paste formats (header, JSON, Netscape, loose) and presents a preview report to the user before saving. It establishes the IdentityPool for managing the lifecycle of identities (cookies, fingerprints, proxies) and tracks their health using database aggregates. Additionally, it implements a Redis-backed logging system (mint\_log) that allows the console to monitor the status of the identity refill job, including current minting activity and recent failures.

src/dtk/identity · high confidence

The API now exposes a structured set of HTTP endpoints for system health, console authentication, content retrieval, media downloads, and archive management, replacing the previous unregistered state. A key behavioral change is the introduction of bilingual OpenAPI documentation: the API now supports language negotiation via the \?lang=\ query parameter or \Accept-Language\ header, serving localized summaries and descriptions for endpoints and parameters instead of the previous concatenated English/Chinese text.

src/dtk/api/routes · high confidence

The application now includes an Alembic migration setup (\env.py\ and \script.py.mako\) specifically configured for TimescaleDB. This configuration ensures that database migrations run with \transaction\_per\_migration=True\ to support continuous aggregates, which cannot be created inside a transaction block. It also filters out TimescaleDB's internal schemas and objects from autogeneration to prevent accidental drops, and resolves the database URL from environment variables or configuration, defaulting to the \asyncpg\ driver.

src/dtk/db/migrations · high confidence

Added a new platform adapter for Douyin, enabling the system to fetch and parse Douyin content, author profiles, comments, and bookmark folders. The implementation includes endpoint definitions, query parameter construction, and response parsers, while explicitly noting that Douyin does not expose follower/following lists or collection details to guest identities.

src/dtk/platforms/douyin · high confidence

The system now supports the TikTok web platform, introducing a new adapter that binds endpoint definitions, parameter builders, and response parsers to the core framework. This addition enables fetching author profiles, posts, liked content, reposts, bookmark folders, and comments, while also providing a session-check capability to verify login validity. A critical fix ensures every API call includes a generated device\_id, which was previously missing and causing silent empty responses from TikTok.

src/dtk/platforms/tiktok · high confidence

This change introduces the foundational database layer for the application, including SQLAlchemy ORM models for relational tables (users, API keys, proxies, tasks) and TimescaleDB hypertables for time-series data (request logs, identity events, content snapshots). It also adds the corresponding async repositories for data access, Alembic migration utilities, and a complete internationalization (i18n) system supporting English and Chinese with locale-aware formatting for numbers and dates.

src/dtk/db · high confidence

Adds a new Model Context Protocol (MCP) server alongside the existing REST API and CLI, giving AI agents a direct, in-process way to query Douyin and TikTok data. The implementation mounts a streamable-HTTP endpoint at /mcp and supports a local stdio mode (python -m dtk.mcp), both sharing the same service layer and authentication rules as the REST API. It exposes a fixed set of eight tools—covering URL parsing, video and user details, post/comment listing, and pool status—while explicitly excluding any credential or proxy management to keep the agent surface secure and focused.

src/dtk/mcp · high confidence

The identity minting module now includes a client for communicating with a browser-RPC service, enabling the system to mint new guest identities and sign requests using a real browser session. This allows cookies and fingerprints to be generated within the same session context as the proxy's GeoIP, ensuring consistency in timezone, language, and screen settings. The client supports health checks, minting with proxy and geo-hint configurations, and signing operations that align with platform-specific algorithms, degrading gracefully if the service is unavailable.

src/dtk/identity/minting · high confidence

A new headless-browser RPC service has been added to handle identity minting and request signing. The service exposes endpoints to drive a real browser through an identity's proxy to obtain guest cookies and fingerprints, and to run the platform's own JavaScript to sign requests when the native algorithm is unavailable. It includes geo-alignment logic to match the browser's locale and timezone to the proxy exit, and manages warm signing contexts to ensure signatures are coherent with the identity's cookie jar.

_docker/browser\rpc · high confidence

This change establishes the foundational infrastructure for the application by adding the \src/dtk/core\ module. It introduces a two-layer configuration system that supports runtime hot-reloading of settings, a credential encryption module using AES-256-GCM, and an async SQLAlchemy database engine with session management. Additionally, it defines a stable error code contract with HTTP status mappings, structured logging with automatic sensitive data redaction, and a Redis client with atomic Lua script support. The module also provides shared enumerations for platform states, task outcomes, and user roles, forming the base layer for the rest of the system.

src/dtk/core · high confidence

This change establishes the core data contracts used by parsers, serializers, and the frontend by adding \src/dtk/models/content.py\ and its \\_\init\\_.py\. It defines normalized Pydantic models for \Author\, \Content\, \Media\, \Comment\, and \Collection\, ensuring platform-neutral metric names (e.g., \digg\_count\) and consistent identifier handling (e.g., using \sec\_uid\ for TikTok author lookups). The \Collection\ model specifically supports folder metadata including ownership and public/private status, while the \Author\ model resolves issues where platform-specific IDs (like TikTok's numeric ID) were unusable for subsequent API calls. These models enforce strict typing and preserve raw platform payloads for future metric back-computation.

src/dtk/models · high confidence

The browser RPC service now supports a pluggable backend architecture, allowing the browser automation layer to be swapped without rewriting the core service. A new registry in \backends/\_\init\\_.py\ manages backend selection, while \backends/base.py\ defines the strict \BrowserBackend\ protocol that all implementations must follow. Two backends are provided: \cloak.py\, which integrates the CloakBrowser library to provide realistic Chromium fingerprints and native signing capabilities for platforms like Douyin and TikTok, and \fake.py\, a lightweight in-memory stub for local development and CI testing that generates deterministic synthetic identities. This change ensures that the service can operate with real browser automation in production while remaining testable and debuggable in development environments.

_docker/browser\rpc/backends · high confidence

The \src/dtk/transport\ package now provides a dedicated HTTP transport layer that enforces strict fingerprint consistency and accurately distinguishes between business errors (such as deleted or unavailable content) and risk-control blocks. This prevents healthy identities from being incorrectly penalized or cooled when encountering non-existent posts or platform-specific refusals. The transport manages one client per identity to avoid connection-level fingerprint collisions, automatically maps browser fingerprints to compatible TLS emulation profiles, and constructs headers to match the identity's claimed browser environment.

src/dtk/transport · high confidence

This change adds the \dtk\ command-line interface, providing operators with a suite of tools to manage the self-hosted instance directly from the terminal. The CLI includes commands for backing up and restoring data, diagnosing system health, fetching and parsing content links, and managing the identity and proxy pools. It also supports administrative tasks such as creating and resetting console user passwords, importing and listing proxies, and modifying runtime configuration settings without requiring a restart. The tool is designed to handle incident response and routine maintenance, ensuring that critical operations remain accessible even if the web console is unavailable.

src/dtk/cli · high confidence

The signing module now supports two signing paths: a fast, pure-Python native signer that replicates platform algorithms (Douyin A-Bogus, TikTok X-Bogus) and a browser-rpc signer that executes the platform's own JavaScript. By default, requests are signed natively; the system automatically falls back to the browser signer when specific endpoints require session-bound values (like msToken or uifid) that the native signer cannot produce, or when shadow comparisons detect that the native algorithm has drifted from the platform's current behavior. This change eliminates the need for a browser for most requests, significantly reducing latency while maintaining reliability through automatic fallback and structural signature comparison.

src/dtk/signing · high confidence

The web application has been replaced by a new React Single Page Application (SPA) for the dtk v5 API, built to static files and served by the API container without a Node runtime in production. This new console implements the v5 design system, including a responsive layout that adapts to mobile devices, full English and Chinese (zh) internationalization, and a strict linting policy that enforces the use of design tokens for colors and translation keys for all user-visible text. The UI is built on a new component inventory featuring a \DataTable\ with sorting and density controls, a \CodeBlock\ for JSON payloads, and a \CopyableId\ component that displays full-length identifiers in monospace font. The application bootstraps theme and language preferences before the first paint and includes a setup gate that redirects users to an initialization page if the API is not yet configured.

web/src/components · high confidence

The signing subsystem now includes a native, pure-Python implementation for Douyin and TikTok signature algorithms (A-Bogus, X-Bogus, X-Dynosaur, X-Gnarly, and x-secsdk-web-signature), eliminating the need for a browser to generate these cryptographic parameters. This change introduces new modules in \src/dtk/signing/native\ that port the platform's bytecode-VM logic into standalone Python functions, allowing the \NativeSigner\ to handle all signing tasks locally. As a result, the system no longer relies on browser-rpc for signature generation, reducing overhead and improving reliability by removing the browser as a bottleneck for request preparation.

src/dtk/signing/native · high confidence

The \docker/\ directory now provides a complete, production-ready orchestration stack for single-host deployment. The main \Dockerfile\ builds a unified API/worker image using a three-stage process (Node for the React console, uv for Python dependencies, and a read-only Python runtime), while optional \Dockerfile.browser\ and \Dockerfile.downloader\ provide headless browser identity minting and a static Go-based media downloader, respectively. The \compose.yml\ file enforces strict security hardening—including non-root users, read-only root filesystems, dropped capabilities, and resource ceilings (e.g., 512MB memory, 256 PIDs for the API)—and configures internal networking for Postgres and Redis. A new \entrypoint.sh\ enforces the presence of a \DTK\_SECRET\_KEY\ at startup to ensure encrypted credentials, and the \README.md\ documents the quick-start workflow, including the requirement to clone the repository before running commands.

docker · high confidence

The \src/dtk/urls\ package introduces a centralized system for parsing, validating, and expanding Douyin and TikTok URLs. It enforces strict host allowlisting and SSRF protection by rejecting private or unlisted hosts, and it safely expands short links (e.g., \v.douyin.com\, \vm.tiktok.com\) through a redirect chain while re-validating every hop. The module also normalizes URLs by stripping tracking and signature parameters (such as \X-Bogus\, \X-Dynosaur\, and \X-Gnarly\) and validates post IDs against Snowflake timestamp constraints to reject malformed requests early.

src/dtk/urls · high confidence

The system now exposes a comprehensive administrative API under \/api/v1/admin\ to manage core resources and monitor system status. Administrators can now view and control which API endpoints are publicly accessible without credentials, manage the lifecycle of API keys (including creating keys with scoped permissions and revoking them), and handle demo credentials for public instances. Identity management has been expanded to allow importing, exporting, and inspecting identity cookies (with sensitive values masked for security), while proxy administration supports CRUD operations and bulk imports of proxy configurations. Additionally, a new health dashboard provides real-time circuit breaker status and success rates for endpoints, and a request log allows filtering recent traffic by endpoint, identity, or outcome to aid in debugging and auditing.

src/dtk/api/routes/admin · high confidence

The system now includes a dedicated background worker process that manages the lifecycle of queued tasks, ensuring no work is lost even if the process crashes. This worker handles identity pool management (minting and monitoring), proxy health probing, and comprehensive maintenance jobs such as data retention, stale task recovery, and media storage limits. It also provides a unified endpoint registry shared across the REST API, MCP, and CLI, and includes a URL parsing module to resolve short links securely through the proxy pool.

src/dtk/worker · high confidence

The worker now supports a dedicated set of maintenance operations—such as running diagnostics, creating backups, minting identities, testing proxies and notifications, and checking archive availability—that are submitted as background tasks rather than handled by the standard upstream endpoint registry. This separation prevents these long-running or administrative jobs from failing with 'unknown endpoint' errors and ensures they execute with their own session scope and dependency injection, providing operators with reliable, asynchronous access to system health and identity management tools from the console.

src/dtk/worker/ops · high confidence

A new Go-based downloader container has been introduced to handle media storage on the operator's own disk. This service exposes an internal HTTP API (listening on port 9100) that accepts download jobs from the main service, fetches media from allowed CDN mirrors, and writes them to a local volume. It enforces strict security measures, including host allowlisting, SSRF protection via dial guards, and safe path construction to prevent directory traversal. The service manages a bounded in-memory job queue, supports bulk file deletion, and provides health and status endpoints, ensuring that the heavy lifting of downloading and storing media is isolated from the main API to prevent overload.

docker/downloader · high confidence

The web application now includes a new core library in \web/src/lib\ and \web/src/hooks\. This introduces a typed API client (\api.ts\) that handles the server's response envelope, error codes, and automatic polling for asynchronous tasks. A suite of React hooks (\useApiQuery\, \useSession\, \useFocusTrap\, etc.) provides standardized data fetching, session management, and UI utilities. The library also adds a centralized endpoint path registry (\endpoints.ts\), locale-aware formatting (\format.ts\), and a language negotiation system (\language.ts\, \i18n.ts\) that supports English and Chinese.

web/src/lib · high confidence

This change introduces a new media storage capability that offloads file downloads to an optional Go sidecar container, allowing the operator to store media on their own disk. The Python module defines a strict allowlist of CDN domains for Douyin and TikTok, enforces file size ceilings and content-type policies, and manages the communication with the sidecar via HTTP. Additionally, exported files are now named using a deterministic format (prefix-platform-id-originalname) to ensure uniqueness and traceability when downloaded by users.

src/dtk/media · high confidence

This release introduces a comprehensive set of operational features designed for self-hosted instances, including backup and restore capabilities with encrypted credentials, a capacity guard that pauses background writers when disk space is low, and a six-step diagnostic suite for troubleshooting connectivity and signing issues. It also adds multi-channel alerting (webhooks, Telegram, SMTP, etc.) with deduplication, liveness and readiness health probes, and safe database query execution to prevent transaction aborts on optional feature checks.

src/dtk/ops · high confidence

The \src/dtk/services\ package introduces a comprehensive backend layer for managing media downloads, content archival, and user collections. It adds a Redis-backed response cache with in-flight request coalescing to reduce redundant upstream calls, and a persistent archive module that retains parsed content metadata (such as title, tags, and classification) beyond the standard task result expiration window. The update also implements a collections system for creating and managing named sets of archived posts, a demo mode for public instances that provisions inert, single-use credentials, and a robust download index that tracks media file lifecycles, handles eviction, and prevents concurrent download races.

src/dtk/services · high confidence

The install scripts (install.sh and install.zh.sh) have been expanded to detect existing installations and present a management menu instead of reinstalling. Users can now upgrade versions, manage passwords and backups, check status, and clean up disk space directly through the installer, which compares the running version against the latest GitHub release to determine upgrade availability.

install · high confidence

The scheduler module now manages identity selection using atomic Redis leases, per-endpoint rate-limiting policies, and a circuit breaker that trips when failures span multiple identities. Users benefit from more reliable rotation and quota enforcement: identities are picked based on health scores and least-recently-used ordering, while concurrent requests are serialized via SET NX locks to prevent over-granting. The system also supports named identity pinning, language-aware trip reasons, and configurable backoff after risk-control hits, ensuring that sensitive endpoints are throttled appropriately and that single-identity failures don't unnecessarily block the entire pool.

src/dtk/scheduler · high confidence

The \src/dtk/platforms\ module introduces a new platform adapter architecture, providing a registry for discovering platform-specific adapters and a base protocol for defining endpoint tables, request building, and response parsing. This change includes a critical behavioral fix for TikTok: the system now enforces a maximum page size of 35 items, preventing requests for larger pages (e.g., 36+) which the TikTok API refuses and would otherwise be misinterpreted as an empty result set. The module also adds common data normalization helpers and a client profile system to ensure request parameters align with browser fingerprints.

src/dtk/platforms · high confidence

The console now uses a dedicated design system with CSS tokens for colors, typography, and spacing, supporting both dark and light themes. This change fixes a usability issue where selected text was nearly invisible by defining explicit selection colors, and introduces consistent page rhythms and form layouts to prevent elements from running off-screen on mobile devices.

web/src/styles · high confidence

Added contract test scaffolding for live platform monitoring; Added full-stack smoke test script; Added i18n integrity check script to catch translation errors; Added replay tests for Douyin and TikTok parsers; Added test infrastructure with isolated policy registry fixture; Added unit tests for platform absence handling, API middleware, and console-API path consistency; Initial test suite for the browser-rpc service; Integration test suite for the application core.

The project has advanced to version 5.1.0, introducing new dependency manifests for the browser-rpc service (pinning FastAPI, uvicorn, and httpx) and the Go-based downloader (using only the standard library). The core Python application and web console have been updated with newer versions of key libraries, including FastAPI, uvicorn, SQLAlchemy, Redis, React, Vite, and i18next, ensuring the platform runs on the latest supported versions of these frameworks.

(dependencies) · high confidence

The dtk package has been updated to version 5.1.0, marking the latest release in the self-hosted Douyin/TikTok data API toolchain. This update reflects the current state of the foundation contracts and tooling established in recent commits.

src/dtk · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• First survey — no prior run to compare against. CAI 65.

• Code Health 62
• Architecture 75
• Maturity 73
• Readiness 65
• Security 71
• Domain Modelling 100
• Accessibility 66

• 210 commits — 151 feature/other, 59 fixes

• src/dtk — 83 commits
• web/src — 43 commits
• (root) — 30 commits
• (repo) — 10 commits
• tests/unit — 8 commits
• .github/workflows — 6 commits
• tests/integration — 6 commits
• documents/en — 5 commits
• web/package-lock.json — 5 commits
• docker/browser_rpc — 3 commits
• .github/RELEASE_v5.0.2.md — 2 commits
• install/install.sh — 2 commits
• .github/ISSUE_TEMPLATE — 1 commit
• .github/RELEASE_TEMPLATE.md — 1 commit
• .github/RELEASE_v5.1.0.md — 1 commit
• docker/Dockerfile.downloader — 1 commit
• docker/compose.yml — 1 commit
• install/README.md — 1 commit
• tests/contract — 1 commit

• fix: ci: scan all three languages, and fix the docs that still say main is v4
• fix: ci: stop two jobs failing for reasons that are not bugs
• fix: fix(api): a session's 403 listed scopes that could never have refused it
• fix: fix(console): an author download could only ever go to Douyin
• fix: fix(console): the downloads copy still said files are never served back
• fix: fix(console): the mobile navigation drawer, and one page that ran off the side
• fix: fix(db): CALL the columnstore policy procedure instead of SELECT
• fix: fix(i18n): translate the ?identity= parameter description
• fix: fix(install): an instance pinned to a sha build had nowhere to upgrade to
• fix: fix(install): upgrade to a release asked Docker Hub for a tag that never existed
• fix: fix(scheduler): SET NX for the lock, per-call tie-break, isolated policy registry
• fix: fix(scheduler): align endpoint names with the platform registry
• fix: fix(services): bridge the platform and transport request specs
• fix: fix(signing): read Douyin's msToken from the jar before inventing one
• fix: fix(signing): replace the guessed browser entry points with what the sites do
• fix: fix(tests): ruff E741 in the installer hygiene test
• fix: fix(tiktok): a profile link could not reach the author's posts
• fix: fix(tiktok): read the id from what fetch returns, not from what it parsed
• fix: fix(tiktok): refuse a page size TikTok will not serve
• fix: fix(tiktok): the handle lookup gave up whenever its answer was cached
• …and 190 more

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

Evil0ctal/Douyin_TikTok_Download_API was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.