joewongjc/type4me

Type4Me is a macOS voice input application that transcribes speech to text using a wide array of local and cloud-based ASR providers. It enhances the raw transcript through LLM-powered features such as text polishing, translation, and voice-controlled system actions. The system also supports a conversational 'Ask Anything' mode and allows users to revise text via voice commands, with usage analytics and subscription management integrated into the cloud infrastructure.

Introduces a new CppJiebaBridge component that provides a C-compatible interface for Chinese word segmentation using the CppJieba library. This bridge exposes functions to create and destroy segmentation handles, cut text into tokens (with optional search mode), insert user-defined words, and purge global caches. It vendors the CppJieba header-only library and a compact dictionary (dict.txt.small) along with HMM models, preserving their MIT licenses. The standard large dictionary is intentionally excluded to keep the footprint small.

CppJiebaBridge · high confidence

A new SwiftUI view, AudioVisualizer, has been added to the Type4MeUI module. This component displays a four-bar animation that pulses in response to audio input levels, using a combination of primary waves and harmonics for organic movement. It is designed to integrate with the system's Liquid Glass vibrancy by using the .primary color and includes preview support for development.

Type4MeUI · high confidence

Introduces a new Swift bridge (SherpaOnnxBridge.swift) that enables local Automatic Speech Recognition (ASR) and Voice Activity Detection (VAD) using the SherpaOnnx library. This change allows the application to perform speech-to-text processing locally on the device, supporting multiple model architectures including Paraformer, Zipformer, and Transducer, without requiring an internet connection.

Type4Me/Bridge · high confidence

Added a collection of static HTML/CSS demo pages for the Type4Me macOS voice input tool, showcasing various design themes including Art Deco, Brutalist, Editorial, Aurora, Capsule, Pulse, Frequency, Neocorp, Orchard, Retro, and Settings (Cards and Classic). These files serve as visual references for the product's landing page and interface styles.

website-demos · high confidence

Added Python scripts to the \scripts/\ directory to benchmark Large Language Model performance on voice polish and translation tasks. The suite includes \benchmark\_llm.py\ (comparing local Qwen3-4B vs. cloud Doubao), \benchmark\_4b\_vs\_9b.py\ (comparing Qwen3-4B vs. Qwen3.5-9B), \benchmark\_9b\_vs\_doubao.py\ (comparing Qwen3.5-9B vs. Doubao on longer samples), \benchmark\_gemini\_retest.py\ (testing Gemini prompt variations), and \benchmark\_llm\_quality.py\ (multi-model comparison via Ark and OpenRouter). These tools evaluate latency, output quality, and prompt effectiveness using samples from the app's recognition history.

scripts · high confidence

The ASR provider configuration layer has been significantly expanded to support a wider range of speech-to-text services. New providers added include Apple Speech, AssemblyAI, Baidu AI Cloud, Alibaba Cloud Bailian, Cartesia, Deepgram, ElevenLabs, Gemini, Grok (xAI), Meta Muse, Xiaomi MiMo, Sherpa (local SenseVoice), Soniox, and StepFun (both real-time and batch). Existing providers have also been updated: OpenAI now supports model selection (GPT-4o Transcribe, GPT-4o Mini, Whisper) and custom base URLs; Volcano supports both new API Key and legacy App ID authentication with auto-detection for model resources; and AWS, Alibaba Cloud, Azure, Custom, Google, iFLYTEK, and Tencent providers now have localized UI labels for their credential fields.

Type4Me/ASR/Providers · high confidence

The hotkey system now supports binding media keys (play, pause, volume, etc.) and mouse buttons as triggers, in addition to standard keyboard keys. It also improves modifier key handling by distinguishing between left and right side modifiers (e.g., left vs. right Command or Shift), ensuring more precise hotkey matching and preventing conflicts with system actions like Apple Music auto-launch. This allows users to create more flexible and context-aware hotkey bindings for voice input modes.

Type4Me/Input · high confidence

The database layer now supports detailed LLM usage analytics and a new Ask Anything conversation history. For LLM usage, the system tracks token counts, costs, and request status in a dedicated \llm\_usage\_history\ table, enabling users to view usage breakdowns by model, feature, and date, as well as recalculate historical costs. Additionally, a new \AskAnythingStore\ manages conversation sessions and turns, allowing users to review and interact with their Ask Anything history.

Type4Me/Database · high confidence

This change introduces the initial public release of Type4Me, a macOS voice input tool supporting dual-engine local ASR (SenseVoice via sherpa-onnx and Qwen3-ASR) and multi-provider cloud ASR. The release includes the full v2.10.0 changelog, a detailed development guide (AGENTS.md) outlining branch naming, build variants (pure vs. local), and ASR provider architecture, and an updated README with bilingual support, download links, and feature overviews. It also establishes the project's MIT license, entitlements for audio input and automation, and a .gitignore configuration that excludes build artifacts, local models, and agent context directories. The updates.json file provides metadata for the in-app updater, pointing to the latest v2.10.0 cloud and local DMG packages.

(repo-wide) · high confidence

This entry introduces the Type4Me macOS voice input tool. The app bundle includes an Info.plist defining the application identity and requesting microphone, speech recognition, and Apple Events permissions, along with a Jieba Chinese text segmentation dictionary and Hidden Markov Model to support Chinese language processing.

Type4Me/Resources · high confidence

Users can now use voice commands to control macOS system settings and applications through a new 'Mac Action' mode. This feature adds a registry of executable actions—including opening apps, managing window states (minimize, fullscreen, close), adjusting volume and brightness, toggling dark mode, taking screenshots, locking the screen, searching the web, checking battery status, and creating reminders—along with vocabulary management actions like opening settings, preparing snippets from selection, and adding hotwords. The system parses LLM tool calls to dispatch these actions safely via AppleScript and shell commands, providing immediate feedback in the floating bar.

Type4Me/Actions · high confidence

This change adds the initial CloudSubscription module, introducing a new account management UI (AccountTab, CloudSettingsCard) that supports email verification and anonymous username/password login. It implements the backend integration for cloud services, including a CloudAPIClient for REST calls, CloudAuthManager for session handling, and CloudQuotaManager for tracking free/paid usage limits. The module also adds CloudASRClient and CloudLLMClient to route voice recognition and LLM requests through the Type4Me Cloud proxy, with region-aware endpoint selection (CN/Overseas) and a debug diagnostics tab for latency testing.

Type4Me/CloudSubscription · high confidence

This change adds the foundational components for the Type4Me Revise feature, enabling the system to process voice or text instructions to edit content. It includes an instruction analyzer to detect intents (such as replace, delete, or rewrite) and constraints, a scope resolver to pinpoint specific text segments (like sentences or paragraphs), and an authorization resolver to safely handle explicit or implicit replacements. The update also introduces a diff calculator to track changes, a fact extractor to protect sensitive data like URLs and numbers, an output validator to enforce safety and formatting rules, and a prompt builder to structure requests for the model.

Type4MeReviseCore · high confidence

The Floating Bar now features a complete Liquid Glass visual overhaul. The recording indicator is a native Metal-rendered orb that reacts to speech energy, using an adaptive noise floor and dual-envelope system to drive fluid motion, pose morphing, and agitation effects across multiple color presets (e.g., Siri Ripple, Blue Crystal Drop). A new Liquid Glass Cancel Button provides a frosted glass appearance with liquid press-and-drag deformation physics. Additionally, the recording text overlay now includes an animated metallic specular sheen sweep that responds to audio energy, enhancing the visual feedback during transcription.

Type4Me/UI/FloatingBar/LiquidGlass · high confidence

This update introduces the backend infrastructure for LLM usage analytics and cost tracking, including a pricing registry that syncs rates from OpenRouter and a seed catalog for local models, alongside a usage recorder that logs token counts and costs per feature. It also adds a local Codex CLI client that reuses persistent App Server sessions for faster text transformations and an ASR variant generator that uses LLMs to suggest vocabulary corrections and hotwords.

Type4Me/LLM · high confidence

A new MenuBarControlCenter component has been added to the menu bar, providing a centralized hub for managing audio input devices, ASR/LLM providers, and system permissions. This feature introduces a \MicrophoneChoice\ enum to handle system-default versus specific device selection, displays batch vs. real-time status for ASR providers, and shows configuration status for LLM providers. It also monitors and reports permission issues (microphone and accessibility) and provides quick actions for copying the latest result or starting revision, ensuring the menu bar reflects the current state of the application's processing capabilities.

Type4Me/UI/MenuBar · high confidence

The floating bar now supports a compact recording style that displays a scrolling audio waveform history and a single-line live transcript that follows the text tail when it overflows. A new correction learning panel appears above the bar to let users accept or ignore transcription candidates with a 12-second auto-ignore timer. Additionally, a manual input panel allows users to type text directly with mode selection, and the floating bar now respects a light/dark recording theme for legibility across different host pages.

Type4Me/UI/FloatingBar · high confidence

The app now includes a dedicated 'Type4Me Permissions' window that appears during the setup flow, managed by a new PermissionGuideModel. Additionally, a RecordingControlCoordinator has been introduced to unify how recording actions (start, stop, follow-up) are handled, distinguishing between standard recording actions and those specific to the SelectionAsk feature. This change also adds environment objects for the app updater, menu bar control center, and navigation model to the main app structure, and introduces start gates to manage recording initiation logic more robustly.

Type4Me · high confidence

Type4Me now supports streaming speech-to-text from multiple new providers. The Protocol layer adds dedicated WebSocket clients for AssemblyAI, Baidu, Alibaba Cloud Bailian, Cartesia, Deepgram, ElevenLabs, Google Gemini Transcribe, xAI Grok, and Meta Muse. Each protocol handles its specific authentication, model selection, hotword/keyterm limits, and transcript event parsing, enabling users to select their preferred real-time transcription service directly within the app.

Type4Me/Protocol · high confidence

Type4Me adds support for eight new real-time speech-to-text providers, allowing users to transcribe audio via Apple Speech, AssemblyAI, Baidu, Alibaba Cloud Bailian, Cartesia Ink-2, Deepgram, ElevenLabs Scribe, and Google Gemini Transcribe. Each provider is implemented as a dedicated client (e.g., AppleASRClient, AssemblyAIASRClient) that manages its own WebSocket connection, handles authentication, and streams recognition events to the app.

Type4Me/ASR · high confidence

The app introduces a new TranslationLanguage model defining 18 supported languages with localized display names and stable prompt identifiers, alongside foundational services including AppBuildInfo for version display, AppDataLocation for separating user profile data from runtime state, AppUpdater for in-app updates, AskAnythingContextBuilder and Coordinator for managing conversation history and context truncation, AudioKeepAliveManager for maintaining microphone connections, BatchCorrectionInference for learning from user edits, ChineseWordSegmenter for text processing, CorrectionAffinityAnalyzer for evaluating correction confidence, CorrectionLearning for tracking and applying text corrections, and DataBackupManager for rotating local snapshots of user data.

Type4Me/Services · high confidence

Users can now view detailed usage statistics for speech recognition (ASR) and large language model (LLM) engines via new standalone views in the History settings. The ASR dashboard displays key performance indicators like audio duration, character count, and average speed, along with a breakdown of engine usage over time. The LLM dashboard provides token volume trends, estimated cost tracking with a syncable pricing table, and detailed breakdowns by model and feature, allowing users to monitor both aggregated usage and individual request history.

Type4Me/UI/Settings/History · high confidence

The Settings interface has been completely redesigned to improve usability and visual feedback. A new Appearance tab now features a live preview stage that renders the recording indicator and text output formatting in real-time as you adjust theme, style, and formatting options. The app also introduces an 'Ask Anything' page, allowing users to search, browse, and continue past voice-driven conversation sessions. Additionally, the settings now include a provenance explanation for history records, clarifying why a record's output differs from its raw recognition, and a new Debug tab provides build diagnostics and log access for troubleshooting.

Type4Me/UI/Settings · high confidence

The qwen3-asr-server location now provides a complete, standalone solution for local speech-to-text using the Qwen3 model. It includes a new FastAPI/uvicorn server (server.py) that exposes a WebSocket endpoint compatible with existing Swift clients, accepting PCM16-LE audio and returning JSON transcripts. To ensure stability on macOS, the server implements explicit MLX Metal cache management (capping GPU buffer growth to \~2GB) and a shared inference lock to prevent concurrent GPU access crashes. For distribution, a new PyInstaller build script (build.sh) and spec file (qwen3-asr-server.spec) are provided, which compile MLX from source with Metal JIT support to ensure compatibility across macOS 14+ versions and correctly bundle the required metallib for runtime discovery.

qwen3-asr-server · high confidence

Users can now choose from eight distinct start-sound styles (including Chime, Pluck, Submerge, Pong, and Water Drops) via the \tf\_startSound\ setting, replacing the previous single synthesized tone. The \SoundFeedback\ module has been refactored to use pre-warmed \AVAudioPlayer\ instances and cached PCM buffers, which eliminates frame-drop issues and ensures consistent audio playback timing for start, stop, and error events.

Type4Me/Session · high confidence

A change to existing behaviour in tools — 2 commits, 1 file.

tools · medium confidence · unverified

Users now have explicit control over whether dictated text remains on the system clipboard after completion or cancellation via the new Clipboard Output Policy settings (Always Copy, On Cancel: Processed, On Cancel: Raw Transcript, Never Copy). The injection engine has been refactored to use a snapshot-and-restore model that preserves the user's previous clipboard content, and internal clipboard writes are now marked as transient to prevent cluttering third-party clipboard history apps. Additionally, text injection now supports tracked replacements with verification to ensure text is correctly applied to the focused field, improving reliability in complex applications.

Type4Me/Injection · high confidence

The audio engine now supports explicit microphone selection via a new preference system, allowing users to prioritize specific input devices (such as external USB mics) over the system default. To prevent Bluetooth headsets from switching to low-bandwidth call profiles during background operations, the system deliberately avoids silent fallback to Bluetooth when a preferred device is unavailable. Additionally, recording start latency has been reduced by initializing the audio engine graph at launch rather than during capture, and the stop process now explicitly detaches audio graph nodes to ensure immediate release of Bluetooth headset resources.

Type4Me/Audio · high confidence

The Selection Ask panel now supports streaming Markdown responses with improved formatting, including soft line-break preservation and paragraph splitting for better readability. It also introduces conversation history management, allowing users to maintain context across multiple turns within the panel. The UI has been enhanced with a light recording theme, native frosted glass effects, and adaptive settings themes that respect the system appearance.

Type4Me/UI/SelectionAsk · high confidence

The floating recording bar now supports two layout modes—Regular and Compact—allowing users to choose a standard capsule or a smaller indicator. A new set of visual styles (such as Siri Ripple, Frost Fluid, and Liquid Chrome) replaces the previous defaults, with a power-saving static option available. The UI also introduces configurable settings to show or hide tooltips, the cancel button, and the finish button, while ensuring the bar remains legible over dark host pages via improved contrast floors for the native frosted glass effect.

Type4Me/UI · high confidence

The permission setup experience has been completely overhauled to guide users more intuitively through granting macOS access. A new floating drag overlay now appears pinned to the System Settings window, allowing users to drag the Type4Me app icon directly into the Accessibility list for authorization, replacing the previous manual navigation steps. The unified permission guide now clearly displays the status of Microphone, Accessibility, and (when Apple Speech ASR is selected) Speech Recognition permissions, providing specific actions to request access or open System Settings if previously denied. Additionally, the guide detects when Accessibility is granted but requires a relaunch for global hotkeys to activate, offering a one-click relaunch button to complete the setup.

Type4Me/Permissions · high confidence

The Type4Me setup experience has been restructured from a linear, multi-step wizard into a two-panel interface featuring a persistent sidebar for navigation and a main content area. The previous demo steps that illustrated Quick Mode and Custom Mode have been removed, simplifying the initial onboarding flow to focus on Welcome and Permissions. The wizard now supports theme switching and language selection within the sidebar, and the overall window dimensions have been adjusted to accommodate the new layout.

Type4Me/UI/Setup · high confidence

Added comprehensive test coverage for ASR provider registry and Ask Anything conversation features; Added vocabulary ASR accuracy test script; Introduces manual semantic evaluation suite for Type4MeIntelliSenseCore.

The main Package.swift now supports conditional compilation for local speech recognition (SherpaOnnx), cloud subscription features, and Chinese text processing (CppJiebaBridge), allowing the application to be built with or without these capabilities. It also exposes new library targets (Type4MeUI, Type4MeIntelliSenseCore, Type4MeReviseCore) and lowers the minimum macOS deployment target to 14. Additionally, a new Evaluation/IntelliSenseEval package is added to test the IntelliSense core, and the qwen3-asr-server directory now includes a requirements.txt specifying dependencies like mlx, fastapi, and uvicorn for the local ASR server.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 48 → 47 (-1.5)
• Rubric changed (rubric-2026.09.11 → rubric-2026.09.18) — scores are not directly comparable.

• Code Health 81 → 81 (-0.4)
• Architecture 99 → 93 (-5.2)
• Maturity 76 → 76 (-0.1)
• Readiness 58 → 43 (-15.4)
• Security 50 → 62 (+12.5)
• Accessibility 33 → 33 (+0.0)
• Performance 78 (new)

• Change coupling: LLMProvider.swift ↔ RecognitionSession.swift (Type4Me/LLM/LLMProvider.swift)
• Dependency hygiene PARTLY measured — SwiftPM pinning read, dependency currency NOT established
• Documentation: no architecture or design documentation (docs/features/meta-muse-asr/development-design.md)
• Documentation: no architecture or design documentation (docs/features/mimo-asr/development-design.md)
• Documentation: no architecture or design documentation (docs/features/translation/development-design.md)
• Documentation: no installation or build instructions (README.md)
• Duplicated block (11–15 lines × 2) (Type4Me/Database/AskAnythingStore.swift)
• Duplicated block (16 lines × 2) (Type4Me/LLM/DoubaoChatClient.swift)
• Duplicated block (16 lines × 2) (Type4Me/Services/IntelliSenseSettings.swift)
• Duplicated block (22 lines × 2) (Type4Me/Database/HistoryStore.swift)
• Duplicated block (5 lines × 2) (Type4Me/Services/SnippetStorage.swift)
• Duplicated block (6 lines × 2) (Type4Me/LLM/ClaudeChatClient.swift)
• Duplicated block (7 lines × 2) (Type4Me/Injection/ReviseAccessibilityClient.swift)
• Duplicated block (7–8 lines × 2) (Type4Me/UI/Settings/ModelProviderListView.swift)
• Duplicated block (8 lines × 2) (Type4Me/ASR/Providers/CartesiaASRConfig.swift)
• Duplicated block (8 lines × 2) (Type4Me/Database/HistoryStore.swift)
• Duplicated block (9 lines × 2) (Type4Me/LLM/DoubaoChatClient.swift)
• Duplicated block (9 lines × 4) (Type4Me/Services/BatchCorrectionInference.swift)
• Hotspot: Type4Me/ASR/SenseVoiceASRClient.swift (Type4Me/ASR/SenseVoiceASRClient.swift)
• Hotspot: Type4Me/Database/HistoryStore.swift (Type4Me/Database/HistoryStore.swift)
• …and 25 more

• Ambiguous method overloading with semantically similar but distinct input types. IntelliSenseRequest contains IntelliSensePromptInput as a subset (plus text), but the API exposes two separate build methods. This forces callers to choose between constructing a full Request or a lighter PromptInput, creating unnecessary cognitive load and potential for misuse if the distinction isn't clear.
• AskAnythingPage.conversationList (cognitive 19) (Type4Me/UI/Settings/AskAnythingPage.swift)
• ClassTooLong: LLMUsageAnalyticsView (Type4Me/UI/Settings/History/LLMUsageAnalyticsView.swift)
• ClassTooLong: TextInjectionEngine (Type4Me/Injection/TextInjectionEngine.swift)
• Coverage not measured — Swift suite
• Dependency hygiene PARTLY measured — Python dependencies read, no exact pin to grade for currency
• Documentation: contradicts the code (docs/archive/plans/2026-03-29-local-llm-integration-plan.md)
• Documentation: no architecture or design documentation (docs/README.md)
• Duplicated block (10 lines × 2) (Type4Me/LLM/ClaudeChatClient.swift)
• Duplicated block (10 lines × 2) (Type4Me/UI/SelectionAsk/SelectionAskView.swift)
• Duplicated block (10 lines × 2) (Type4Me/UI/Settings/IntelliSenseModeDetail.swift)
• Duplicated block (10 lines × 2) (Type4Me/UI/Settings/IntelliSenseModeDetail.swift)
• Duplicated block (10 lines × 2) (scripts/benchmark_4b_vs_9b.py)
• Duplicated block (10–11 lines × 3) (Type4Me/UI/Settings/HistoryTab.swift)
• Duplicated block (11 lines × 2) (Type4Me/Protocol/MiMoASRProtocol.swift)
• Duplicated block (11 lines × 2) (Type4Me/UI/FloatingBar/AudioVisualizer.swift)
• Duplicated block (11 lines × 2) (Type4Me/UI/Settings/HistoryTab.swift)
• Duplicated block (11 lines × 2) (Type4Me/UI/Settings/HomeDashboardView.swift)
• Duplicated block (11 lines × 2) (Type4Me/UI/Settings/SmartCorrectionSheet.swift)
• Duplicated block (11 lines × 2) (Type4Me/UI/Settings/VocabularyTab.swift)
• …and 164 more

• 27 commits — 16 feature/other, 11 fixes

• Type4Me/ASR — 6 commits
• Type4Me/LLM — 4 commits
• Type4Me/UI — 4 commits
• (root) — 3 commits
• Type4Me/Injection — 2 commits
• Type4Me/Resources — 2 commits
• Type4Me/Services — 2 commits
• docs/features — 2 commits
• Type4Me/Session — 1 commit
• Type4MeTests/AppDataLocationTests.swift — 1 commit

• fix: fix(asr): send Volcano inline hotwords in corpus.context (#324)
• fix: fix(revise): enforce tracking capture authorization on frontmost app and element owner (#322)
• fix: fix(revise): support output from every input mode (#316)
• fix: fix: avoid retrying cancelled empty streaming recordings (#311)
• fix: fix: explain why a history record's correction text differs from its output (#303)
• fix: fix: refresh Intelli Sense context before final processing (#309)
• fix: fix: render formatted release notes (#317)
• fix: fix: separate the shared user profile from per-app runtime state (#305)
• fix: fix: stack bad feedback metrics (#318)
• fix: fix: suppress Gemini hotword echoes on short recordings (#299)
• fix: refactor(llm): revert Requesty provider and formalize inclusion policy (#330)
• change: Localize TCC permission usage descriptions (English + zh-Hans) (#298)
• change: Update to 3.6 model version in AssemblyAIASRConfig (#328)
• change: feat(asr): support Qwen-Audio-3.1-ASR streaming models in Bailian (#323)
• change: feat(llm): add Requesty provider (#325)
• change: feat: add LLM and ASR usage analytics dashboard with cost estimation (#310)
• change: feat: add StepFun realtime region selection (#312)
• change: feat: add Xiaomi MiMo LLM provider (#320)
• change: feat: add processing engine submenu and native checkmarks to menu bar (#321)
• change: feat: align model provider icons with OpenRouter and enhance dark mode adaptation (#329)
• …and 7 more

• Unchanged — 0 containers · 1 contexts · 0 edges

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

joewongjc/type4me was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.