microsoft/ailab

This repository is a collection of diverse AI and machine learning demonstration projects, each serving as a self-contained lab or sample application. The codebase implements specific capabilities including image-to-story generation, sketch-to-HTML conversion, video background matting, and intelligent chatbots. It also includes cross-platform desktop tools for image snipping and analysis, alongside infrastructure for Azure-based cognitive services and search pipelines.

12 changes.

This period focused on the initial implementation and documentation of several AI-powered applications, including Sketch2Code for UI generation, Pix2Story for image-to-story conversion, and Snip-Insights for cross-platform desktop features. The work involved building core object detection pipelines, integrating Azure AI services, and establishing comprehensive test suites and cross-platform UI components.

7 changes.

This period focused on expanding the codebase with diverse feature sets, including a cognitive search demonstration, a Google Assistant bot sample, and machine teaching simulations. It also introduced significant multimedia capabilities, such as a UWP-based video player and a background matting pipeline for video processing.

The Sketch2Code project now includes a comprehensive README.md file that explains how the AI solution transforms handwritten UI designs into HTML markup. The documentation details the process flow, architecture (involving Custom Vision, Computer Vision, Azure Blob Storage, Azure Functions, and Azure Web Apps), and configuration steps for each component. Additionally, the Sketch2Code.sln solution file has been added, defining the project structure including the core, AI, API, and web components.

Sketch2Code · high confidence

Added two new machine teaching examples: a CNC machine calibration demo that trains a brain to adjust for friction-induced precision loss, and a motion control demo that trains a brain to guide a drill head through underground oil repositories. Each demo includes the necessary simulation code, Inkling models, training data, and configuration files to run locally or on the Bonsai server.

MachineTeaching · high confidence

Added the Google Assistant connector library (DirectLineToActionsOnGoogleLib) and a demo bot project (BotUsingCards) to the repository. The connector includes JavaScript modules for mapping Direct Line activities to Google Actions and handling fulfillment, along with deployment scripts and configuration files for local and development environments. The demo bot sample provides a C\# implementation using the Bot Framework that showcases rich card interactions, complete with VS Code launch and task configurations, a .bot configuration file, and a README with setup instructions.

GoogleAssistantConnector · high confidence

The frontend now includes a new HOCR (Hierarchical OCR) preview component that renders a graphical representation of a document's text extraction, displaying placeholders for each recognized word alongside the source image. This component supports zoom modes (page-full, page-width, original) and highlights target words. It is paired with a corresponding HOCR document component that renders the same content in a navigable text format, enabling users to hover over words in the text view to highlight them in the graphical preview, facilitating a proofreading workflow.

JFKFiles/frontend · high confidence

A new command-line utility named KinectMaskGenerator has been added to the VirtualStage directory. This tool processes Azure Kinect recordings to generate body mask images and corresponding timestamp files for video alpha channel creation. The implementation relies on the Azure Kinect SDK (v1.4.0) and the Azure Kinect Body Tracking SDK (v1.0.1) for body tracking, alongside OpenCV 4.3.0 for image processing. Users can specify input video paths, output directories, and time ranges to generate masks and optional color images.

VirtualStage/KinectMaskGenerator · high confidence

Introduced the core UI and logic for the snipping tool and image insights. On macOS, the app now uses native CoreGraphics APIs to capture screenshots from all active displays. The snipping workflow includes a GTK-based selection area and window for capturing and cropping images. Additionally, the app now features an Insights tab that analyzes images to identify celebrities and landmarks, displaying related information and news. The home screen has been updated with a tabbed interface to navigate between Insights, Library, and Settings.

(repo-wide) · high confidence

Introduced a new library feature allowing users to view, open, and delete saved snips. This includes the \LibraryPage\ and \LibraryViewModel\ in the shared Forms layer, which handle grouping images by date and managing the UI state. The GTK-specific implementation adds a \TopMenuWindow\ for the main toolbar, a \PopupWindow\ for settings toggles, and a \SnipInsightsTrayIcon\ for system tray access. Additionally, a \DrawedPath\ class was added to support drawing operations within the GTK rendering pipeline.

(repo-wide) · high confidence

The web application now includes a new routing configuration that maps URLs to specific controller actions, including 'details', 'ready-to-start', 'upload', 'work-in-progress', 'finished', 'generated-html', and 'camera' endpoints. Additionally, the project has been equipped with Application Insights telemetry for monitoring, global filter configurations for error handling and HTTPS enforcement, and the necessary CSS assets (including Bootstrap 4.1.1) to support the user interface.

Sketch2Code/Sketch2Code.Web · high confidence

Introduced the core components for an object detection feature, including the \ObjectDetector\ class for interfacing with Azure Custom Vision and Computer Vision APIs, and a \CustomVisionManager\ for managing project and tag operations. The \ObjectDetectionAppService\ orchestrates the detection workflow, while supporting classes like \PredictedObject\, \BoundingBox\, and \GroupBox\ model the detection results. The \Sketch2Code.Api\ exposes a \detection\ HTTP function that triggers this pipeline, and \ApplicationInsights.config\ is added to enable telemetry for the new service.

Sketch2Code/Sketch2Code.AI, Sketch2Code/Sketch2Code.Api, Sketch2Code/Sketch2Code.Core · high confidence

The Pix2Story application is introduced, enabling users to generate literary stories from images. The system processes input images through a VGG-19 convnet to extract features, which are then embedded into a joint space using a Visual Semantic Embedding (VSE) model. These embeddings are converted into captions via a skip-thoughts encoder, and finally, a GRU-based decoder generates a story conditioned on the image features and style biases. The codebase includes all necessary components for this pipeline, including Azure ML deployment configurations, model loading utilities, and text moderation services.

Pix2Story · high confidence

The BuildAnIntelligentBot lab has been updated to include a complete bot implementation using the Microsoft Bot Framework SDK V4. The changes introduce middleware for personality-based chat responses, automatic text and speech translation (converting user audio to English for processing and translating bot responses back to the user's language), and text-to-speech generation. The lab now provides the necessary resources (LUIS intents, QnA knowledge base, pronunciation guides) and C\# code (EchoBot, ReservationDialog, services) to build a restaurant assistant bot that handles natural language, speech, and multi-language interactions.

BuildAnIntelligentBot · high confidence

Added a new background matting system for the Virtual Stage, enabling users to process video files to extract foregrounds and generate composited outputs. The update includes a Python-based inference engine that utilizes a DeepLab segmentation model and a PyTorch-based neural network to generate alpha mattes, foregrounds, and composited video frames. The pipeline supports splitting videos by fixed thresholds, applying morphological operations to masks, and reconstructing final video files, effectively adding a complete workflow for background matting tasks.

VirtualStage/BackgroundMatting · high confidence

Added a new UWP-based video player control (VideosPlayer) and associated UI components (VideoButton, ChangeAttachedPropertyAction, TypeConverterHelper) for the Speaker.Recorder application. This introduces a multi-video playback interface with grid layout, visual state management, and media timeline synchronization. The UWP project structure includes App, Package manifest, and control libraries, enabling the application to handle video playback and recording workflows on Windows 10/11.

VirtualStage/Speaker.Recorder · high confidence

Added the complete source code and configuration for the JFK Files lab, including the JfkInitializer project to set up Azure Search resources (index, skillset, indexer), the JfkWebApiSkills project containing custom skills for OCR, image analysis, and cryptonym linking, along with JSON definitions for the search index and pipeline. This provides the necessary components to deploy and run the cognitive search demonstration.

JFKFiles · high confidence

The Snip-Insights project now includes the full source code and build infrastructure for a cross-platform desktop application. This adds Linux, Mac, and Windows installers (DEB, PKG, and ZIP/MSIX) generated via dedicated scripts, alongside the GTK\#-based UI implementation using Xamarin.Forms. The change also introduces a .gitignore file, a MIT LICENSE, a NuGet configuration, and a StyleCop configuration that disables documentation rules.

Snip-Insights · high confidence

The web application's user interface has been completely overhauled to improve the user experience across the entire workflow. The new design introduces a cleaner, more modern aesthetic with updated styling for the landing page, the upload and processing steps, and the final results page. Key changes include a streamlined layout for the 'Details' view that better showcases the original sketch, the predicted HTML, and the generated object details. The 'Index' page now features a more intuitive upload mechanism and a clearer explanation of the workflow steps. Additionally, the 'Step2' and 'Step3' views have been updated to provide better visual feedback during processing, and the 'Step5' (results) page now includes social sharing buttons for Facebook, Twitter, and LinkedIn, allowing users to easily share their generated HTML code.

Sketch2Code/Sketch2Code.Web/Views · high confidence

Added initial test suite for Sketch2Code; Added unit tests for insights, products, and text features.

Added or updated dependency configuration files across several projects: .csproj and packages.config files for C\# projects (ChatBot, CardsBot, JfkInitializer, JfkWebApiSkills, Sketch2Code.AI, Sketch2Code.Api, Sketch2Code.Core) specifying .NET and NuGet package versions; package.json and package-lock.json files for Node.js projects (DirectLineToActionsOnGoogleLib, GoogleAssistantProxy) specifying npm dependencies; and requirements.txt files for Python projects (Machine-Calibration, Motion-Control, Smart-Building) specifying Python package versions.

(dependencies) · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• CAI 43 → 37 (-6.3)
• Rubric changed (rubric-2026.08.19 → rubric-2026.09.15) — scores are not directly comparable.

• Code Health 68 → 64 (-3.8)
• Architecture 76 → 77 (+0.6)
• Maturity 58 → 57 (-1.6)
• Readiness 20 → 20 (+0.0)
• Security 69 → 41 (-27.5)

• (anonymous) (cognitive 23) (GoogleAssistantConnector/GoogleAssistant/DirectLineToActionsOnGoogleLib/lib/googleActions/RichMessageLimitationsManager.js)
• (anonymous) (cyclomatic 20) (GoogleAssistantConnector/GoogleAssistant/DirectLineToActionsOnGoogleLib/lib/googleActions/RichMessageLimitationsManager.js)
• Analyzed solution does not cover the bulk of the repository
• Build did not complete in the analyzer
• Duplicated block (10 lines × 2) (Snip-Insights/SnipInsight/Util/AppDiagnosticsLogger.cs)
• Duplicated block (8 lines × 2) (Snip-Insights/SnipInsight/StateMachine/StateMachine.cs)
• Duplicated block (9 lines × 2) (Snip-Insights/SnipInsight/Controls/Ariadne/AriModernWindow.cs)
• Duplicated block (9 lines × 2) (Snip-Insights/SnipInsight/Util/DpiUtilities.cs)
• High CVE: [GHSA redacted] (GoogleAssistantConnector/GoogleAssistant/DirectLineToActionsOnGoogleLib/package-lock.json)
• High CVE: [GHSA redacted] (GoogleAssistantConnector/GoogleAssistant/DirectLineToActionsOnGoogleLib/package-lock.json)
• High CVE: [GHSA redacted] (GoogleAssistantConnector/GoogleAssistant/DirectLineToActionsOnGoogleLib/package-lock.json)
• High CVE: [GHSA redacted] (GoogleAssistantConnector/GoogleAssistant/DirectLineToActionsOnGoogleLib/package-lock.json)
• High CVE: [GHSA redacted] (JFKFiles/JfkWebApiSkills/JfkInitializer/packages.config)
• High CVE: [GHSA redacted] (Sketch2Code/Sketch2Code.AI/packages.config)
• High CVE: [GHSA redacted] (GoogleAssistantConnector/GoogleAssistant/DirectLineToActionsOnGoogleLib/package-lock.json)
• High CVE: [GHSA redacted] (Sketch2Code/Sketch2Code.AI/packages.config)
• High CVE: [GHSA redacted] (GoogleAssistantConnector/GoogleAssistant/DirectLineToActionsOnGoogleLib/package-lock.json)
• High CVE: [GHSA redacted] (GoogleAssistantConnector/GoogleAssistant/DirectLineToActionsOnGoogleLib/package-lock.json)
• High CVE: [GHSA redacted] (GoogleAssistantConnector/GoogleAssistant/DirectLineToActionsOnGoogleLib/package-lock.json)
• High CVE: [GHSA redacted] (GoogleAssistantConnector/GoogleAssistant/DirectLineToActionsOnGoogleLib/package-lock.json)
• …and 44 more

• CarouselCardMapper._buildCarouselCard (cognitive 16) (GoogleAssistantConnector/GoogleAssistant/DirectLineToActionsOnGoogleLib/lib/mapper/CarouselCardMapper.js)
• CommentedOutCode (Snip-Insights/SnipInsight/Views/ToolWindow.xaml.cs)
• DeadPreprocessorBranch (Snip-Insights/SnipInsight/ImageCapture/ImageCaptureWindow.xaml.cs)
• Documentation: no installation or build instructions (README.md)
• Documentation: no usage examples (README.md)
• Duplicated block (11 lines × 2) (Snip-Insights/SnipInsight/Controls/Ariadne/AriModernWindow.cs)
• Duplicated block (12 lines × 2) (Snip-Insights/SnipInsight/AIServices/AIViewModels/ImageSearchViewModel.cs)
• Duplicated block (12 lines × 3) (Snip-Insights/SnipInsight/AIServices/AILogic/ContentModerationHandler.cs)
• Duplicated block (13 lines × 2) (Snip-Insights/SnipInsight/Util/AppDiagnosticsLogger.cs)
• Duplicated block (15–22 lines × 2) (Snip-Insights/SnipInsight/ImageCapture/DpiScalor.cs)
• Duplicated block (16 lines × 2) (Snip-Insights/SnipInsight/AIServices/AIModels/ImageSearchModel.cs)
• Duplicated block (22 lines × 2) (Snip-Insights/SnipInsight/Email/EmailManager.cs)
• Duplicated block (22 lines × 3) (Snip-Insights/SnipInsight/Email/EmailManager.cs)
• Duplicated block (8 lines × 4) (Snip-Insights/SnipInsight/AIServices/AIViewModels/OCRViewModel.cs)
• Duplicated block (9 lines × 2) (Snip-Insights/SnipInsight/StateMachine/StateMachine.cs)
• End-of-life runtime: .NET Framework net461
• End-of-life runtime: .NET netcoreapp2.0
• End-of-life runtime: .NET netcoreapp2.1
• End-of-life runtime: .NET netcoreapp2.2
• End-of-life runtime: .NET netcoreapp3.1
• …and 197 more

• Unchanged — 2 HTTP endpoints

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

microsoft/ailab was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.