Skip to content

architecture.md

Latest commit

 

History

History
214 lines (169 loc) · 39.2 KB

architecture.md

File metadata and controls

214 lines (169 loc) · 39.2 KB
Raw

🏗️ Architecture & Subsystem Deep Dive

← Back to README · Visual Diagrams Guide · Model Hub & Engines · MultiModal Studio

intel-arc-models-runner is engineered from the ground up for heterogeneous Intel silicon, coupling high-throughput matrix execution on Intel Arc discrete GPUs and Core Ultra iGPUs with ultra-low-power (5W) continuous token drafting on Intel AI Boost NPUs.

📐 Architecture at a Glance

flowchart TD
    subgraph ClientSubsystem ["Client & Tooling Layer"]
        CLI["arc-runner CLI (arc_runner.cli.main / Rich Terminal)"]
        WebUI["arc-runner Studio WebUI & Open WebUI (:11434)"]
        IDE["Cursor / Claude Code / LangChain / OpenAI SDK"]
    end

    subgraph DaemonSubsystem ["arc-runner Daemon (:11434)"]
        Router["Asynchronous HTTP Gateway (FastAPI / Uvicorn)"]
        
        subgraph APIRoutes ["Protocol Emulation Routes"]
            OllamaRoutes["routes_ollama.py (/api/generate, /api/chat, /api/tags)"]
            OpenAIRoutes["routes_openai.py (/v1/chat/completions, /v1/models)"]
            StudioRoutes["routes_studio.py (/studio, WebUI Static, SSE Status)"]
        end
        
        subgraph CoreSubsystems ["Core Execution & Resource Managers"]
            ModelMgr["ModelManager (Engine Cache, VRAM Budgeting, Dynamic Loading)"]
            BatchPipeline["BatchExecutionPipeline (Async Worker Loop, Queue, Client Abort)"]
            ContBatcher["ContinuousBatcher (Token-Level Step, Little's Law Admission Proof M30)"]
            PagedAttn["PagedBlockTable & RadixPrefixCache (Virtual Block Allocator Proof M31)"]
            LoRAMgr["LoRAManager (Dynamic Adapter Swapping, Safetensors Fused Linear)"]
        end

        Router --> OllamaRoutes & OpenAIRoutes & StudioRoutes
        OllamaRoutes & OpenAIRoutes --> BatchPipeline & ModelMgr
        BatchPipeline --> ContBatcher
        ContBatcher --> PagedAttn
    end

    subgraph HardwareExecutionEngine ["Hardware Execution Subsystem"]
        subgraph EngineAdapters ["Pluggable Backend Engines (arc_runner.engine)"]
            OVEngine["OpenVINOGenAIEngine (Unified Arc GPU / NPU / CPU Pipeline)"]
            SpeculativeEngine["HybridSpeculativeEngine (NPU Draft γ=4 + Arc Target; power not measured)"]
            VisionEngine["VisionEngine (Multimodal Vision Pipeline: Phi-3.5-V, MiniCPM-V)"]
            AudioEngine["AudioEngine (ASR Whisper & Kokoro TTS via Level Zero, Proof M59)"]
            ImageEngine["ImageEngine (Text2ImagePipeline: SD 1.5, SDXL, LCM, Turbo via Level Zero, Proof M73)"]
            VideoEngine["VideoEngine (Text2VideoPipeline: LTX-Video, CogVideoX, AnimateDiff, SVD via Level Zero, Proof M94)"]
            EmbeddingEngine["EmbeddingEngine (NPU / GPU Embedding Pipeline, L2 Normalized Proof M1)"]
            GGUFTranspiler["GGUFTranspiler & QuantizationMathEngine (Symmetric INT4/INT8 + OpenVINOGraphSynthesizer DAG)"]
            StabilityGuardEngine["StabilityGuard (Win32 P-Core Pinning 0x00FF & GC Burst Gate)"]
            FaithfulnessEngine["FaithfulnessGuard (KL Divergence <= 0.05 & Context Grounding Attribution)"]
            SiliconTelemetry["SiliconTelemetryLogger & BottleneckAnalyzer (Roofline Profiling & Auto-Tuning)"]
        end

        ContBatcher -->|forward_batch| OVEngine
        ModelMgr -->|get_or_load_engine| OVEngine & SpeculativeEngine & VisionEngine & AudioEngine & ImageEngine & VideoEngine & EmbeddingEngine
        ModelMgr -->|apply_adapter| LoRAMgr
        GGUFTranspiler -.->|transpiled IR| ModelMgr
        OVEngine --> StabilityGuardEngine & FaithfulnessEngine & SiliconTelemetry
    end

    subgraph SiliconLayer ["Physical Intel Silicon (Level Zero & OneAPI Drivers)"]
        ArcGPU["Intel Arc GPU (A770 / B580 / iGPU)\n[Level Zero Immediate Cmds, XMX Matrix Engines]"]
        IntelNPU["Intel NPU (3720 / 4000)\n[MCDM Driver Systolic Array, 5W TDP]"]
        IntelCPU["Intel Host CPU\n[AVX-512 / AMX Matrix Fallback]"]
        Sysman["LevelZeroSysman (Telemetry, VRAM Allocation, Thermal & Power)"]
        
        OVEngine --> ArcGPU & IntelNPU & IntelCPU
        SpeculativeEngine --> ArcGPU & IntelNPU
        VisionEngine --> ArcGPU & IntelCPU
        AudioEngine --> ArcGPU & IntelCPU
        ImageEngine --> ArcGPU & IntelCPU
        EmbeddingEngine --> IntelNPU & ArcGPU & IntelCPU
        Sysman -.-> ArcGPU
    end

    ClientSubsystem --> DaemonSubsystem
Loading

Native Layer, Ring Buffer and Telemetry (v0.2.0)

FlashStreamWeightManager --acquire(L)--> slot L % N  (EMPTY -> PENDING -> READY)
        |                                   one loader per slot, waiters share its result
        v
NativeMemoryEngine (bridge.py) --ctypes, ABI 3 + sidecar sha256--> libarc_native_sys
        |   _MappingGuard keeps the mapping alive while NumPy views exist
        v                                   arc_mmap_*: every extent checked with arc_range_ok
Python read path (identical bytes)  <-- used when the library is missing, disabled, stale or rejects a layer

NativeLevelZeroBridge --ctypes--> arc_ze_*  --dlopen--> ze_loader  (first ZE_DEVICE_TYPE_GPU, never the NPU;
                                                                    waits bounded below the 2 s Windows TDR)
  • Bounds and lifetime: the native layer never forms a pointer outside the mapped file (arc_range_ok, validated registration, sealed table); views are read-only and the unmap is deferred until the last view is collected. Hostile GGUF length fields are rejected by the parser.
  • Ring buffer: direct-mapped slots, a single loader per slot, no pin counts, safe shutdown. The protocol is modelled in docs/formal/RingSlots.tla and exhaustively checked by scripts/check_ring_model.py.
  • Telemetry: EngineTelemetry carries metrics_source, is_estimate and validity flags; unmeasured values are zero with the flag False. Sysman returns None plus a reason for power and free GPU memory on integrated GPUs.
  • Context window: the OpenVINO engine counts prompt tokens with the pipeline tokenizer and rejects prompt (+ max_new_tokens) beyond the window before generation.
  • Benchmarks: statistics and claim gates in arc_runner.benchmark.stats, schema-v1 result files, method in docs/benchmarks/METHOD.md.
  • Limits: the streaming path does not use the USM/DMA functions yet; the flash-stream engine is a demonstrator that emits canned tokens; no Level Zero driver was exercised in CI.

Diagrams of these parts are the "v0.2.0 native layer" sections at the end of each file in uml/ and the "Native layer (v0.2.0)" page of each file in drawio/; they are generated by python scripts/generate_diagrams.py.

📐 Visual Architecture & Diagrams

Comprehensive UML and editable Draw.io diagrams are maintained in docs/diagrams/, generated via connected MCP servers (uml-diagram and drawio-diagram) adhering to Rule 45-uml-diagram-lifecycle.md:

# Diagram Type Mermaid Source Editable Draw.io Web Editor Link
1 Package Diagram uml/package_diagram.md drawio/package_diagram.drawio Open in app.diagrams.net
2 Component Diagram uml/component_diagram.md drawio/component_diagram.drawio Open in app.diagrams.net
3 Class Diagram uml/class_diagram.md drawio/class_diagram.drawio Open in app.diagrams.net
4 Sequence Diagram uml/sequence_diagram.md drawio/sequence_diagram.drawio Open in app.diagrams.net
5 State Diagram uml/state_diagram.md drawio/state_diagram.drawio Open in app.diagrams.net
6 Activity Diagram uml/activity_diagram.md drawio/activity_diagram.drawio Open in app.diagrams.net
7 Deployment Diagram uml/deployment_diagram.md drawio/deployment_diagram.drawio Open in app.diagrams.net
8 Use Case Diagram uml/usecase_diagram.md drawio/usecase_diagram.drawio Open in app.diagrams.net
9 Object Diagram uml/object_diagram.md drawio/object_diagram.drawio Open in app.diagrams.net
10 Composite Structure Diagram uml/composite_diagram.md drawio/composite_diagram.drawio Open in app.diagrams.net
11 Interaction Diagram uml/interaction_diagram.md drawio/interaction_diagram.drawio Open in app.diagrams.net
12 Communication Diagram uml/communication_diagram.md drawio/communication_diagram.drawio Open in app.diagrams.net
13 Call Graph Diagram uml/call_graph_diagram.md drawio/call_graph_diagram.drawio Open in app.diagrams.net

📚 Comprehensive R&D & Architectural Reports

Before engineering the runtime, our multi-agent squad of 11+ specialist agents and 9 Opus Mathematical Masters from the Claude Global Library performed deep technical forensics and mathematical derivations:

Report Document ID Key Topics & Deliverables
1. Ollama Defects on Intel RND-OLLAMA-INTEL-01 Forensic deconstruction of Ollama's Go runtime (gpu/gpu.go), CGo dynamic loading failure, the "Silent CPU Fallback" anatomy, Level Zero collisions, and the NPU blindspot.
2. Intel Hardware & Drivers RND-INTEL-HARDWARE-02 Xe-HPG (Alchemist) vs. Xe2 (Battlemage) XMX matrix engines, Intel NPU 3720/4000 systolic arrays, Level Zero driver stack, and ecosystem fragmentation analysis.
3. Mission & Solution Blueprint RND-SOLUTION-BLUEPRINT-03 Dual-backend engine architecture (OpenVINO GenAI + llama.cpp SYCL), client-side hybrid speculative decoding, zero-setup Windows DLL loading, and API contracts.
4. R&D Synthesis & Proofs RND-SYNTHESIS-MANIFESTO-04 Multi-agent consensus, complete mathematical proofs (M1–M6: Rooflines, Speculative Acceleration, PagedAttention), NNCF INT4 quantization manifesto, and Triad Testing roadmap.
5. Next-Gen Multi-Modal Proofs RND-NEXTGEN-ORCHESTRATION-05 Formal mathematical proofs (M7–M12: YUV420p subsampling & QSV, Alpha cross-dissolve, Audio-First Reactive DAG, Dynamic karaoke word pacing, Dual-time constant audio ducking, and Closed-loop VLM inspection gate).

🎛️ Multi-LoRA Adapter Hot-Swapping on Intel Arc GPU

Engineered for enterprise deployments requiring rapid domain switching (e.g., Code, Medical, Legal, Multilingual) without reloading base model weights or re-allocating unified GPU memory:

  • Zero-Recompile Dynamic Weight Blending: Uses OpenVINO GenAI's native AdapterConfig binding to inject low-rank delta matrices $W = W_0 + \sum_{k=1}^K \lambda_k \frac{\alpha_k}{r_k} B_k A_k$ dynamically at per-request granularity.
  • Adapter swap without re-allocation: LoRA adapters are switched through a cached manager without re-allocating GPU Unified Shared Memory (USM). The swap overhead has not been measured; the tests are functional.
  • Bounded LRU adapter cache ($K_{\max} = 8$): at most 8 adapters are cached (DEFAULT_MAX_CACHED in the LoRA manager) with LRU eviction. Adapter sizes vary, so no memory bound in MB is claimed.
  • Web UI & CLI Studio: Blend adapters dynamically via glassmorphic sliders in the Chat Playground, or manage adapters via the CLI.

CLI Management

# List all discovered LoRA adapters
arc-runner adapter list

# Inspect low-rank adapter metadata and tensor shapes
arc-runner adapter inspect lora-coder

REST API Integration

Blend multiple adapters on the fly through /api/generate or /api/chat:

curl -X POST http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.2-3b",
    "prompt": "Write a secure AES-GCM encryption helper in Rust.",
    "stream": false,
    "adapters": [
      {"name": "lora-coder", "scale": 0.8},
      {"name": "lora-security", "scale": 0.2}
    ]
  }'

⚡ Continuous Batching & Multi-User KV-Cache Session Isolation

intel-arc-models-runner includes an asynchronous continuous batching execution pipeline and paged virtual memory manager engineered specifically for multi-user LLM serving on Intel Arc GPUs:

  • Token-Level Iteration Batching (Proof M29): Dynamically interleaves prefill and decode iterations across concurrent client sessions. Finished sequences are retired immediately upon EOS, eliminating Head-of-Line blocking.
  • Little's Law Admission Controller ($\theta_{\text{sat}} \le 0.92$, Proof M30): Protects discrete and integrated Arc VRAM against Out-Of-Memory (OOM) faults by admitting sequences only when physical memory block availability bounds are satisfied.
  • Multi-Tenant Session Isolation (Proof M34): Per-request virtual-to-physical block table mappings guarantee zero cross-session data leakage.
  • Radix Tree Prefix Deduplication (Proof M31): Shared system instructions and tool declarations are automatically deduplicated in physical blocks. The TTFT reduction this would give has not been measured.
  • Instant Client Disconnect Cleanup (Proof M32): Aborted client connections instantly reclaim all associated physical KV blocks back to the free pool.

Batch Telemetry & Administrative Preemption CLI

# View live continuous batching telemetry and VRAM block allocation
arc-runner batch status

# Administratively preempt an active sequence to relieve memory pressure
arc-runner batch preempt req-c3a7f809b1

REST API Parity Endpoints

# Retrieve real-time continuous batching and KV-cache telemetry
curl http://localhost:11434/api/batch/status

# Preempt an active sequence
curl -X POST http://localhost:11434/api/batch/preempt \
  -H "Content-Type: application/json" \
  -d '{"seq_id": "req-c3a7f809b1"}'