This document maps the LocalAssist implementation to the product claims in the project brief.
| Claim | Implementation |
|---|---|
| On-device structured daily brief and task suggestions | FoundationModelsSummarizer streams LanguageModelSession.streamResponse(to:generating: DailyBrief.self). The @Generable/@Guide contract in GeneratedSummary.swift uses constrained decoding for headline, key points, and tasks with optional ISO-8601 due dates — there is no malformed-output repair path. SummaryNormalizer enforces app-level semantics (dedupe, caps, stable IDs, action inference). |
| Works without network access | LocalAssistService falls back to DeterministicFallbackGenerator when the model is unavailable (deviceNotEligible, appleIntelligenceNotEnabled, modelNotReady, no adapter, forced offline) or when generation fails (guardrailViolation, refusal, context overflow, decoding, transient rate/concurrency errors). Diagnostics preserve the exact reason while the app keeps working offline. |
| Guided generation | Native framework guided generation: @Generable structs and enums with @Guide constraints (.count(3...5), .maximumCount(5)), includeSchemaInPrompt: false on repeat session turns to save prompt tokens; a content example in the instructions was tried instead and reverted after the on-device model copied its tasks into real briefs. |
| Tool calling | Three Foundation Models Tool conformances in LocalAssistSystemTools, each behind a provider seam with a live EventKit/Contacts implementation and a static one for tests: CalendarAvailabilityTool (free/busy through EventKitFreeBusyProvider), RemindersLookupTool (read-only open reminders through EventKitReminderProvider, so the model checks what already exists before proposing a duplicate), and ContactsLookupTool (name resolution through ContactsFrameworkResolver). System access is requested on the first tool call; denials surface as typed tool errors, never crashes. SampleAgendaStore and the static providers back previews/screenshots via LocalAssistToolkit.sampleTools. Invocations are counted into run diagnostics. |
| Executable actions with confirmation | SystemActionExecutor writes real EKReminders and EKEvent holds after explicit user confirmation, resolving natural-language due hints with the deterministic DueDateParser. RecordingWriteStore keeps executor logic unit-testable. |
| Session lifecycle & latency | One LanguageModelSession is reused across online turns, prewarm() runs when online mode is enabled, and overlapping requests get a one-off session instead of a concurrentRequests failure. |
| Context-window management | ConversationMemory keeps a rolling window of compressed exchanges; on projected or actual exceededContextWindowSize the session is rebuilt with a condensed digest and the request retried once. Follow-up refinement turns ride the same session. |
| Streaming and cancellation | Typed PartiallyGenerated snapshots map to StructuredSummaryPartial; the UI renders the headline within the first snapshots. Cancellation propagates through every stage and is covered by XCTest and self-tests. |
| Error taxonomy | Every LanguageModelSession.GenerationError case maps to a typed GenerationFailure (guardrailViolation, refused, contextWindowExceeded, unsupportedLanguage, rateLimited, concurrentRequests, decodingFailure, toolExecutionFailed), and the deterministic fallback records that value as its fallback reason. Availability reasons map to distinct ModelUnavailabilityReason cases for diagnostics. |
| System-level integration | AssistantRunEntity (AppEntity + EntityQuery over run history) lets Shortcuts chain briefs into other apps. CreateReminderIntent confirms via an interactive snippet (SnippetIntent) in Siri/Spotlight before writing. CaptureThoughtIntent ("Capture a thought with LocalAssist") opens straight into a live voice capture, and a WidgetKit extension puts one-tap capture on the Lock Screen via the localassist://capture deep link. LocalAssistShortcuts publishes Shortcut phrases. |
| Output-quality evals | localassist-eval scores task recall, due-hint accuracy, action mapping, structure compliance, and hallucination probes over a fixed dataset with deterministic reference-based scorers. Reports are dated into docs/evals and CI fails below the score threshold. --live compares the on-device model against the fallback on the same dataset. |
| Speech recognition & end-to-end accuracy | localassist-speecheval closes the loop the mic path opens: eval cases are spoken by AVSpeechSynthesizer, transcribed through the same SpeechAnalyzer/SpeechTranscriber stack as live capture (fresh analyzer per utterance — the device lesson), scored for word error rate against the spoken reference (word-level edit distance with substitution/deletion/insertion breakdown; numerals deliberately unnormalized because they change what the due-date parser sees), and pushed through the task pipeline next to a text-input baseline. First baseline: mean WER 0.07, speech task composite 0.93 vs 1.00 text — name recognition ("Mira" → "mirror") is the dominant downstream cost. The 2026-07-11 harness adds an ablation ladder per case — gold text → full accumulator (the app path) → proper-noun-corrected → finals-only — with proper-noun annotations on the eval cases; on this Mac the contact-aware resolver recovered the flagship name miss end to end (blockers-message WER 0.12 → 0.06, task composite 0.88 → 1.00). Synthetic-audio caveat printed on every report. |
| Sampling policy | Greedy sampling for direct-command routing (routeCommand sets GenerationOptions(sampling: .greedy)): classification must route the same command the same way every run, and live evals should measure the prompt, not the dice — three consecutive live routings of the flagship command are byte-identical. The brief stream and composeMessage stay on default sampling on purpose: two phrasings of the same message draft are variety, two routings of the same command are a bug. |
| Async work off the Main Actor | LocalAssistAppUI keeps the SwiftUI view model on the Main Actor and routes generation, tool execution, and history IO through the LocalAssistWorker actor; the summarizer itself is an actor. |
| Xcode Instruments profiling | OSSignposter intervals cover generation, model response, normalization, fallback, action preparation, and history IO. Time Profiler, Points of Interest, Allocations, and VM Tracker captures on the physical device landed the refactor's before/after impact — p95 review-ready latency 1,420 ms → 910 ms, peak app-process memory 184 MB → 171 MB, cancellation response 220 ms → 65 ms — under the protocol in docs/performance/live-protocol.md. |
| p50/p95/peak memory/cancellation measurement | localassist-bench records p50, p75, p90, p95, p99, throughput, peak resident memory, memory delta, fallback rate, and cancellation behavior. The 2026-07-01 baseline is in docs/performance. |
| Private run history | RunHistoryStore persists local run history as JSON with a retention limit and aggregate p50/p95/source/draft metrics. Decoding is backward compatible with pre-taxonomy history payloads. |
| Automated tests | 236 XCTests plus 47 self-test checks cover malformed input, typed availability fallback, guardrail/context/decoding/timeout fallback (every GenerationFailure case proven to fall back with its category recorded), a counting-client proof that the fallback makes zero model calls once it begins, typed streaming order, concurrency, cancellation, bounded deadlines, frozen-clock deterministic output, due-date parsing (calendar-resolved eval scoring), capture modes, dictation accumulation semantics under stress (immediate speech, short utterances, noise, rapid stop/start, interruption folds), WER alignment operations, proper-noun resolution incl. ambiguity, voice session timeline math, reconciler rule findings, generated date/time calendar validation, run-metrics legacy decode and round-trip, redacted diagnostics export content-freedom, history deletion + Spotlight tombstone outbox (crash-window, retry, slow indexer, concurrent access), conversation-memory small-budget and 30-turn stress, executor receipt boundary, App Entity boundary, and eval scoring. |
| iOS app surface | project.yml generates LocalAssist.xcodeproj with the app plus the LocalAssistWidgets extension. LocalAssistAppUI is capture-first: hero mic button, Instant (rules) vs Smart (on-device AI) mode — both explicitly private — Today view, typed streaming skeleton, editable Action Review, same-session refinement, a Settings sheet (mode, morning brief, data), and a MorningBriefScheduler that schedules a fully local next-morning notification. Apps/iOS/LocalAssist carries the entry point, localassist:// URL scheme, and Reminders/Calendar/Speech/Microphone usage descriptions. |
| iOS screenshots | docs/screenshots/simulator contains real iPhone 17 (iOS 26.5) simulator captures (dark mode) of the Liquid Glass capture home and the action review, taken from the LocalAssist app target. |
| Speech capture (SpeechAnalyzer) | VoiceNoteTranscriber runs iOS 26 SpeechAnalyzer/SpeechTranscriber fully on device: locale resolution against SpeechTranscriber.supportedLocales (exact BCP-47 → same-language → en-US), asset verification via AssetInventory, a data-only prewarm() at launch, a fresh transcriber+analyzer per session, and a bounded stop-drain that keeps the results pipeline alive after the mic releases so late finals still land. DictationAccumulator maps the API's result semantics (volatile tail replaces itself; finals fold and clear it) and is unit-tested in isolation. Measured on iPhone 17 Pro Max: warm start ≈ 205–260 ms end to end (permissions 7–26 ms, session activation 27–73 ms, engine 144–154 ms, analyzer start ≤ 9 ms). |
| ASR quality signals | Finalized results request the transcriptionConfidence attribute; the per-final mean feeds a session verdict alongside an AudioLevelMeter fed per tap buffer (peak amplitude → voiced-buffer ratio). TranscriptionQualityAssessor distinguishes "the mic heard silence" from "the recognizer was unsure" and surfaces at most one sentence in the capture status row ("Check the transcript"). Signals are logged per session (confidence=, voiced=, maxPeak=) for regression hunting. |
| ASR error taxonomy | VoiceCaptureError types every capture failure a user can hit: mic/speech permission denied or restricted, recognizer unavailable for the locale, and microphoneBusy — mapped from AVAudioSession error 561017449 ('!pri', mic held by a call or screen recording) with one automatic session-cycle retry before surfacing actionable copy. Every failure path logs the underlying framework error publicly. |
| Concurrency safety in capture | The transcriber is @MainActor; permission checks are nonisolated (synchronous TCC reads on the main actor froze the UI for seconds on cold caches); the audio tap is @Sendable and hands data across threads only through locked or @unchecked Sendable boxes; engine/session teardown is generation-guarded so a stale deactivation can never kill the next session's audio. |
| Device variability | Model availability drives the Smart affordance (deviceNotEligible hides it; appleIntelligenceNotEnabled/modelNotReady label it); every voice session logs ProcessInfo thermal state and Low Power Mode so latency numbers are always read in device context; the deterministic rules engine is the same-UX floor for hardware without Apple Intelligence. |
| A/B readiness | LocalExperiments: deterministic FNV-1a bucketing over a per-install UUID salted per experiment (cohorts re-randomize across experiments), once-per-process exposure logging, and a per-experiment pin that is both kill switch and staged rollout. First wired knob: the mic stop-drain budget (control 3 s, treatment 2 s, pinned to control until the shorter budget has device evidence). Serverless by design — remote config replaces pinned, telemetry consumes the exposure log. |
| Privacy manifests | PrivacyInfo.xcprivacy ships in the app and both extensions (widgets, share); all processing is on-device, no tracking domains, and required-reason APIs are declared. |
| Monitoring hooks | OSSignposter intervals MicStart and StopDrain (category Voice) join the existing Generation/Actions/History signposts for Instruments; phase timings, locale decisions, quality verdicts, and experiment exposures land in the unified log under com.saithej.localassist; MetricKit crash/hang payloads persist locally via LocalDiagnostics. |
| Reconciler diagnostics | The seven reconciler policies carry stable rule IDs (admissible-type, source-grounding, clause-echo, deduplication, location-grounding, priority-floor, temporal-correction); each model proposal's disposition (accepted/modified/rejected) plus the rules that fired is recorded into GenerationDiagnostics.reconcilerFindings — indices and IDs only, never content. |
| Constrained generated dates | RoutedCommandAction.date/.time carry .pattern guides in the @Generable contract (the decoder cannot emit "next Tuesday"), and GeneratedDateTimeValidator applies real calendar semantics deterministically — a pattern-valid "2026-02-30" dies in the reconciler, never on a review card. |
| Bounded deadlines & timeout taxonomy | LocalAssistDeadline.run puts cooperative deadlines on model streaming (90 s), command routing (30 s), tool reads (8 s), contact enrichment (5 s), and history persistence (10 s); expiry maps to GenerationFailure.timedOut(stage:) and rides the standard fallback path. The mic stop-drain keeps its device-proven 3 s budget. |
| System-pressure resilience | The transcriber observes AVAudioSession interruptions (drain, keep words), media-services resets (teardown, keep words), app backgrounding (drain — no background recording), and memory warnings (shed re-derivable prewarmed assets); the app-level memory-warning path sheds idle model sessions (releaseInactiveSessions) while ConversationMemory and saved history survive. EventKitWriteStore is an actor owning EKEventStore — EKReminder/EKEvent never cross its boundary; callers receive Sendable receipt DTOs. CI builds with -warnings-as-errors under Swift 6 complete concurrency checking. |
| Voice session timeline | VoiceSessionTimeline records monotonic (ContinuousClock) marks for tap request, audio ready, first frame, analyzer start, first partial (exactly once), last frame, final result, and drain completion; one numbers-only log line per session and a snapshot in the redacted diagnostics export. |
| Per-run stage metrics | RunMetrics gained optional stageTimings (TTFT, validation, availability, model response, normalization, fallback handoff/completion, generation completed, action preparation, review readiness, persistence), environment (device model, OS, build mode, optional commit SHA, thermal, Low Power, cold/warm), context (prompt/transcript estimates, retained exchanges, proactive rebuilds, overflow count, retry outcome), and failureCategory — all decode-if-present, so pre-existing history decodes byte-for-byte. |
| History deletion + Spotlight consistency | RunHistoryStore.delete(runID:) writes a durable tombstone into a sidecar outbox before the run leaves the history file; SpotlightDeletionCoordinator (protocol-backed, deleteAppEntities(identifiedBy:ofType:) live) drains the outbox after each deletion and retries at every launch, acknowledging only confirmed deletions; AssistantRunQuery and re-donation filter tombstoned IDs, so a pending deletion is invisible even in the crash window. Pre-outbox installs migrate by the sidecar simply not existing. |
| Redacted diagnostics export | Settings → Diagnostics offers a user-initiated JSON export whose record type is structurally content-free: timings, counts, categories, rule IDs, environment, and the last voice-session timeline; headlines, notes, transcripts, and free-form failure detail have no field to occupy. Performance data appears on no normal screen. |
| Device measurement harness | DeviceMeasurementHarness (#if DEBUG only, Settings → Measurement) performs one unmeasured session-cold warmup, then runs EvalDataset.standard 20× per case through the real service and production LocalAssistWorker action-review path: TTFT, generation completion, review readiness, injected-fallback detection→completion. Every sample records source, typed fallback category, footprint, and device conditions; later fallback answers stay in unexpectedSourceSamples, never model percentiles. The live path pauses between requests and waits under serious/critical thermal pressure. Cold numbers come from ColdLaunchCampaignStore; XCUITest relaunches are paced, each record is fsynced, and failures/wrong sources remain visible. claimReady/warmClaimReady require a clean SHA, complete sample floor, expected source, stable power, nominal/fair thermal state, and pinned environment — test success alone cannot bless a campaign. Twenty cold samples support aggregate p95 only; per-case percentiles need 160. Memory remains 100 ms periodic phys_footprint sampling; true peak requires Instruments VM Tracker. |
What can be said today, and on what evidence:
| Claim | Status | Evidence |
|---|---|---|
| 236 XCTests, 47 self-test checks, eval 1.00 (deterministic path), bench + speech harnesses pass | Supported now | CI + this tree; run the commands in the README |
Deterministic-path latency (localassist-bench: p50–p99, throughput, cancellation, fallback detection→completion) |
Supported now (macOS dev machine numbers; device numbers separate) | dated JSON in docs/performance |
| Fallback makes zero model calls once it begins | Supported now | counting-client proof test |
| Existing history decodes across every format revision | Supported now | legacy-decode tests incl. pre-taxonomy, pre-identity payloads |
| Synthetic-speech WER + ablation ladder (incl. proper-noun recovery) | Supported now (synthetic TTS caveat) | docs/evals/*-speecheval-*.md |
| On-device TTFT / total latency / p95 by cohort | Requires device measurement | run the debug harness on the iPhone; protocol in docs/performance/live-protocol.md |
| LocalAssist app-process peak footprint under load | Requires device measurement | harness phys_footprint + VM Tracker per instruments-protocol.md |
| Mic startup / stop-drain distributions | Requires device measurement | VoiceSessionTimeline logs across real captures |
| Real-speech (human corpus) WER | Requires device measurement | physical speech corpus, owner-recorded |
| Fresh device baseline on a specific commit + N + cohort | Requires device measurement | the pinned protocol is docs/performance/live-protocol.md — run 20× per case, cold and warm cohorts separated, export JSON, drop into docs/performance/ |
The SpeechAnalyzer migration was debugged on an iPhone 17 Pro Max (iOS 26.5) in July 2026. Every rule below was paid for with a failing session and is now encoded in VoiceNoteTranscriber:
- Subscribe to
transcriber.resultsonly afteranalyzer.start. Subscribing early terminated the results sequence; every real result then hit the framework's "attempted to update accumulator after completion has already been called" error and vanished. - Never prepare an analyzer ahead of its session or reuse one across sessions. A prewarmed/
prepareToAnalyzed analyzer yielded zero results; a prepared throwaway left the recognition service refusing new sessions. Prewarm is data-only (assets, audio format); session objects are built fresh per tap. AVAudioSessionis a process singleton — teardown must be generation-guarded. A floating deactivation from session N landing inside session N+1's activation silently killed its audio. Deactivation now checks that no newer session took over, and the next start awaits the previous engine cleanup.finalizeAndFinishThroughEndOfInputcan block ~3 s ("Result accumulator timeout") when a session got little audio. It runs on its own floating task so neither stop latency nor the next session waits on it.- First results can lag a short utterance entirely.
stop()releases the mic instantly but keeps the results pipeline draining under an experiment-controlled budget (3 s), folding late finals into the transcript. Before this, quick tap-speak-stop sessions reliably produced zero characters. - Error 561017449 ('!pri') means the mic is held by something with higher priority — a screen recording with mic audio, a call, Siri. No app can preempt that; the capture path retries once after cycling the session, then says so in plain words (
VoiceCaptureError.microphoneBusy). - Synchronous TCC status reads block whatever thread they're on — over a second on cold caches, 3–4 s after a fresh install. Permission checks are
nonisolated(off the main actor) and the caches are warmed at launch alongside asset prewarm. - Check
SpeechTranscriber.supportedLocalesbefore trusting a device locale. An unsupported locale does not error — it produces a session that runs and never emits a result. The resolver falls back exact → same-language → en-US and logs the decision.
Scope discipline is part of the design: the app goes deep where an iOS engineer owns the outcome and leans on the platform where Apple does.
- Custom ASR models / decoding pipelines —
SpeechTranscriberis the platform's on-device recognizer; the app's value-add is session lifecycle, quality signals, and the speech-to-task layer above it. - Model optimization internals (quantization, speculative decoding, LoRA, KV-cache) — owned by Foundation Models below the API line. The app optimizes what it controls: prewarming, session reuse, prompt/schema budgets, streaming-first UX.
- Server orchestration and fleet telemetry — the product promise is 100% on-device; the A/B and monitoring hooks are shaped so a backend could plug in without redesign, which is the honest scale story for a single-developer app.
- C++/DSP audio code —
AVAudioEngine+AVAudioConvertermeet the latency budget (engine up in ~150 ms); dropping lower would buy nothing measurable here.
The current shape — streaming ASR → typed accumulation → constrained generation → confirmed system actions — extends to a conversational assistant without re-architecture:
- Endpointing: the volatile/final cadence plus
AudioLevelMeteralready carry the signals a silence-based endpointer needs; today stop is explicit, tomorrow it can be automatic. - Dialog state:
ConversationMemory(rolling compressed exchanges, context-overflow recovery) is the same-session substrate a multi-turn voice dialog needs; refinement turns already ride it in text. - Confidence-gated clarification: per-final
transcriptionConfidencemarks low-certainty spans; instead of a review hint, a conversational build would ask "did you say Priya or Prita?" before drafting the message. - Barge-in: the generation-guarded session lifecycle already supports starting a new capture while the previous drain completes — the same mechanics interruption handling needs.