Skip to content

Microphone API + Permissions [WIP] - #2096

Draft
jplexer wants to merge 15 commits into
coredevices:mainfrom
jplexer:voice-api-permissions
Draft

jplexer wants to merge 15 commits into
coredevices:mainfrom
jplexer:voice-api-permissions

Conversation

@jplexer

@jplexer jplexer commented Sep 17, 2026

Copy link
Copy Markdown
Member

requires the corresponding coreapp pr https://github.com/coredevices/CoreApp/pull/277

jplexer and others added 15 commits September 17, 2026 13:26
Apps declare the microphone permission with `capabilities: ["microphone"]`
in their manifest. Record that declaration in the binary header (bit 11)
so the firmware can tell "not declared" apart from "denied" when it
enforces the upcoming per-app permission system, and surface it through
PebbleProcessMd and AppInstallEntry.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ekdUsEpAe9dNq5wyLYhGc
Signed-off-by: Joshua Jun <lets@throw.rocks>
Adds BlobDBIdAppPermissions (0x0D), a settings-file backed store of
per-app permission grants keyed by app UUID. The phone is the source of
truth and pushes a small versioned record with a granted mask and a
declared mask. Records are removed when the app is uninstalled, but kept
across upgrades so grants persist.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ekdUsEpAe9dNq5wyLYhGc
Signed-off-by: Joshua Jun <lets@throw.rocks>
Turns the phone-pushed grant records and the app's manifest declaration
into a per-app AppPermissionState (not declared / denied / granted).
System apps are always granted; declared apps without a record are
denied, except on SDK shell builds where sideloaded apps default to
granted so development works without the phone.

When the running app's record changes the service emits
PEBBLE_APP_PERMISSION_EVENT so the app can react.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ekdUsEpAe9dNq5wyLYhGc
Signed-off-by: Joshua Jun <lets@throw.rocks>
Lets apps query the state of a declared permission and subscribe to
changes (app_permission_get_state, app_permission_is_granted,
app_permission_service_subscribe/unsubscribe), exported as SDK revision
110. Also advertises app_permissions_support (capability bit 25) to the
phone so it knows the watch accepts grant records; enforcement does not
depend on the bit.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Joshua Jun <lets@throw.rocks>
`perm list` dumps the grant records and whether the connected phone
supports pushing them; `perm grant <id> mic` / `perm revoke <id> mic`
write a grant locally so permissions can be exercised without the phone.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Joshua Jun <lets@throw.rocks>
The mic driver has a single client and no arbitration. Introduce a small
kernel arbiter so dictation and the upcoming app capture service can
share it: dictation always wins and preempts an app (which is told
through a callback before the mic restarts), while an app is refused
while dictation runs. Route the dictation service through it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Joshua Jun <lets@throw.rocks>
Kernel side of live PCM capture for apps. Serves the app task only,
requires focus and the microphone permission, acquires the mic through
mic_manager and buffers samples in a 320 ms kernel ring the app drains
through syscalls, with coalesced PEBBLE_MIC_CAPTURE_EVENT data events.

Capture stops, with a reason reported to the app, when the app loses
focus to a modal window, when the grant is revoked, or when dictation
preempts the mic. App exit tears it down silently.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Joshua Jun <lets@throw.rocks>
A transparent, unfocusable strip at the bottom of the screen, pushed at
discreet modal priority for as long as an app captures the microphone.
Like Timeline Peek it reserves its area through the unobstructed area
service, and the two obstructions compose so an app sees the union.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Joshua Jun <lets@throw.rocks>
mic_data_service_subscribe() delivers batches of raw 16 kHz mono PCM
to a foreground app that declares and was granted the microphone
permission, draining the kernel capture ring through syscalls. Capture
ends with a MicDataStopReason when the app loses focus, the grant is
revoked, or the system takes the microphone. Exported as SDK revision
111.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Joshua Jun <lets@throw.rocks>
SELECT toggles live capture through the Microphone API. The screen shows
the permission state, the unobstructed height (which shrinks under the
OS banner), the RMS of the last batch and the overrun count, and logs
why capture stopped. Enable with CONFIG_DEMO_APP_MIC_DEMO=y.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Joshua Jun <lets@throw.rocks>
Dictation's Speex wrapper becomes a backend of a small encoder service
with a single, task-owned instance and a codec descriptor
(AudioEncoderInfo) that tells the consumer how to decode. Dictation
opens it as the system owner and takes precedence over an app. This is
the seam for the upcoming app-facing encoder API and for an Opus/SILK
backend.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Joshua Jun <lets@throw.rocks>
Lets an app compress microphone frames with the firmware's speech
encoder (Speex wideband today) before shipping them to the phone over
AppMessage: audio_encoder_open() returns the frame size and packet
bound, audio_encoder_encode_frame() produces one packet per frame.
Exported as SDK revision 112.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Joshua Jun <lets@throw.rocks>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Joshua Jun <lets@throw.rocks>
Built-in apps launched without an install id (e.g. from the console)
were not recognised as system apps. Use the process metadata's
privilege flag as well as the install id.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Joshua Jun <lets@throw.rocks>
mic_stream_to_phone_start() lets an app send its microphone to the phone
without touching the samples: the capture service opens the Speex
encoder, sets up an audio endpoint transfer and asks the phone for a new
VoiceEndpointSessionTypeAudioStream session tagged with the app UUID.
Once the phone accepts, frames are encoded on KernelBG and sent over the
audio endpoint like dictation, with no result expected. The app hears
about acceptance (started) and about the phone refusing, stopping or not
answering (MicDataStopReasonPhone). Dictation preempts a stream before
it takes the encoder and endpoints. SDK revision 113.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GvaCa5cZd2nfywBVhRP9QE
Signed-off-by: Joshua Jun <lets@throw.rocks>
@jplexer
jplexer force-pushed the voice-api-permissions branch from ac7cc76 to 02a48f6 Compare September 17, 2026 12:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant