Repository navigation
Say a button: guide to the button you name - #66
Merged
Merged
Conversation
A Say a button control listens on the device (the same speech setup as LiveCameraView) and checks what it hears against the scanned panel as the words come in, so "start" starts guidance as soon as it's heard. It gives up after 8 seconds and shows what it heard. ButtonLookup tries TargetMatcher on the whole request, then word by word through TargetMatcher so its synonyms still apply, against each word of each label. That finds Start on a key read as "Start Pause" off two lines. The shortest matching label wins. Guide to Start uses it too.
3 of 11 tasks
…urous testing for microwaves: Pre-Test panel: | Photo | start | stop | pause | timer | time | clock | add 30 seconds | power | |---|---|---|---|---|---|---|---|---| | Angle.jpeg | nil | nil | nil | Time (0.9) | Time (1.0) | Time (0.9) | nil | nil | | Glare.jpeg | nil | nil | nil | nil | nil | nil | nil | nil | | MicrowaveGlare.heif | nil | nil | nil | nil | nil | nil | nil | nil | | MicrowaveStraight.HEIC | nil | nil | nil | nil | nil | nil | nil | nil | | Neat1.jpeg | nil | nil | nil | nil | nil | nil | nil | nil | | Neat2.jpeg | nil | Cancel (0.9) | Cancel (0.9) | nil | nil | nil | nil | nil | | angle-microwave.heif | nil | nil | nil | Clock (0.9) | Clock (0.9) | Clock (1.0) | nil | nil | | angle_microwave.jpg | nil | nil | nil | nil | nil | nil | nil | nil | | glare_microwave.jpg | Start (1.0) | Pause (0.9) | Pause (1.0) | nil | nil | nil | Add 30 Sec (0.9) | Power (1.0) | | microwave1_angle.heif | START (1.0) | CANCEL (0.9) | CANCEL (0.9) | Clock (0.9) | Clock (0.9) | Clock (1.0) | nil | nil | | microwave1_glare.heif | START (1.0) | CANCEL (0.9) | CANCEL (0.9) | Clock (0.9) | Clock (0.9) | Clock (1.0) | + 30 (0.9) | nil | | microwave1_neat.heif | START (1.0) | CANCEL (0.9) | CANCEL (0.9) | Clock (0.9) | Clock (0.9) | Clock (1.0) | nil | nil | | microwave1_neat2.heif | START (1.0) | CANCEL (0.9) | CANCEL (0.9) | Clock (0.9) | Clock (0.9) | Clock (1.0) | nil | nil | | microwave2-glare.jpg | START (1.0) | STOP (1.0) | STOP (0.9) | Clock (0.9) | Clock (0.9) | Clock (1.0) | nil | Power (1.0) | | microwave2-neat.jpg | nil | nil | nil | nil | nil | nil | nil | nil | | microwave2-neat2.jpg | nil | nil | nil | nil | nil | nil | nil | nil | | neat-microwave.heif | nil | nil | nil | nil | nil | nil | nil | nil | | normal_microwave.jpg | nil | nil | nil | nil | nil | nil | nil | Power (1.0) | Post-Test panel: | Photo | start | stop | pause | timer | time | clock | add 30 seconds | power | |---|---|---|---|---|---|---|---|---| | Angle.jpeg | nil | nil | nil | nil | Cook Time (0.8) | nil | nil | nil | | Glare.jpeg | nil | nil | nil | Timer on/off (0.8) | nil | nil | nil | Power Level (0.8) | | MicrowaveGlare.heif | nil | nil | nil | nil | Time Cook (0.9) | nil | nil | nil | | MicrowaveStraight.HEIC | nil | nil | nil | nil | nil | nil | nil | Level Power (0.8) | | Neat1.jpeg | Start Pause (0.8) | Cancel Off (0.7) | Start Pause (0.8) | Timer on/off (0.8) | Time Cook (0.9) | Clock (1.0) | 30 Add Sec (0.8) | Power Level (0.8) | | Neat2.jpeg | Start Pause (0.8) | Cancel (0.7) | Start Pause (0.8) | Timer on/off (0.8) | Time Cook (0.9) | Clock (1.0) | 30 Add Sec (0.8) | Power Level (0.8) | | angle-microwave.heif | nil | nil | nil | nil | COOk Time (0.8) | Clock (1.0) | nil | Level Power (0.8) | | angle_microwave.jpg | Start Pause (0.8) | Start Pause (0.7) | Start Pause (0.8) | Kitchen Timer (0.8) | Time Cook (0.9) | nil | nil | nil | | glare_microwave.jpg | Start (1.0) | Cancel Off (0.7) | Pause (1.0) | Kitchen Timer (0.8) | Time Cook (0.9) | Clock (1.0) | Add 30 Sec (0.9) | Power (1.0) | | microwave1_angle.heif | START (1.0) | CANCEL (0.7) | nil | Timer Set/Off (0.8) | Cook Time (0.8) | Clock (1.0) | 30 Sec (0.8) | Power Cook (0.8) | | microwave1_glare.heif | START (1.0) | CANCEL (0.7) | nil | Timer SeYOT (0.8) | Cook Time (0.8) | Clock (1.0) | + 30 (0.9) | Cook Power (0.8) | | microwave1_neat.heif | START (1.0) | CANCEL (0.7) | nil | Timer Set/Off (0.8) | Time Cook (0.9) | Clock (1.0) | + 30 Sec (0.8) | Cook Power (0.8) | | microwave1_neat2.heif | START (1.0) | CANCEL (0.7) | nil | Timer Set/Off (0.8) | Cook Time (0.8) | Clock (1.0) | 30 Sec (0.8) | Cook Power (0.8) | | microwave2-glare.jpg | START (1.0) | STOP (1.0) | STOP (0.7) | nil | nil | Clock (1.0) | nil | Power (1.0) | | microwave2-neat.jpg | nil | STOP CANCEL (0.8) | STOP CANCEL (0.7) | nil | nil | nil | nil | nil | | microwave2-neat2.jpg | START →vûsec. (0.8) | nil | nil | Timer (1.0) | Time Cook (0.9) | Clock (1.0) | nil | nil | | neat-microwave.heif | nil | nil | nil | Timer (1.0) | nil | Clock (1.0) | - 30 sec (0.8) | Level Power (0.8) | | normal_microwave.jpg | Start Pause (0.8) | Cancel Off (0.7) | Start Pause (0.8) | Kitchen Timer (0.8) | nil | Clock (1.0) | 30 Sec (0.8) | Power (1.0) |
This was referenced Oct 13, 2026
ContentView conflicted with #69's guidance flow. Say a button now goes through it too: guidanceTarget finds Start with ButtonLookup in place of startButton, and both buttons start guidance through startGuidance(to:), which cancels any running guidance and tracks it by guidanceID. Say a button is disabled when Guide to Start is (no map, canGuide false, a scan running), and refuses a heard button if canGuide has gone false or the panel was replaced while it listened. Listening is a tracked task like scanning: scan, the manual tracking test and stopWork (leaving the screen, going to the background) cancel it, and a cancelled request never starts guidance.
Scanning again or leaving the screen cancels the request. Before, the mic stayed on until the 8 second limit; now cancelling ends it right away, so a new request doesn't have to wait for the old one to let go of the mic.
#78 goes into main before this PR. Merging it here so ButtonLookup can be built on its whole-word matching instead of working around it.
With #78's whole-word matching, TargetMatcher finds Start on a key read as "Start Pause" by itself, so the word-by-word fallback and the shortest-label rule aren't needed. They also went around the matcher's guards: "one" found the 1 in the 1:35 display, "start" found Delay Start on an oven with no Start key, "popcorn" found a 20-word label merged off glare, and "cook time" found Cook instead of Time Cook. Each now returns nil or the right key, with a test for each. Guide to Start picks the same key as before on all 23 panels in #78's real labels, except the oven, where it used to pick Delay Start and now finds nothing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #65.
What changed
Say a button: say the button you want, and guidance starts to it.
VoiceRequestlistens on the device, using the sameSpeechTranscriber/SpeechAnalyzersetup asLiveCameraView(Evan's code from button ui #41). It checks what it hears against the scanned panel as the words come in, so "start" starts guidance as soon as it's heard, not after a pause. It gives up after 8 seconds. The screen shows what it heard, and says so when nothing on the panel matches.ButtonLookupdecides which button a request names:TargetMatcheron the whole request first.TargetMatcherso its synonyms apply ("go" is Start, "five" is 5), against each word of each label.That finds Start on a key read as "Start Pause" off two lines, which the matcher can't split yet. The shortest matching label wins, so "Start" beats "Start Time". Words like "press" and "please" are ignored.
Guide to Start uses the same lookup, replacing the temporary helper from Scan the panel from the live camera #62.
Which package
ContourApp— app target / wiringWhich lane
Officers (integration). This uses Experience's speech work and Surface's TargetMatcher.
How it was tested
xcodebuild test(ContourTests) on an iOS 27 simulator: 12 pass. 5 are new lookup tests:Scripts/check-dependencies.shpasses.Does this touch ContourCore?