Skip to content

Commit c7b6920

Browse files
enaboappsOwenMcGirrOwen McGirr
authored
Add offline word prediction to the scanning keyboard (#796)
* feat: integrate offline keyboard word prediction (#795) * fix: discard fallback edits after external activity * fix: align prediction tracking with active switch capture * fix: keep keyboard visible and repaint only changed tiles * fix: restart scan interval when suggestions receive focus --------- Co-authored-by: Owen McGirr <o.a.mcgirr@gmail.com> Co-authored-by: Owen McGirr <owenmcgirr@Owens-Mac-Studio-2.local>
1 parent 8105a51 commit c7b6920

26 files changed

Lines changed: 2393 additions & 27 deletions

.github/workflows/ci.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ jobs:
4040
- name: Reject retired product identity
4141
run: |
4242
retired_identity="pre""view"
43-
if git grep -n -i "$retired_identity" -- ':!src-tauri/Cargo.lock'; then
43+
if git grep -I -n -i "$retired_identity" -- ':!src-tauri/Cargo.lock'; then
4444
exit 1
4545
fi
4646
- uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6

docs/qwerty-keyboard.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ The bottom control row starts with Close keyboard, followed by Letters, Navigati
1616

1717
Modifier keys are pressed only around each emitted shortcut and immediately released. Selecting a locked modifier does not hold that operating-system key while the scanner runs. Ordinary characters use text injection; command combinations and navigation use native key events. Native shortcuts retain the operating system's layout semantics.
1818

19-
The keyboard closes when the foreground target or display environment changes, or its scan session ends. Failed input clears keyboard modifiers and requires Select to resume. Stop, disconnect and application exit use the shared deterministic input cleanup. No typed text is logged, and no prediction data, context reader, protocol extension or persistent keyboard settings are added.
19+
The keyboard closes when the foreground target or display environment changes, or its scan session ends. Failed input clears keyboard modifiers and requires Select to resume. Stop, disconnect and application exit use the shared deterministic input cleanup. Word prediction can be enabled in Scanning settings. Five suggestions appear on the Letters page. Selecting a suggestion inserts its missing suffix and a space. Prediction uses local, read-only data; no personal vocabulary is saved. See [word prediction](word-prediction.md) for context availability and validation.
2020

2121
## Manual validation
2222

docs/word-prediction.md

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# Word prediction
2+
3+
Word prediction is enabled by default in Scanning settings. Its five-position row appears only on the Letters page. Empty positions are skipped. Select the row and then a word using the existing switches. Acceptance appends the missing suffix and a space without deleting text, selecting text or using the clipboard. Shift and Caps affect completion casing; Ctrl, Alt/Option and Windows/Command suppress suggestions.
4+
5+
The bundled English SQLite database works offline. Ranking backs off from the longest available three-word context to shorter contexts, then to word frequency. Prefix matching is case-insensitive. Duplicate words are removed. The data is fixed; typing is never used to train or update it.
6+
7+
Windows uses UI Automation and macOS uses Accessibility range attributes. Both read at most 512 preceding characters and a short following range, reject protected fields, and hide predictions for selections or mid-word insertion. Applications that do not expose reliable text ranges may have no predictions. No compatibility claim follows from compilation alone.
8+
9+
If an editable non-protected field can be identified but its text is unavailable, the fallback remembers only successful Switchify typing after a word boundary entered through Switchify. It does not reconstruct existing text. Navigation, deletion, shortcuts, external typing/clicks, focus changes and failures clear this buffer. Fallback is unavailable when its passive activity observer cannot start. The observer records only an activity counter, never external keys or text. Numbers and navigation pages do not retain fallback typing.
10+
11+
A separate process performs accessibility and database work. Private bounded inherited pipes carry results to native rendering. Text and suggestions are not sent to the React UI, diagnostic history or telemetry. One request is outstanding at a time with a two-second deadline. Timeout stops predictions until the keyboard is reopened; ordinary keyboard operation remains available. Closing the keyboard, ending scanning, or exiting kills and reaps the worker.
12+
13+
Before a suggestion is accepted, the worker checks the field, protection state, text/selection snapshot and external activity again. Any mismatch rejects the action. Verification and native insertion cannot be atomic across applications; a field can still change in that short interval.
14+
15+
## Validation
16+
17+
Automated tests use in-memory databases, fake accessibility adapters and sleeping subprocesses. They never inject desktop input. The database benchmark can be run explicitly with:
18+
19+
```powershell
20+
cargo test --manifest-path src-tauri/Cargo.toml bundled_database_benchmark -- --ignored --nocapture
21+
```
22+
23+
This benchmark prints counts and timings only and does not read a focused field. Its ignored status keeps performance measurement separate from correctness checks.
24+
25+
Native integration checks remain pending on Windows and macOS. The older standalone probe's Notepad result does not qualify this integration. Manually test synthetic text in Notepad, TextEdit, Edge/Chrome, Safari and Word where installed. Exercise typing, prediction acceptance, Shift/Caps, external edits, selection, caret movement, passwords, page changes, top/bottom docking, disconnect and app exit. On macOS use `npm run macos:run` to retain the signed Accessibility identity.
26+
27+
For each app tested, collect at least 100 successful reads and report median, p95, maximum and failures. Target warm context-plus-query p95 below 50 ms; report startup and the 250-ms polling interval separately. Unsupported and untested apps must remain identified as such.
28+
29+
Prepare a disposable field containing `I would like some wa`, with the caret after `wa`. From the repository root run the command below, then focus that field during its five-second countdown. It samples every 250 ms, stops after 100 supported warm samples (or 200 attempts), and prints only counts, fixture comparisons and timings. It sends read-only queries; it cannot accept a word or inject input. The first request includes worker/database startup and is excluded from warm percentiles.
30+
31+
```powershell
32+
node scripts/measure-word-prediction.mjs src-tauri/target/debug/switchify-pc.exe Notepad water
33+
```
34+
35+
For macOS, pass the signed app's `Contents/MacOS/switchify-pc` executable after building with `npm run macos:run`. Use a fresh run for each app. Keep all focused fields synthetic throughout measurement. Repeat functional checks with an empty field, `Hello. wa`, multiline text, `café naïve wa`, a pasted sentence, deletion, a caret inside a word, a nonempty selection, selection replacement, and switching between two fields. Verify that protected fields produce no suggestions and that closing the keyboard removes the worker. Do not count empty/unsupported responses as successful latency samples.
36+
37+
Measured locally on Windows in a debug build (200 synthetic database queries): median 2.59 ms, p95 10.78 ms, maximum 14.35 ms, zero query failures. Database startup took 251.13 ms. These numbers exclude accessibility and IPC and are not app compatibility results.
38+
39+
| App | Integration status |
40+
| --- | --- |
41+
| Notepad, Edge, Chrome, Word on Windows | Untested with this integration |
42+
| TextEdit, Safari, Chrome, Word on macOS | Untested; macOS host unavailable locally |
Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
// Manual, read-only fixture measurement. Never sends an Accept request.
2+
import { spawn } from 'node:child_process';
3+
import { resolve } from 'node:path';
4+
import { setTimeout as delay } from 'node:timers/promises';
5+
import { performance } from 'node:perf_hooks';
6+
7+
const [executable, app, expectedWord] = process.argv.slice(2);
8+
if (!executable || !app || !expectedWord) {
9+
console.error('Usage: node scripts/measure-word-prediction.mjs <executable> <app-label> <expected-word>');
10+
process.exit(1);
11+
}
12+
console.log('Focus a synthetic test field now. Sampling starts in five seconds. No input will be injected.');
13+
await delay(5000);
14+
const child = spawn(resolve(executable), [
15+
'--switchify-prediction-worker',
16+
resolve('src-tauri/resources/WordData2017051601.db'),
17+
'[]',
18+
], { stdio: ['pipe', 'pipe', 'ignore'], windowsHide: true });
19+
let buffer = Buffer.alloc(0);
20+
let pending;
21+
let failed = false;
22+
const fail = () => {
23+
failed = true;
24+
pending?.reject(new Error('Worker unavailable'));
25+
pending = undefined;
26+
};
27+
child.on('error', fail);
28+
child.on('exit', fail);
29+
child.stdin.on('error', fail);
30+
child.stdout.on('data', chunk => {
31+
buffer = Buffer.concat([buffer, chunk]);
32+
if (buffer.length < 4) return;
33+
const length = buffer.readUInt32LE();
34+
if (!length || length > 16384 || buffer.length > length + 4) return fail();
35+
if (buffer.length !== length + 4) return;
36+
try {
37+
const response = JSON.parse(buffer.subarray(4).toString('utf8'));
38+
buffer = Buffer.alloc(0);
39+
const request = pending;
40+
pending = undefined;
41+
if (!request) return fail();
42+
request.resolve(response);
43+
} catch { fail(); }
44+
});
45+
async function query(generation) {
46+
if (failed) throw new Error('Worker unavailable');
47+
const bytes = Buffer.from(JSON.stringify({ Query: {
48+
generation, edit: null, reset: true, shift: false, caps: false,
49+
} }));
50+
const header = Buffer.alloc(4);
51+
header.writeUInt32LE(bytes.length);
52+
let timer;
53+
try {
54+
return await new Promise((resolve, reject) => {
55+
pending = { resolve, reject };
56+
timer = setTimeout(() => { fail(); child.kill(); }, 2000);
57+
child.stdin.write(Buffer.concat([header, bytes]));
58+
});
59+
} finally { clearTimeout(timer); }
60+
}
61+
const stop = () => { fail(); child.kill(); };
62+
process.on('SIGINT', stop);
63+
process.on('SIGTERM', stop);
64+
const times = [];
65+
let unsupported = 0;
66+
let failures = 0;
67+
let fixtureMatches = 0;
68+
let firstRequestMs;
69+
try {
70+
// Up to 200 attempts, stopping after 100 supported samples.
71+
for (let generation = 0; generation < 200 && times.length < 100; generation++) {
72+
const start = performance.now();
73+
const response = await query(generation);
74+
const elapsed = performance.now() - start;
75+
if (generation === 0) firstRequestMs = elapsed;
76+
const result = response.Suggestions;
77+
if (!result || result.generation !== generation) throw new Error('Invalid response');
78+
if (result.batch?.words?.length) {
79+
if (generation !== 0) times.push(elapsed);
80+
if (result.batch.words.some(word => word.toLowerCase() === expectedWord.toLowerCase())) fixtureMatches++;
81+
} else unsupported++;
82+
await delay(Math.max(0, 250 - elapsed));
83+
}
84+
} catch { failures++; }
85+
finally {
86+
child.kill();
87+
process.off('SIGINT', stop);
88+
process.off('SIGTERM', stop);
89+
}
90+
times.sort((a, b) => a - b);
91+
const percentile = fraction => times.length ? +times[Math.ceil(times.length * fraction) - 1].toFixed(2) : null;
92+
console.log(JSON.stringify({
93+
app, successfulSamples: times.length, unsupported, failures, fixtureMatches,
94+
firstRequestMs: firstRequestMs === undefined ? null : +firstRequestMs.toFixed(2),
95+
medianMs: percentile(0.5), p95Ms: percentile(0.95), maximumMs: percentile(1),
96+
}, null, 2));
97+
if (failures || times.length < 100) process.exitCode = 1;

src-tauri/Cargo.lock

Lines changed: 90 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

src-tauri/Cargo.toml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,8 @@ enigo = "=0.6.1"
2121
hmac = "0.12"
2222
keyring = "4.1.6"
2323
rand = "0.8"
24+
rusqlite = { version = "0.40.2", features = ["bundled"] }
25+
unicode-segmentation = "1.12"
2426
reqwest = { version = "0.13", default-features = false, features = ["json", "rustls"] }
2527
serde = { version = "1", features = ["derive"] }
2628
serde_json = "1"
@@ -60,11 +62,15 @@ windows = { version = "0.62.2", features = [
6062
"Win32_Security",
6163
"Win32_Storage_FileSystem",
6264
"Win32_System_LibraryLoader",
65+
"Win32_System_Com",
66+
"Win32_System_Ole",
67+
"Win32_System_Variant",
6368
"Win32_System_Power",
6469
"Win32_System_Registry",
6570
"Win32_System_Threading",
6671
"Win32_System_WinRT",
6772
"Win32_UI_HiDpi",
73+
"Win32_UI_Accessibility",
6874
"Win32_UI_Input_KeyboardAndMouse",
6975
"Win32_UI_WindowsAndMessaging"
7076
] }

src-tauri/resources/PROVENANCE.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# English prediction database
2+
3+
`WordData2017051601.db` was recovered from the user's private `enaboapps/sayit-ios` history, path `Assets/Databases/WordData2017051601.db`, commit `95bda9265758e50787c5da2779b16f36b53c319d`, dated May 20, 2017. Recovery took place September 18, 2026. The user authorized bundling this database with Switchify.
4+
5+
- Size: 102,690,816 bytes.
6+
- SHA-256: `dedd65d263bde8315e7e5ed7d2c8e04f17c33598a68506c9d70661a6d1f57318`.
7+
- SQLite integrity check during recovery: `ok`.
8+
- Rows: WORDS 165,420; BIGRAMS 976,804; TRIGRAMS 996,846; QUADGRAMS 953,630.
9+
10+
This is the unchanged original SQLite file. The later Realm archive is not used. Runtime connections are read-only and ignore personal frequency columns. No captured text or user vocabulary is written into this resource.
97.9 MB
Binary file not shown.

src-tauri/src/lib.rs

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ mod point_scan;
2020
mod point_scan_activation;
2121
mod point_scan_runtime;
2222
mod point_workflow;
23+
mod prediction;
2324
mod protocol;
2425
mod remote_scan;
2526
mod scan_executor;
@@ -1361,6 +1362,9 @@ fn point_scan_prepare(app: &AppHandle) -> Result<(), String> {
13611362
}
13621363

13631364
pub fn run() {
1365+
if prediction::worker::run_from_args() {
1366+
return;
1367+
}
13641368
#[cfg(target_os = "windows")]
13651369
if switch_input::run_worker_from_args() {
13661370
return;

0 commit comments

Comments
 (0)