Add native QWERTY switch-scanning keyboard (#790)

* Add native QWERTY switch-scanning keyboard (#789)

* Verify keyboard focus handoff and fix native key rendering

* Use rectangular image helper for macOS keyboard tiles

* Pad and vertically centre Windows keyboard labels

Measure wrapped key text before drawing and centre it inside scaled horizontal and vertical margins. Keep action-menu and modifier-overlay placement unchanged.

* Preserve room for wrapped keyboard labels on smaller displays

* Refine switch keyboard layout and scan feedback

Use weighted editing keys, a dedicated wide Space bar and one copy of each modifier. Add keyboard-specific visual roles, active-state indicators, rounded key artwork and a solid panel. Clarify row return and scan target feedback while preserving input and switch timing.

* Round and pad the scanning keyboard panel

Render a transparent rounded backdrop with scaled outer padding. Keep the backdrop cached during key scanning and verify transparent corners and inset control bounds.

* Skip empty native labels to prevent keyboard opening crash

Avoid passing an empty UTF-16 slice to DrawTextW for the rounded keyboard backdrop. Cover empty labels through the native drawing path and render the full keyboard in the interactive overlay smoke test.

* Remove number row from the letters keyboard

Move Backspace beside QWERTY and retain the removed digits and UK symbols on the Numbers page. Cover character availability and the letters layout in regression tests.

---------

Co-authored-by: Owen McGirr <o.a.mcgirr@gmail.com>
Co-authored-by: Owen McGirr <owenmcgirr@Owens-Mac-Studio-2.local>

Author: 3 people

docs/qwerty-keyboard.md

@@ -0,0 +1,32 @@
+# QWERTY scanning keyboard
+
+Switchify has its own UK English keyboard on Windows and macOS. It uses the saved local switches or the existing Android remote scanning actions. It does not use the operating system's on-screen keyboard.
+
+Choose a point and select **Type here** to click that point once and open the keyboard. To keep the existing input focus without clicking, select **More → Keyboard**. The keyboard does not take focus or accept mouse clicks.
+
+Scan a row, select it, then scan and select a key. The escape slot returns to rows. After typing, scanning starts again at the first row. Existing automatic/manual movement, speed, pause, reverse and inactivity suspension apply. The keyboard stays open after Space, Backspace, Enter and Tab. **Close** returns to point scanning.
+
+Pages:
+
+- **Letters:** UK QWERTY and punctuation, with Backspace beside the top letter row.
+- **Functions:** Esc, F1–F12, arrows, Home, End, Page Up, Page Down and Delete. Windows also has Insert, Print Screen, Scroll Lock and Pause.
+- **Numbers:** digits, decimal point, arithmetic operators, UK number-row symbols (including £), Enter and Backspace. Digits are independent of hardware Num Lock.
+
+The bottom control row starts with Close keyboard, followed by Letters, Navigation, Numbers and docking. Space has a dedicated wide key. Editing and modifier keys are wider than ordinary characters; duplicate Shift and Caps controls are removed from the letters page. A header names the current page and scan target. The return-to-rows slot highlights only the header. Active modifiers and the current page have an indicator separate from the scan highlight. Modifiers are available on every page. Shift, Ctrl, Alt/Option and Windows/Command cycle through **off → once → locked → off**. The label shows once or locked. Caps is a keyboard-local uppercase setting; Shift reverses its letter case. Page and position controls do not consume a one-shot modifier.
+
+Modifier keys are pressed only around each emitted shortcut and immediately released. Selecting a locked modifier does not hold that operating-system key while the scanner runs. Ordinary characters use text injection; command combinations and navigation use native key events. Native shortcuts retain the operating system's layout semantics.
+
+The keyboard closes when the foreground target or display environment changes, or its scan session ends. Failed input clears keyboard modifiers and requires Select to resume. Stop, disconnect and application exit use the shared deterministic input cleanup. No typed text is logged, and no prediction data, context reader, protocol extension or persistent keyboard settings are added.
+
+## Manual validation
+
+Use synthetic text in Notepad and a browser on Windows, and TextEdit and a browser on macOS. Launch macOS through `npm run macos:run` to retain its Accessibility identity.
+
+1. Use Type here and verify exactly one click focuses the intended field. Use More → Keyboard and verify no click or focus change occurs.
+2. Type lowercase letters, uppercase letters with Shift/Caps, and UK punctuation including £, @ and double quotes. Test Space, Backspace, Enter and Tab.
+3. Test Ctrl+A/C/V on Windows and Command+A/C/V on macOS. Cycle once/locked/off, including Shift with another modifier, and confirm no modifier remains physically held between selections.
+4. Visit every page. Verify row/key highlighting, reverse movement, row escape, suspension/resume, returning to the first row after a key and Close.
+5. Move the keyboard between the top and bottom. Check a scaled display and a secondary display, including negative coordinates. Verify the keyboard fits the work area and never activates its native windows.
+6. Switch foreground apps, disconnect remote scanning, change display configuration and close Switchify. Verify overlays disappear and owned input is released.
+
+Automated tests use fake input adapters only. Native manual results must be recorded separately; compilation and unit tests do not establish live application compatibility.

src-tauri/src/input.rs

@@ -154,6 +154,14 @@ fn pointer_button(button: MouseButton) -> Button {
 #[cfg(target_os = "windows")]
 fn windows_alphanumeric_key(name: &str) -> Option<Key> {
     Some(match name {
+        #[cfg(target_os = "windows")]
+        "Insert" => Key::Insert,
+        #[cfg(target_os = "windows")]
+        "PrintScreen" => Key::PrintScr,
+        #[cfg(target_os = "windows")]
+        "ScrollLock" => Key::Scroll,
+        #[cfg(target_os = "windows")]
+        "Pause" => Key::Pause,
         "=" => Key::OEMPlus,
         "-" => Key::OEMMinus,
         "[" => Key::OEM4,

src-tauri/src/lib.rs

@@ -24,6 +24,7 @@ mod protocol;
 mod remote_scan;
 mod scan_executor;
 mod scan_host;
+mod scan_keyboard;
 mod scan_menu;
 mod scan_tile;
 mod scan_tree;

src-tauri/src/modifier_overlay_windows.rs

@@ -13,7 +13,7 @@ use windows::Win32::Graphics::Gdi::{
     CreateCompatibleDC, CreateDIBSection, CreateFontW, DeleteDC, DeleteObject, DrawTextW, GetDC,
     GetMonitorInfoW, MonitorFromPoint, ReleaseDC, SelectObject, SetBkMode, SetTextColor,
     AC_SRC_ALPHA, AC_SRC_OVER, BITMAPINFO, BITMAPINFOHEADER, BI_RGB, BLENDFUNCTION,
-    CLIP_DEFAULT_PRECIS, DEFAULT_CHARSET, DEFAULT_QUALITY, DIB_RGB_COLORS, DT_CENTER,
+    CLIP_DEFAULT_PRECIS, DEFAULT_CHARSET, DEFAULT_QUALITY, DIB_RGB_COLORS, DT_CALCRECT, DT_CENTER,
     DT_SINGLELINE, DT_VCENTER, DT_WORDBREAK, FF_DONTCARE, FW_BOLD, HDC, HGDIOBJ, MONITORINFO,
     MONITOR_DEFAULTTONEAREST, OUT_DEFAULT_PRECIS, TRANSPARENT,
 };
@@ -411,6 +411,7 @@ fn present(window: HWND, labels: &[String], layout: &Layout) -> Result<(), Strin
         layout,
         pixmap.data(),
         labels.iter().any(|label| label.contains('\n')),
+        false,
     )
 }
 
@@ -466,6 +467,7 @@ fn present_pixmap_with_text(
     layout: &Layout,
     rgba: &[u8],
     wrap: bool,
+    center_wrapped: bool,
 ) -> Result<(), String> {
     unsafe {
         let screen = ScreenDc(GetDC(None));
@@ -534,8 +536,24 @@ fn present_pixmap_with_text(
         SetBkMode(memory.0, TRANSPARENT);
         SetTextColor(memory.0, COLORREF(0x00ff_ffff));
         for (label, chip) in labels.iter().zip(&layout.chips) {
+            if label.is_empty() {
+                continue;
+            }
             let mut text = label.encode_utf16().collect::<Vec<_>>();
             let mut text_rect = *chip;
+            if center_wrapped {
+                let mut measured = text_rect;
+                let height = DrawTextW(
+                    memory.0,
+                    &mut text,
+                    &mut measured,
+                    DT_CENTER | DT_WORDBREAK | DT_CALCRECT,
+                );
+                let vertical_padding = text_rect.top.min((layout.height - height).max(0) / 2);
+                text_rect.top = vertical_padding;
+                text_rect.bottom = layout.height - vertical_padding;
+                text_rect.top += ((text_rect.bottom - text_rect.top - height) / 2).max(0);
+            }
             let _ = DrawTextW(
                 memory.0,
                 &mut text,
@@ -595,6 +613,35 @@ pub(crate) fn present_scan_tile(
     tile: &crate::scanning::FrameTile,
 ) -> Result<(), String> {
     let pixmap = crate::scan_tile::bitmap(tile)?;
+    if tile.icon == crate::scan_menu::Item::KeyboardKey {
+        let width = pixmap.width() as i32;
+        let height = pixmap.height() as i32;
+        let horizontal_padding = (8.0 * tile.scale).round().max(1.0) as i32;
+        let vertical_padding = (6.0 * tile.scale).round().max(1.0) as i32;
+        let horizontal_padding = horizontal_padding.min((width - 1).max(0) / 2);
+        let vertical_padding = vertical_padding.min((height - 1).max(0) / 2);
+        let layout = Layout {
+            x: tile.rect.x.round() as i32,
+            y: tile.rect.y.round() as i32,
+            width,
+            height,
+            scale: tile.scale * crate::scan_tile::keyboard_font_size(tile) / FONT_SIZE,
+            chips: vec![RECT {
+                left: horizontal_padding,
+                top: vertical_padding,
+                right: width - horizontal_padding,
+                bottom: height - vertical_padding,
+            }],
+        };
+        return present_pixmap_with_text(
+            window,
+            std::slice::from_ref(&tile.text),
+            &layout,
+            pixmap.data(),
+            true,
+            true,
+        );
+    }
     let size = pixmap.width() as i32;
     let layout = Layout {
         x: tile.rect.x.round() as i32,
@@ -615,6 +662,7 @@ pub(crate) fn present_scan_tile(
         &layout,
         pixmap.data(),
         true,
+        false,
     )
 }
 
@@ -646,6 +694,34 @@ pub(crate) fn present_scan_prompt(
 
 #[cfg(test)]
 mod tests {
+    #[test]
+    fn empty_keyboard_backdrop_label_never_reaches_native_text_drawing() {
+        let layout = super::Layout {
+            x: 0,
+            y: 0,
+            width: 100,
+            height: 50,
+            scale: 1.0,
+            chips: vec![windows::Win32::Foundation::RECT {
+                left: 8,
+                top: 6,
+                right: 92,
+                bottom: 44,
+            }],
+        };
+        let pixels = vec![255; 100 * 50 * 4];
+        for wrapped in [false, true] {
+            let result = super::present_pixmap_with_text(
+                windows::Win32::Foundation::HWND::default(),
+                &[String::new()],
+                &layout,
+                &pixels,
+                wrapped,
+                wrapped,
+            );
+            assert!(result.is_err());
+        }
+    }
     use super::*;
     use windows::Win32::Foundation::{LPARAM, WPARAM};
     use windows::Win32::UI::WindowsAndMessaging::{
@@ -800,6 +876,24 @@ mod tests {
     #[test]
     fn native_window_render_smoke_becomes_visible_and_hides() {
         let mut host = NativeHost::new().unwrap();
+        let keyboard = crate::scan_keyboard::Keyboard::new(false);
+        let frame = keyboard.frame(
+            crate::scanning::Rect {
+                x: 0.0,
+                y: 0.0,
+                width: 1280.0,
+                height: 720.0,
+            },
+            1.0,
+            crate::scanning::ScannerColor::default(),
+        );
+        for tile in &frame.tiles {
+            super::present_scan_tile(host.window, tile).unwrap();
+            unsafe {
+                assert!(IsWindowVisible(host.window).as_bool());
+            }
+        }
+        host.hide();
         host.render(&["Ctrl".into(), "Shift".into()]).unwrap();
         unsafe {
             assert!(IsWindowVisible(host.window).as_bool());

src-tauri/src/overlay_macos.rs

@@ -228,6 +228,20 @@ pub(crate) fn image_from_rgba(
     height: usize,
     logical_size: f64,
 ) -> Result<Retained<NSImage>, String> {
+    image_from_rgba_rect(rgba, width, height, logical_size, logical_size)
+}
+
+pub(crate) fn image_from_rgba_rect(
+    rgba: &[u8],
+    width: usize,
+    height: usize,
+    logical_width: f64,
+    logical_height: f64,
+) -> Result<Retained<NSImage>, String> {
+    if !logical_height.is_finite() || logical_height <= 0.0 {
+        return Err("Invalid overlay height.".into());
+    }
+    let logical_size = logical_width;
     let layout = MacBitmapLayout::new(rgba.len(), width, height, logical_size)?;
     let representation = unsafe {
         NSBitmapImageRep::initWithBitmapDataPlanes_pixelsWide_pixelsHigh_bitsPerSample_samplesPerPixel_hasAlpha_isPlanar_colorSpaceName_bitmapFormat_bytesPerRow_bitsPerPixel(
@@ -251,13 +265,13 @@ pub(crate) fn image_from_rgba(
     }
     representation.setSize(NSSize {
         width: layout.logical_size,
-        height: layout.logical_size,
+        height: logical_height,
     });
     let image = NSImage::initWithSize(
         NSImage::alloc(),
         NSSize {
             width: layout.logical_size,
-            height: layout.logical_size,
+            height: logical_height,
         },
     );
     image.addRepresentation(&representation as &NSImageRep);

src-tauri/src/point_scan_runtime.rs

@@ -11,6 +11,26 @@ use tauri::{AppHandle, Manager};
 pub struct Environment {
     display: Display,
     foreground: usize,
+    keyboard_area: Rect,
+    handoff: Option<FocusHandoff>,
+}
+#[derive(Clone)]
+struct FocusHandoff {
+    target: usize,
+    ready_at: std::time::Instant,
+}
+impl FocusHandoff {
+    fn verify(&self, now: std::time::Instant, foreground: usize) -> Result<bool, String> {
+        if now < self.ready_at {
+            return Ok(false);
+        }
+        if foreground != self.target {
+            return Err(
+                "The selected target did not receive focus. Please select it again.".into(),
+            );
+        }
+        Ok(true)
+    }
 }
 pub struct PointScan;
 pub type Controller = scanning_runtime::Controller<PointScan>;
@@ -59,13 +79,43 @@ impl Adapter for PointScan {
         crate::scan_executor::cleanup()?;
         crate::point_scan_prepare(app)
     }
-    fn activate(app: &AppHandle, request: Request) -> Result<(), String> {
+    fn activate(app: &AppHandle, request: Request) -> Result<Option<Environment>, String> {
         crate::point_scan_ready(app)?;
         match request {
+            Request::OpenKeyboard { point: Some(point) } => {
+                let target = crate::scan_host::target_at(point)?;
+                crate::scan_executor::activate(request)?;
+                let (_, mut environment) =
+                    new_engine(app, app.state::<Controller>().view().config)?;
+                // Allow the one requested click to establish focus before accepting
+                // any key. No typing or scan advancement occurs during this handoff.
+                environment.handoff = Some(FocusHandoff {
+                    target,
+                    ready_at: std::time::Instant::now() + std::time::Duration::from_millis(150),
+                });
+                return Ok(Some(environment));
+            }
             Request::Setting(setting) => scanning_runtime::update_point_setting(app, setting),
             Request::Display(next) => scanning_runtime::restart_point_on_display(app, next),
             request => crate::scan_executor::activate(request),
         }
+        .map(|()| None)
+    }
+    fn settle_environment(
+        app: &AppHandle,
+        environment: Option<&mut Environment>,
+    ) -> Result<bool, String> {
+        if let Some(environment) = environment {
+            if let Some(handoff) = &environment.handoff {
+                crate::point_scan_ready(app)?;
+                if !handoff.verify(std::time::Instant::now(), crate::scan_host::foreground()?)? {
+                    return Ok(false);
+                }
+                environment.foreground = handoff.target;
+                environment.handoff = None;
+            }
+        }
+        Ok(true)
     }
     fn cleanup(_app: &AppHandle) -> Result<(), String> {
         crate::scan_executor::cleanup()
@@ -81,7 +131,7 @@ fn new_engine(app: &AppHandle, config: Config) -> Result<(Workflow, Environment)
     } else {
         1.0
     };
-    let e = Workflow::new(
+    let mut e = Workflow::new(
         config.point(),
         Rect {
             x: display.x.into(),
@@ -91,11 +141,20 @@ fn new_engine(app: &AppHandle, config: Config) -> Result<(Workflow, Environment)
         },
         units,
     )?;
+    let keyboard_area = crate::scan_host::work_area(Rect {
+        x: display.x.into(),
+        y: display.y.into(),
+        width: display.width.into(),
+        height: display.height.into(),
+    })?;
+    e.set_keyboard_area(keyboard_area);
     Ok((
         e,
         Environment {
             display,
             foreground: crate::scan_host::foreground()?,
+            keyboard_area,
+            handoff: None,
         },
     ))
 }
@@ -109,6 +168,36 @@ fn validate_display(app: &AppHandle, display: Option<&Environment>) -> Result<()
         if !displays.contains(&expected.display) {
             return Err("Display geometry changed. Scanning restarts.".into());
         }
+        let d = &expected.display;
+        if crate::scan_host::work_area(Rect {
+            x: d.x.into(),
+            y: d.y.into(),
+            width: d.width.into(),
+            height: d.height.into(),
+        })? != expected.keyboard_area
+        {
+            return Err("Display work area changed. Scanning restarts.".into());
+        }
     }
     Ok(())
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn focus_handoff_waits_and_only_accepts_the_clicked_target() {
+        let start = std::time::Instant::now();
+        let handoff = FocusHandoff {
+            target: 42,
+            ready_at: start + std::time::Duration::from_millis(150),
+        };
+        assert_eq!(handoff.verify(start, 7), Ok(false));
+        assert_eq!(handoff.verify(start, 42), Ok(false));
+        assert_eq!(handoff.verify(handoff.ready_at, 42), Ok(true));
+        // A failed activation and an unrelated app stealing focus both fail closed.
+        assert!(handoff.verify(handoff.ready_at, 7).is_err());
+        assert!(handoff.verify(handoff.ready_at, 99).is_err());
+    }
+}