Skip to content

Commit 1c5101b

Browse files
committed
Mirror text fields as real inputs, so an agent can type
Running playwright-mcp against a device page found what our own examples could not: browser_type fails with "Element is not an <input>, <textarea> or [contenteditable]". Every field was a div, and fill() — which browser_type calls — refuses anything else. An agent could read the app perfectly and then could not enter a character, which it reports as a broken page. The gap survived because every example shipped here drives a field with click + keyboard.type, so the path an agent actually takes was never exercised. Text fields are now real <input> elements holding the device's own contents. Typed keystrokes still flow through keydown; a value set in one go arrives as a single input event instead, so the keystrokes that would have produced it are reconstructed by diffing against what the device was last told — backspaces for what went, characters for what came. fill() and browser_type never click, so the browser's focus and the device's caret would otherwise sit in different places and the text would land wherever the device was last focused. Focus therefore taps the field — but only when a pointer did not already do it, since a click plus a focus tap is a double tap, which selects a word rather than placing a caret. That was measured: "devicelab" arrived as "vxdicelab", also because each edit waited out the focus settle on its own timer and the differing delays let a later keystroke overtake an earlier one. Edits now run in a chain. This changes the contract for field contents, which live in value rather than text: assertions move from toHaveText to toHaveValue. Better now than after a release, and it is the API a web developer would have reached for first.
1 parent db51734 commit 1c5101b

7 files changed

Lines changed: 177 additions & 12 deletions

File tree

‎README.md‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,8 @@ drive it with no mobile-specific code and no Appium:
1111

1212
```ts
1313
await page.goto(`/device/${udid}?app=dev.devicelab.testhive`);
14-
await page.getByRole('textbox', { name: 'Username' }).click();
15-
await page.keyboard.type('devicelab');
14+
await page.getByRole('textbox', { name: 'Username' }).fill('devicelab');
15+
await page.getByRole('textbox', { name: 'Password' }).fill('robustest');
1616
await page.getByRole('button', { name: 'Sign In' }).click();
1717
await expect(page.getByText('Hello, devicelab!')).toBeVisible();
1818
```
@@ -84,6 +84,11 @@ Arm **Assert** and the next tap records an assertion instead of tapping. A point
8484
to nothing addressable is refused rather than recorded as a coordinate — "there are pixels here"
8585
is not worth putting in a flow.
8686
87+
**Text fields are real `<input>` elements**, so `fill()`, `inputValue()` and `toHaveValue()` work
88+
as they would on any page — and so does the `browser_type` an AI agent reaches for first. A
89+
field's contents live in its value, not its text, which is the one place the mirror departs from
90+
"every native node is a div".
91+
8792
**Your own tests** point at `/device/{udid}?app={bundleId}` and use ordinary selectors.
8893
`examples/` has runnable Playwright, Cypress and Puppeteer projects, and `examples/captured/`
8994
holds flows recorded through the console — including one from a Flutter app and one from React

‎examples/playwright/tests/ownership-live.spec.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,9 +31,9 @@ test('a second driver is refused on a real device', async ({ browser, request })
3131
await expect(async () => {
3232
await field.click();
3333
await first.keyboard.press('q');
34-
await expect(field).toHaveText(/q/);
34+
await expect(field).toHaveValue(/q/);
3535
}).toPass({ timeout: 25_000 });
36-
console.log('FIRST PAGE still drives, field =', JSON.stringify(await field.textContent()));
36+
console.log('FIRST PAGE still drives, field =', JSON.stringify(await field.inputValue()));
3737

3838
await second.close();
3939
await first.close();

‎examples/playwright/tests/testhive-login.spec.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,11 +37,11 @@ test('logs into TestHive on a real simulator', async ({ page }) => {
3737
// the effect itself.
3838
await expect(async () => {
3939
await username.click();
40-
for (let i = 0; i < 40 && (await username.textContent()); i++) {
40+
for (let i = 0; i < 40 && (await username.inputValue()); i++) {
4141
await page.keyboard.press('Backspace');
4242
}
4343
await page.keyboard.press('x');
44-
await expect(username).toHaveText('x');
44+
await expect(username).toHaveValue('x');
4545
}).toPass({ timeout: 30_000 });
4646
await page.keyboard.press('Backspace');
4747

‎examples/playwright/tests/testhive-selectors.spec.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,11 +41,11 @@ test('drives TestHive by role, label and text', async ({ page }) => {
4141
// the effect itself.
4242
await expect(async () => {
4343
await username.click();
44-
for (let i = 0; i < 40 && (await username.textContent()); i++) {
44+
for (let i = 0; i < 40 && (await username.inputValue()); i++) {
4545
await page.keyboard.press('Backspace');
4646
}
4747
await page.keyboard.press('x');
48-
await expect(username).toHaveText('x');
48+
await expect(username).toHaveValue('x');
4949
}).toPass({ timeout: 30_000 });
5050
await page.keyboard.press('Backspace');
5151

‎internal/web/static/device.js‎

Lines changed: 136 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -188,6 +188,12 @@ function syncNode(el, node, anchorFrame) {
188188
// which makes it unusable as a pointer-events signal.
189189
el.setAttribute("data-dd-enabled", node.enabled === false ? "false" : "true");
190190
el.setAttribute("data-dd-hittable", node.hittable ? "true" : "false");
191+
// An <input> holds its contents in .value and can have no children, so
192+
// it takes neither the text node below nor anything nested.
193+
if (el.tagName === "INPUT") {
194+
syncField(el, node);
195+
return;
196+
}
191197
// Text for getByText: label, else value, painted transparent. Kept in
192198
// a dedicated leading text node — assigning textContent would destroy
193199
// the nested child elements.
@@ -203,6 +209,18 @@ function syncNode(el, node, anchorFrame) {
203209
}
204210
}
205211

212+
// syncField mirrors the device's own contents into the input — except
213+
// while it holds focus. The tree lags a keystroke behind, so writing its
214+
// value back mid-edit would delete characters as fast as they are typed.
215+
// ddValue records what the device has already been told, and is the
216+
// baseline the next edit is diffed against.
217+
function syncField(el, node) {
218+
const value = node.value || "";
219+
if (el !== document.activeElement && el.value !== value) el.value = value;
220+
el.dataset.ddValue = el.value;
221+
setOrRemove(el, "placeholder", node.placeholder || "");
222+
}
223+
206224
// twinOf returns the immediately preceding rendered sibling when it
207225
// occupies exactly the same frame, meaning the two nodes describe one
208226
// control. Only an exact match counts: a merely overlapping element is a
@@ -226,14 +244,33 @@ function mirrorable(node) {
226244
return !!(node.identifier || node.label || node.value || ROLES[node.type]);
227245
}
228246

229-
// Reuse the keyed element from a previous render when one exists.
230-
function acquireEl(existing, key) {
247+
// A text field is mirrored as a real <input>, every other node as a div.
248+
// The reason is what the tools refuse: fill(), and the browser_type an AI
249+
// agent reaches for first, both reject anything that is not an <input>,
250+
// <textarea> or [contenteditable]. Against a div-based field an agent
251+
// cannot type at all — it reports the page as broken and stops. Every
252+
// example shipped here drives a field with click + keyboard.type, which
253+
// is why the gap survived until an agent was actually pointed at it.
254+
//
255+
// The node's type is part of its key, so a node that changes type gets a
256+
// new key and therefore a correctly-tagged new element.
257+
function acquireEl(existing, key, node) {
231258
const found = existing.get(key);
232259
if (found) {
233260
existing.delete(key);
234261
return found;
235262
}
236-
const el = document.createElement("div");
263+
const el = document.createElement(FIELD_TYPES.has(node.type) ? "input" : "div");
264+
if (el.tagName === "INPUT") {
265+
// type=text even for secure fields: the role attribute below already
266+
// says textbox, and type=password invites the browser's own password
267+
// manager into a page that is meant to contain nothing but the app.
268+
// The device reports a secure field's contents as bullets, so the
269+
// value stays honest about its length either way.
270+
el.type = "text";
271+
el.autocomplete = "off";
272+
el.spellcheck = false;
273+
}
237274
el.setAttribute("data-dd-node", "");
238275
el.setAttribute("data-dd-key", key);
239276
return el;
@@ -284,7 +321,7 @@ function renderMirror(nodes) {
284321
// something clickable.
285322
const anchor = twinOf(parentAnchor, node) || parentAnchor;
286323
const key = nodeKey(node, anchor.key, counts);
287-
const el = acquireEl(existing, key);
324+
const el = acquireEl(existing, key, node);
288325
syncNode(el, node, anchor.frame);
289326
// Append-or-move keeps DOM order tracking native order; moving
290327
// (including across parents) preserves element identity.
@@ -386,6 +423,9 @@ function normalized(event) {
386423
let pointerDown = false;
387424
mirror.addEventListener("pointerdown", (e) => {
388425
pointerDown = true;
426+
// Stamped on the way down, not up: focus lands on pointerdown, so the
427+
// focus handler must already be able to see that a click caused it.
428+
pointerAt = Date.now();
389429
// Synthetic events (Cypress, jsdom) may carry no capturable pointerId;
390430
// capture is an optimization for drags, never a precondition.
391431
try { mirror.setPointerCapture(e.pointerId); } catch {}
@@ -401,19 +441,111 @@ mirror.addEventListener("pointermove", (e) => {
401441
mirror.addEventListener("pointerup", (e) => {
402442
if (!pointerDown) return;
403443
pointerDown = false;
444+
pointerAt = Date.now();
404445
const { x, y } = normalized(e);
405446
input.send(touchFrame(PHASE.up, x, y));
406447
noteActivity();
407448
});
408449

409450
document.addEventListener("keydown", (e) => {
451+
// A focused mirror field types itself: the browser writes the character
452+
// into the <input>, the input listener below diffs the value and sends
453+
// the keystroke. Taking this path too would send it twice, and the
454+
// preventDefault would stop the value ever changing in the first place.
455+
// Keys that leave the value alone still belong here.
456+
if (editsAField(e)) return;
410457
const frame = keyEventFrame(e);
411458
if (!frame) return;
412459
e.preventDefault();
413460
input.send(frame);
414461
noteActivity();
415462
});
416463

464+
// ---------- typing into a mirrored field ----------
465+
466+
// editsAField reports whether this keystroke will change a mirrored
467+
// input's value, and so will arrive as an input event instead.
468+
function editsAField(e) {
469+
const el = e.target;
470+
return (
471+
el instanceof HTMLInputElement &&
472+
el.hasAttribute("data-dd-node") &&
473+
(e.key.length === 1 || e.key === "Backspace" || e.key === "Delete")
474+
);
475+
}
476+
477+
// tapField taps the device where el sits. Focus is not enough on its own:
478+
// fill() and browser_type focus the element and write the value without
479+
// ever clicking, so the device's own field would still be unfocused and
480+
// every keystroke would land on whatever was focused before.
481+
function tapField(el) {
482+
const r = el.getBoundingClientRect();
483+
const m = mirror.getBoundingClientRect();
484+
const x = (r.left + r.width / 2 - m.left) / m.width;
485+
const y = (r.top + r.height / 2 - m.top) / m.height;
486+
input.send(touchFrame(PHASE.down, x, y));
487+
input.send(touchFrame(PHASE.up, x, y));
488+
}
489+
490+
// The device needs a moment to raise its keyboard and place the caret
491+
// after that tap. Text sent inside the window would be typed into a field
492+
// that is not yet listening, so an edit that follows a focus waits it out.
493+
const FOCUS_SETTLE_MS = 400;
494+
// A click focuses the field on its way through, and the pointer handlers
495+
// have already tapped the device at that point. Tapping again on focus
496+
// would make it a double tap, which selects a word instead of placing a
497+
// caret — measured as scrambled text when a test clicked and then typed.
498+
const POINTER_FOCUS_MS = 600;
499+
let focusTapAt = 0;
500+
let pointerAt = 0;
501+
502+
mirror.addEventListener("focusin", (e) => {
503+
if (!(e.target instanceof HTMLInputElement)) return;
504+
if (Date.now() - pointerAt < POINTER_FOCUS_MS) return;
505+
tapField(e.target);
506+
focusTapAt = Date.now();
507+
noteActivity();
508+
});
509+
510+
// Edits run one at a time, in order. Each used to wait out the settle
511+
// window on its own timer, and because those delays differed, a later
512+
// keystroke could fire before an earlier one — "devicelab" reached the
513+
// device as "vxdicelab". A chain keeps the order the typist produced.
514+
let editChain = Promise.resolve();
515+
516+
mirror.addEventListener("input", (e) => {
517+
const el = e.target;
518+
if (!(el instanceof HTMLInputElement)) return;
519+
const before = el.dataset.ddValue ?? "";
520+
const after = el.value;
521+
el.dataset.ddValue = after;
522+
editChain = editChain.then(async () => {
523+
const wait = Math.max(0, FOCUS_SETTLE_MS - (Date.now() - focusTapAt));
524+
if (wait > 0) await new Promise((r) => setTimeout(r, wait));
525+
sendEdit(before, after);
526+
});
527+
noteActivity();
528+
});
529+
530+
// sendEdit turns a value change into the keystrokes that would produce
531+
// it: backspaces for what was removed, characters for what was added.
532+
// fill() replaces the whole value in a single event, so diffing is the
533+
// only way to know what the device actually has to be told.
534+
function sendEdit(before, after) {
535+
let shared = 0;
536+
while (shared < before.length && shared < after.length &&
537+
before[shared] === after[shared]) {
538+
shared++;
539+
}
540+
for (let n = before.length - shared; n > 0; n--) {
541+
input.send(keyFrame(0, KEY_USAGE.Backspace));
542+
}
543+
for (const ch of after.slice(shared)) {
544+
const frame = keyFrameForChar(ch);
545+
if (frame) input.send(frame);
546+
}
547+
}
548+
417549
// ---------- scripting escape hatch ----------
418550

419551
// Gestures DOM events cannot express, callable from page.evaluate().

‎internal/web/static/input-common.js‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,29 @@ const KEY_USAGE = (() => {
4141
return map;
4242
})();
4343

44+
// Characters a US keyboard only reaches with shift held, mapped to the
45+
// unshifted key that produces them.
46+
const SHIFTED_CHARS = {
47+
"!": "1", "@": "2", "#": "3", "$": "4", "%": "5", "^": "6", "&": "7",
48+
"*": "8", "(": "9", ")": "0", _: "-", "+": "=", "{": "[", "}": "]",
49+
"|": "\\", ":": ";", '"': "'", "~": "`", "<": ",", ">": ".", "?": "/",
50+
};
51+
const MOD_SHIFT = 0x02;
52+
53+
// keyFrameForChar encodes one character of text as the keystroke that
54+
// types it. A value set in one go — fill(), or an agent's browser_type —
55+
// arrives as a finished string rather than as key events, so the
56+
// keystrokes have to be reconstructed from it. Returns null for anything
57+
// with no HID equivalent: an emoji has no key that types it, and sending
58+
// nothing is better than sending the wrong one.
59+
function keyFrameForChar(ch) {
60+
if (ch >= "A" && ch <= "Z") return keyFrame(MOD_SHIFT, KEY_USAGE[ch.toLowerCase()]);
61+
const shifted = SHIFTED_CHARS[ch];
62+
if (shifted) return keyFrame(MOD_SHIFT, KEY_USAGE[shifted]);
63+
const usage = KEY_USAGE[ch];
64+
return usage ? keyFrame(0, usage) : null;
65+
}
66+
4467
// keyEventFrame encodes a KeyboardEvent, or returns null when the key
4568
// has no HID equivalent (the caller then leaves the event alone).
4669
function keyEventFrame(e) {

‎internal/web/web_test.go‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,11 @@ func TestConsoleHeaderWraps(t *testing.T) {
7979
// snapshot beside the app's own elements — indistinguishable from them —
8080
// and an agent told to "tap the button" could drive DeviceDeck instead
8181
// of the app under test. The console is where our controls belong.
82+
// This checks the page as served. The mirror does create <input> elements
83+
// at runtime, one per native text field, and those are the point — they
84+
// are the app's controls, not ours. What must never appear is a control
85+
// authored here: an agent cannot tell our button from the app's, and
86+
// "tap the button" would drive DeviceDeck instead of the app under test.
8287
func TestDevicePageCarriesNoControlsOfOurOwn(t *testing.T) {
8388
body := get(t, "/device/EB69B42A-4763-4A33-AF0F-CD233F721951").Body.String()
8489
// Everything between <body> and the scripts is the test target.

0 commit comments

Comments
 (0)