feat(harmonyos): report the keyboard band in snapshot responses

docs/evidence/harmonyos-keyboard-band-2026-11-08.md

@@ -0,0 +1,139 @@
+# HarmonyOS keyboard band fact, live, 2026-11-08 (fixed run 2026-11-09)
+
+Evidence for #4285 (report the keyboard band in snapshot responses), re-run at the
+rebased head after the review found the published frame wrong. The window-manager
+table on this build never carries the key plane: the panel row reports the whole
+display whether the keyboard shows or hides. The producer therefore publishes
+visibility, and publishes a band frame only when the table proves the rect covers
+under a 0.9 share of the display's area (`DISPLAY_SPANNING_RECT_SHARE`: at or above
+that share the panel is the window manager's full-surface container, not a key
+plane). This run shows that contract live, with the guard regressions
+the old full-screen frame caused now retired. Fix commit: `2878b86e4`.
+
+## Device
+
+- HUAWEI 畅享 90 Plus, model `JUY-AL50` (`param get const.product.model`;
+  the earlier revision of this file wrote `JDY-AL50`, which is a typo),
+  HarmonyOS 7.0.0.111 (SP10C00E111R4P3), API 26
+- Serial `77TYD26826028730`, USB
+- Surface: system Settings → search field on the Settings home page, then the
+  search page. One character typed by the in-band press, gone with the page
+  (search-page state is transient). No user data read or written.
+
+## What the window table actually holds
+
+`hidumper -s WindowManagerService -a -a`, keyboard raised (Settings search field
+focused). Both keyboard windows hold the full display rect; only the z-order
+column moves:
+
+```
+softKeyboard1        0         6100    16    2105 102  0    103  0           [ 0    0    1080 2406 ] [ 0       0       ] [ 1       1       0.5     0.5     ]
+SCBKeyboardPanel     0         5714    17    2134 1    0    102  0           [ 0    0    1080 2406 ] [ 0       0       ] [ 1       1       0.5     0.5     ]
+```
+
+After one Back dismisses the IME, the same rows flip only in z-order:
+
+```
+softKeyboard1        0         6100    16    2105 102  0    -1   0           [ 0    0    1080 2406 ] [ 0       0       ] [ 1       1       0.5     0.5     ]
+SCBKeyboardPanel     0         5714    17    2134 1    0    -1   0           [ 0    0    1080 2406 ] [ 0       0       ] [ 1       1       0.5     0.5     ]
+```
+
+The key plane on this build sits at roughly y 1630–2325 in display pixels (the
+keys-only crop below starts at y 1600); the IME window surface covers the display.
+No public source reports that inner band: the window table carries window-surface
+rects, and `hidumper -s InputMethodService -a '-a'` lists installed IMEs
+(`小艺输入法`) with no geometry. A rect the table cannot prove smaller than the
+display is the container, not a band — publishing it as `visible` made the tap
+guard read every on-screen point as keyboard-covered (the `@e20` refusal in the
+previous revision of this file was that false refusal, presented as a feature).
+
+## The fact through a raise and a dismiss (branch CLI, head `2878b86e4`)
+
+Settings home, keyboard hidden:
+
+```json
+"keyboard": {"kind": "visibility-only", "visible": false}
+```
+
+`press 540 381` on the search field raises the IME. The fact states visibility
+and publishes no frame:
+
+```json
+"keyboard": {"kind": "visibility-only", "visible": true}
+```
+
+Keyboard raised, portrait (keys-only crop; the strip above it holds the app's
+search page and is left out of the repo):
+
+![keyboard keys with the IME raised](harmonyos-keyboard-band-keys-2026-11-09.jpg)
+
+One Back hides the IME; the fact returns to `{"kind": "visibility-only",
+"visible": false}`. A second Back then leaves the search page — the dismissal
+choreography is one Back for the keyboard, one for the page, and the previous
+revision's "two Back keys" line described the same two steps as one event.
+
+## The guard at the new head
+
+A ref captured with the keyboard up, pressed above the band, succeeds — this is
+the class the display-spanning frame refused (`Ref @e20 is behind the visible
+keyboard`):
+
+```
+$ agent-device press @e2 --platform harmonyos --serial 77TYD26826028730
+Tapped @e2 (291, 192)
+```
+
+A bare coordinate press inside the band dispatches (the guard stays fail-open on
+`visibility-only`; the press landed on a key and typed one character into the
+transient search field):
+
+```
+$ agent-device press 540 1700 --platform harmonyos --serial 77TYD26826028730
+Tapped (540, 1700)
+```
+
+## Probe cost
+
+The probe is one `hidumper` round trip, measured 0.129 s alone on this device.
+`snapshot --json` wall clock, three runs per CLI on the same page: with the probe
+(built from #4285) 1.593 / 1.527 / 1.578 s; without it (a build from upstream
+main) 1.976 / 1.437 / 1.495 s. Medians differ by 0.083 s, inside the run-to-run
+spread; the first no-probe run shows the cold-start tail. The probe runs on every
+HarmonyOS capture by design — every capture's keyboard feeds the snapshot state
+later taps read.
+
+## Landscape
+
+Not run. The device exposes no public rotation channel — `uitest` has no rotate
+subcommand, and agent-device's own `setHarmonyOrientation` answers
+UNSUPPORTED_OPERATION on HarmonyOS — so the landscape run needs a physical
+rotation this session cannot perform. The rect-share rule is
+orientation-independent: it compares the panel rect against the largest rect in
+the same table, whatever the display's aspect, and the real-row fixtures pin both
+states.
+
+## Landing order
+
+Proposed order in #4318 stands: #4292, #4290, #4293, #4287, then this PR, then
+#4239. This PR lands fifth, rebased onto main (`b24ecf209`) with the snapshot
+help conflict resolved.
+
+## Reproduce
+
+```sh
+# head 2878b86e4, built in the PR worktree
+node bin/agent-device.mjs open com.huawei.hmos.settings --activity com.huawei.hmos.settings.MainAbility \
+  --platform harmonyos --serial 77TYD26826028730
+node bin/agent-device.mjs snapshot --json --platform harmonyos --serial 77TYD26826028730   # visibility-only false
+node bin/agent-device.mjs press 540 381 --platform harmonyos --serial 77TYD26826028730      # search field
+node bin/agent-device.mjs snapshot --json --platform harmonyos --serial 77TYD26826028730   # visibility-only true, no frame
+node bin/agent-device.mjs snapshot -i --json --platform harmonyos --serial 77TYD26826028730 # refs
+node bin/agent-device.mjs press @e2 --platform harmonyos --serial 77TYD26826028730          # above band: succeeds
+node bin/agent-device.mjs press 540 1700 --platform harmonyos --serial 77TYD26826028730     # in band: dispatches
+node bin/agent-device.mjs back --platform harmonyos --serial 77TYD26826028730               # IME hides
+node bin/agent-device.mjs snapshot --json --platform harmonyos --serial 77TYD26826028730   # visibility-only false
+hidumper -s WindowManagerService -a -a   # on the device: the raw rows quoted above
+```
+
+CI cannot exercise a HarmonyOS device; this file is the device-side proof of the
+parse, the rect-share rule, and the guard behavior.

packages/command-surface/src/capture/snapshot.ts

@@ -75,7 +75,7 @@ export const snapshotCommandFacet = defineCommandFacet({
   name: SNAPSHOT_COMMAND_NAME,
   text: {
     summary: 'Capture or diff the accessibility tree',
-    cliDetail: `Scoped output (--scope) renders every returned ref with its ref generation pin (@eN~s<refsGeneration>) and one Refs generation line, so a pasted ref targets the ref frame that issued it; --json keeps plain refs plus the response-level refsGeneration, and --raw keeps byte-for-byte output. Regular iOS snapshots clip overflowing elements to their visible area and omit fully outside elements. --json reports clipping counts in snapshotQualityNotes with code regular-geometry-clipped; --raw retains reported frames. --observe-only is local iOS snapshot-only: an already-ready compatible runner reads the session app without activation or foreground repair, refuses unavailable observation or a readable system alert over the app, and reports XCUIApplication state rather than verified foreground ownership (#2696). Regular targetActivation disclosure stays unchanged (#2682, #2694). Repeated equivalent unfiltered Android snapshots return a compact unchanged acknowledgement. Use --force-full to re-emit the tree; --json and --raw retain full output. Android Appium JSON snapshots publish a display viewport only when measurements before and after the tree agree, independent of tree extent or keyboard content resizing. On Android, JSON value carries field content rather than static labels; saved selectors and replay scripts using value= for static labels must use label= instead; a reported showing hint means value is empty, and text equal to the placeholder is preserved when hint state is unknown. For iOS raw-coordinate fallback after a no-op ref press, inspect rects with snapshot -i --json, press the rect center, then verify with diff snapshot -i or snapshot --diff. iOS backend capability contract: ${snapshotBackendCapabilityHelp}.`,
+    cliDetail: `Scoped output (--scope) renders every returned ref with its ref generation pin (@eN~s<refsGeneration>) and one Refs generation line, so a pasted ref targets the ref frame that issued it; --json keeps plain refs plus the response-level refsGeneration, and --raw keeps byte-for-byte output. Regular iOS snapshots clip overflowing elements to their visible area and omit fully outside elements. --json reports clipping counts in snapshotQualityNotes with code regular-geometry-clipped; --raw retains reported frames. --observe-only is local iOS snapshot-only: an already-ready compatible runner reads the session app without activation or foreground repair, refuses unavailable observation or a readable system alert over the app, and reports XCUIApplication state rather than verified foreground ownership (#2696). Regular targetActivation disclosure stays unchanged (#2682, #2694). Repeated equivalent unfiltered Android snapshots return a compact unchanged acknowledgement. Use --force-full to re-emit the tree; --json and --raw retain full output. Android Appium JSON snapshots publish a display viewport only when measurements before and after the tree agree, independent of tree extent or keyboard content resizing. On Android, JSON value carries field content rather than static labels; saved selectors and replay scripts using value= for static labels must use label= instead; a reported showing hint means value is empty, and text equal to the placeholder is preserved when hint state is unknown. HarmonyOS reads the on-screen keyboard's own system window and publishes the keyboard field of --json output: the field carries a band rect only when the window table proves that rect smaller than the display, current builds state visibility only, and a failed window probe publishes unmeasurable, never a guessed band. For iOS raw-coordinate fallback after a no-op ref press, inspect rects with snapshot -i --json, press the rect center, then verify with diff snapshot -i or snapshot --diff. iOS backend capability contract: ${snapshotBackendCapabilityHelp}.`,
   },
   metadata: snapshotCommandMetadata,
   run: (client, input) => client.capture.snapshot(input),

packages/platform-harmonyos/src/__tests__/keyboard-band.test.ts

@@ -0,0 +1,212 @@
+import assert from 'node:assert/strict';
+import fs from 'node:fs';
+import { beforeEach, test, vi } from 'vitest';
+
+const { runHarmonyHdc, runHarmonyShell } = vi.hoisted(() => ({
+  runHarmonyHdc: vi.fn(),
+  runHarmonyShell: vi.fn(),
+}));
+
+vi.mock('../hdc.ts', () => ({ runHarmonyHdc, runHarmonyShell }));
+
+import { harmonyKeyboardBandFromWindowDump, readHarmonyKeyboardBand } from '../keyboard-band.ts';
+import { snapshotHarmony } from '../snapshot.ts';
+
+const DEVICE = {
+  platform: 'harmonyos' as const,
+  id: 'harmony-1',
+  name: 'HarmonyOS test device',
+  kind: 'device' as const,
+  target: 'mobile' as const,
+  booted: true,
+};
+
+const WINDOW_TABLE_HEADER = [
+  '-------------------------------[ability]-------------------------------',
+  '----------------------------------WindowManagerService----------------------------------',
+  '-------------------------------------ScreenGroup 0-------------------------------------',
+  'WindowName           DisplayId Pid     WinId Type Mode Flag ZOrd Orientation [ x    y    w    h    ] [ OffsetX OffsetY ] [ ScaleX  ScaleY  PivotX  PivotY  ]',
+  'SCBScenePanel12      0         5714    4     2109 1    0    100  0           [ 0    0    1080 2406 ] [ 0       0       ] [ 1       1       0.5     0.5     ]',
+  'app-under-test0      0         8131    38    1    1    0    101  0           [ 0    0    1080 2406 ] [ 0       0       ] [ 1       1       0.5     0.5     ]',
+  '---------------------------------------------------------------------------------------',
+  'SCBStatusBar17       0         5714    23    2108 1    0    -1   0           [ 0    0    1080 108  ] [ 0       0       ] [ 1       1       0       0       ]',
+].join('\n');
+
+/** The measured hidden keyboard panel: the zero rect, not mere presence, says "hidden". */
+const HIDDEN_KEYBOARD_TABLE = [
+  ...WINDOW_TABLE_HEADER.split('\n'),
+  'SCBKeyboardPanel     0         5714    17    2134 1    0    -1   0           [ 0    0    0    0    ] [ 0       0       ] [ 1       1       0       0       ]',
+].join('\n');
+
+/**
+ * The real JUY-AL50 7.0.0.111 rows, captured verbatim from
+ * `hidumper -s WindowManagerService -a -a` with the Settings search field focused and the IME
+ * raised, then after a Back dismissal. The panel and the IME's own `softKeyboard1` window both
+ * hold the full display rect in both states; only the z-order column moves (102/103 up,
+ * both -1 hidden). No column in the public table carries the key plane.
+ */
+const REAL_KEYBOARD_UP_ROWS = [
+  'softKeyboard1        0         6100    16    2105 102  0    103  0           [ 0    0    1080 2406 ] [ 0       0       ] [ 1       1       0.5     0.5     ]',
+  'SCBKeyboardPanel     0         5714    17    2134 1    0    102  0           [ 0    0    1080 2406 ] [ 0       0       ] [ 1       1       0.5     0.5     ]',
+].join('\n');
+const REAL_KEYBOARD_HIDDEN_ROWS = [
+  'softKeyboard1        0         6100    16    2105 102  0    -1   0           [ 0    0    1080 2406 ] [ 0       0       ] [ 1       1       0.5     0.5     ]',
+  'SCBKeyboardPanel     0         5714    17    2134 1    0    -1   0           [ 0    0    1080 2406 ] [ 0       0       ] [ 1       1       0.5     0.5     ]',
+].join('\n');
+
+/** A panel rect materially smaller than the display: the shape a true key-plane report would take. */
+const TRUE_BAND_KEYBOARD_ROW =
+  'SCBKeyboardPanel     0         5714    17    2134 1    0    102  0           [ 0    1546    1080    860    ] [ 0       0       ] [ 1       1       0.5     0.5     ]';
+
+beforeEach(() => {
+  runHarmonyHdc.mockReset();
+  runHarmonyShell.mockReset();
+  runHarmonyShell.mockResolvedValue({ exitCode: 0, stdout: '', stderr: '' });
+});
+
+test('a zero-rect keyboard panel reads as hidden, not as a band', () => {
+  assert.deepEqual(harmonyKeyboardBandFromWindowDump(HIDDEN_KEYBOARD_TABLE), {
+    kind: 'visibility-only',
+    visible: false,
+  });
+});
+
+test('a zero rect on a positively ordered window still reads as hidden', () => {
+  const zeroRectShownRow =
+    'SCBKeyboardPanel     0         5714    17    2134 1    0    4400 0           [ 0    0    0    0    ] [ 0       0       ] [ 1       1       0       0       ]';
+  const dump = `${WINDOW_TABLE_HEADER}\n${zeroRectShownRow}`;
+  assert.deepEqual(harmonyKeyboardBandFromWindowDump(dump), {
+    kind: 'visibility-only',
+    visible: false,
+  });
+});
+
+test('the real display-spanning up-state rows publish visibility-only, not a full-screen band', () => {
+  const dump = `${WINDOW_TABLE_HEADER}\n${REAL_KEYBOARD_UP_ROWS}`;
+  assert.deepEqual(harmonyKeyboardBandFromWindowDump(dump), {
+    kind: 'visibility-only',
+    visible: true,
+  });
+});
+
+test('the real hidden-state rows publish visibility-only false', () => {
+  const dump = `${WINDOW_TABLE_HEADER}\n${REAL_KEYBOARD_HIDDEN_ROWS}`;
+  assert.deepEqual(harmonyKeyboardBandFromWindowDump(dump), {
+    kind: 'visibility-only',
+    visible: false,
+  });
+});
+
+test('a panel rect materially smaller than the display publishes the band frame', () => {
+  const dump = `${WINDOW_TABLE_HEADER}\n${TRUE_BAND_KEYBOARD_ROW}`;
+  assert.deepEqual(harmonyKeyboardBandFromWindowDump(dump), {
+    kind: 'visible',
+    frame: { x: 0, y: 1546, width: 1080, height: 860 },
+  });
+});
+
+test('a keyboard-only table with no display-sized row stays visibility-only', () => {
+  // The panel's own rect is the only positive rect, so the display cannot be derived and a
+  // frame must not be published from it: the band would be an unproven guess.
+  const dump = [
+    'WindowName           DisplayId Pid     WinId Type Mode Flag ZOrd Orientation [ x    y    w    h    ] [ OffsetX OffsetY ] [ ScaleX  ScaleY  PivotX  PivotY  ]',
+    TRUE_BAND_KEYBOARD_ROW,
+  ].join('\n');
+  assert.deepEqual(harmonyKeyboardBandFromWindowDump(dump), {
+    kind: 'visibility-only',
+    visible: true,
+  });
+});
+
+test('a positive rect on a negatively ordered window stays hidden', () => {
+  const staleRectRow =
+    'SCBKeyboardPanel     0         5714    17    2134 1    0    -1   0           [ 0    1546    1080    860    ] [ 0       0       ] [ 1       1       0       0       ]';
+  const dump = `${WINDOW_TABLE_HEADER}\n${staleRectRow}`;
+  assert.deepEqual(harmonyKeyboardBandFromWindowDump(dump), {
+    kind: 'visibility-only',
+    visible: false,
+  });
+});
+
+test('a table without the keyboard panel answers unmeasurable, not hidden', () => {
+  assert.deepEqual(harmonyKeyboardBandFromWindowDump(WINDOW_TABLE_HEADER), {
+    kind: 'unmeasurable',
+    reason: 'keyboard-window-not-listed',
+  });
+});
+
+test('a panel row without a readable rect answers unmeasurable', () => {
+  const rectlessRow = 'SCBKeyboardPanel     0         5714    17    2134 1    0    -1   0';
+  const dump = `${WINDOW_TABLE_HEADER}\n${rectlessRow}`;
+  assert.deepEqual(harmonyKeyboardBandFromWindowDump(dump), {
+    kind: 'unmeasurable',
+    reason: 'keyboard-window-rect-unparseable',
+  });
+});
+
+test('a failed window probe answers unmeasurable and never throws', async () => {
+  runHarmonyShell.mockResolvedValue({ exitCode: 1, stdout: '', stderr: 'dump failed' });
+  assert.deepEqual(await readHarmonyKeyboardBand(DEVICE), {
+    kind: 'unmeasurable',
+    reason: 'keyboard-window-probe-failed',
+  });
+
+  runHarmonyShell.mockRejectedValue(new Error('hdc disconnected'));
+  assert.deepEqual(await readHarmonyKeyboardBand(DEVICE), {
+    kind: 'unmeasurable',
+    reason: 'keyboard-window-probe-failed',
+  });
+});
+
+test('readHarmonyKeyboardBand dumps the window manager table', async () => {
+  runHarmonyShell.mockImplementation(async (_device: unknown, words: string[]) => {
+    if (words[0] === 'hidumper') {
+      return { exitCode: 0, stdout: HIDDEN_KEYBOARD_TABLE, stderr: '' };
+    }
+    return { exitCode: 0, stdout: '', stderr: '' };
+  });
+
+  await readHarmonyKeyboardBand(DEVICE);
+
+  const hidumperCall = runHarmonyShell.mock.calls.find(
+    (call) => (call[1] as string[])[0] === 'hidumper',
+  );
+  assert.deepEqual(hidumperCall?.[1], ['hidumper', '-s', 'WindowManagerService', '-a', '-a']);
+});
+
+test('snapshotHarmony carries the keyboard band beside its tree', async () => {
+  runHarmonyShell.mockImplementation(async (_device: unknown, words: string[]) => {
+    if (words[0] === 'hidumper') {
+      return {
+        exitCode: 0,
+        stdout: `${WINDOW_TABLE_HEADER}\n${REAL_KEYBOARD_UP_ROWS}`,
+        stderr: '',
+      };
+    }
+    return { exitCode: 0, stdout: '', stderr: '' };
+  });
+  runHarmonyHdc.mockImplementation(async (_device: unknown, args: string[]) => {
+    if (args[0] === 'file' && args[1] === 'recv') {
+      fs.writeFileSync(
+        args[3] as string,
+        JSON.stringify({
+          attributes: { type: 'root', bounds: '[0,0][1080,2406]' },
+          children: [{ attributes: { type: 'Button', text: 'first', clickable: 'true' } }],
+        }),
+        'utf8',
+      );
+    }
+    return { exitCode: 0, stdout: '', stderr: '' };
+  });
+
+  const snapshot = await snapshotHarmony(DEVICE);
+
+  assert.deepEqual(snapshot.keyboard, {
+    kind: 'visibility-only',
+    visible: true,
+  });
+  assert.deepEqual(
+    snapshot.nodes.map((node) => node.value ?? node.type),
+    ['Application', 'first'],
+  );
+});