wristivo/docs/GESTURE-SPEC.md

Gesture interaction and recognition specification

Status: proposed MVP requirements and experiment plan, 2026-09-16. Thresholds and performance targets below are hypotheses, not validated recognition results. Follow PLATFORM-FEASIBILITY.md for runtime and transport constraints and the shared contract document for command serialization.

User interaction

The user wears the watch, opens Wristivo, and chooses Gesture Mode. A short guided calibration establishes a comfortable neutral pose and the meaning of right, left, up, and down for that wrist. The watch provides deliberate wrist flicks, not finger tracking or spatial tracking of a hand through the room.

Directions are user-relative: right should navigate the TV focus right even if the watch is worn on the other wrist. Avoid requiring a precisely horizontal arm or large movement. Train and validate the mapping for both wrists and supported crown orientations; never infer a universal screen-space direction directly from a single raw sensor axis.

Input	Proposed output	Scope
Short right wrist flick, then return to neutral	One Right command	MVP after validation
Short left wrist flick, then return to neutral	One Left command	MVP after validation
Short upward wrist flick, then return to neutral	One Up command	MVP after validation
Short downward wrist flick, then return to neutral	One Down command	MVP after validation
Select button	One Select command	Required fallback
System double tap on supported hardware/OS	Activate visible Select primary action	Optional enhancement; test separately
Back, Home, volume and playback controls	Explicit accessible watch buttons	No motion mapping in initial release
Stop button or loss of active scene	Disarm and cancel candidate	Required

Do not map ordinary wrist shaking, finger pinching, clenches, a return-to-neutral movement, or prolonged tilt to additional commands in the MVP. Avoid motion-based power, purchase, delete, or reset actions. The optional system primary action is public SwiftUI behavior, not an API exposing raw double-tap recognition. Apple double-tap documentation

State machine

disarmed -> calibrating -> ready -> candidate -> cooldown -> ready
    ^            |          |          |            |
    +------------+----------+----------+------------+
         stop / inactive scene / target change / failure

• Disarmed: no sensor updates, no motion commands. Touch controls remain available when the transport permits them.
• Calibrating: collect a stable neutral pose and guided labeled examples. Do not send TV commands. Show progress, Retry, and Cancel.
• Ready: only entered when scene active, calibration valid, selected TV identified, and relay available. Low-confidence or mixed-axis movement produces no command.
• Candidate: observe a bounded motion window and classify at most one direction. Reject stale samples, discontinuities, and conflicting movements. On accepted classification, ask the shared command gate to send one command.
• Cooldown: suppress the return motion. Require both elapsed cooldown and a stable return to the neutral zone before recognizing again.
• Any transition to disarmed clears candidate buffers and pending local intent. Foreground return requires explicit Start/Resume. A completed cooldown does not re-arm an inactive app.

One gesture means one button press. Disable held-key repeat in MVP. Recognition acceptance produces a short haptic only when the command gate accepts the intent; errors use a distinct, rate-limited cue. Describe this as command feedback, not a guarantee that the TV changed state.

Motion pipeline for the implementing agent

Use a platform-independent recognizer over timestamped motion samples and a small watch-only Core Motion adapter. CMMotionManager provides processed motion and availability checks; stop its updates when disarmed. CMMotionManager

1. Request an initial 50 Hz sample interval and record actual monotonic sample intervals. Rate is a tuning parameter; it is not guaranteed delivery. Process off the main actor and bound the buffer.
2. Work from relative orientation, rotation rate, and user acceleration. Prefer a short-session neutral reference over compass-dependent global heading. Do not integrate acceleration into a claim of accurate hand position.
3. Normalize using a tested device/wrist orientation transform and the calibration record. Document quaternion multiplication order, units, handedness, and timestamp clock in the recognizer interface.
4. Establish stability from a rolling neutral window, initially 500 ms. Capture at least three labeled examples per direction during engineering trials; simplify consumer calibration only after evidence supports it.
5. Segment deliberate departures from neutral and score dominant direction. Start experiments with a 120–650 ms candidate window, roughly 15–35 degree relative rotation, and a dominant-axis ratio of at least 1.5. These values are not production defaults until tested.
6. Reject ambiguous movement, slow posture drift, bad sample gaps, non-finite values, and implausible jumps. Keep confidence and thresholds tunable and versioned.
7. Begin with 400 ms cooldown plus 200 ms stable-neutral dwell. Neither timer alone permits a second press. A rapid reverse movement used to return to neutral must not become an opposite command.

No health, movement-disorder, private sensor, or private gesture API is needed. Include an accurate motion purpose string and test denied/unavailable data states using the exact chosen SDK and OS. Do not promise a system permission prompt will appear for every Core Motion call. Core Motion privacy and availability guidance

Interfaces and ownership boundaries

The motion agent owns Core Motion acquisition, deterministic sample-to-intent logic, calibration records, and replay fixtures. The watch-shell agent owns SwiftUI state, native buttons and haptics, and calls the motion adapter start/stop lifecycle interface. The relay agent owns immediate Watch Connectivity messaging. The integration owner approves shared protocol changes.

The recognizer output is semantic direction plus recognition timing/confidence; it must not contain TV key strings, URLs, pairing tokens, transport retries, or UI objects. Use the shared command type when creating the outgoing command. The transport gate handles target binding, request identity, freshness, and duplicate handling as defined centrally.

Use injected monotonic clocks in tests. Serialize only what the shared protocol defines: clocks on different devices cannot be assumed to have identical monotonic origins. Do not invent cross-device timestamp arithmetic inside the recognizer.

Privacy and calibration

Keep raw motion in a bounded, ephemeral in-memory buffer by default. Store only the minimal local calibration values and their algorithm version. Reset calibration when wrist/orientation changes or the user requests it. Never silently upload motion samples.

An engineering recording mode may save labeled samples only after explicit opt-in, with visible recording status and a clear delete/export action. Exclude device identifiers, TV tokens, and household addresses from fixtures. Use original recordings with the participant's permission; do not copy an unlicensed gesture dataset.

Verification and acceptance

Unit tests must cover useful behavioral failures: return-to-neutral suppression, noisy samples, ambiguous diagonals, gaps/out-of-order samples, opposite wrist transforms, cooldown boundaries, scene deactivation mid-gesture, and no replay after resume. A synthetic trace only proves logic consistency; record real samples before claiming recognition quality.

For the hardware gate, recruit at least five consenting testers where practical. Include both wrists, varied wrist sizes and comfortable poses, and at least two watch models. Run 20 prompted trials per direction per tested wrist/orientation configuration. Count wrong directions, missing outputs, and duplicate commands separately; do not discard failed trials. Keep tuning participants separate from the final evaluation group when possible.

Proposed pass criteria:

• Per-direction precision >= 98% and recall >= 90%; publish raw counts and a confusion matrix with the result.
• Zero duplicate presses in the evaluation corpus; every accepted motion emits at most one command.
• At most one false command per 30 minutes of representative non-command motion while deliberately armed, tested for each configuration. Include conversation gestures, reaching for a cup, posture changes, and resting.
• Zero commands while disarmed or inactive, including interruptions and resumption.
• Motion completion to visible TV response p95 <= 700 ms in a connected warm session; measure cold/wake paths separately and report failures.
• Calibration can be completed without discomfort; each action has a touch and accessibility fallback. Do not gate essential control behind gesture proficiency.

These are minimum engineering targets, not statistical proof of general reliability. Expand trials before public claims. Any persistent false-action pattern blocks gesture-mode release even if average accuracy passes. Follow Apple's recommendation to provide alternatives to required body movements. Human Interface Guidelines: Gestures

Implementation handoff checklist

Submit the recognizer with deterministic tests, a calibration schema/version, documented units and transforms, sample recording procedure, results with device/build metadata, and unresolved limitations. Submit the watch integration with verified disarm behavior outside Xcode and a short video of right/left/up/down control on a physical TV. Mark every absent artifact as pending; never treat the scaffold's demo command screen as motion recognition.

Foundation implementation boundary

The existing DirectionalGestureGate is an already-normalized excursion gate only. Its unit-test defaults (0.7 activation, 0.25 neutral, 300 ms minimum interval, 10-second maximum arm) are not the production motion pipeline above. Stable-neutral dwell, candidate windows, physical calibration and confidence scoring are still to be implemented by C. The watch UI must explicitly arm the recognizer after calibration; calibration completion never starts live input on its own. Sensor shutdown at the arm deadline needs a lifecycle timer in the adapter, even if no new samples arrive.