samiandClaude Sonnet 5
1d359af5a3
gui-dod (per-PR: scroll-60fps + unhappy-path) and gui-dod-nightly (rss-flat, schedule/workflow_dispatch) are live in ci.yml, driving GUI's newly-landed gui/tests/dod/run.sh + gui-dod-harness. No Xvfb step: run.sh already runs QT_QPA_PLATFORM=offscreen itself. Each gate forced red once before being trusted (tests/integration/README.md has the transcripts): VELOX_DOD_FRAME_BUDGET_MS=0.01 for scroll-60fps, VELOX_DOD_RSS_SLACK_KIB=-999999999 for rss-flat, and — since run.sh always starts a working mockd — a direct gui-dod-harness invocation against an unreachable socket for unhappy-path, which hit the harness's own 75s watchdog exactly as documented. Recorded GUI's live finding (gui/docs/proto-requests-m1.md) that mockd --drop-connection is a no-op over the UDS transport, so unhappy-path's drop-connection phase can't yet exercise a real drop — coordinating with PROTO on the fix rather than working around it locally. gui-dod stays required regardless: its other two phases and the crash/hang/watchdog paths still catch real regressions. Added gui-dod to BRANCH_PROTECTION.md's required-checks table. ADR 0019: the live-veloxd conformance runner (run.sh step 3b, already unconditional inside the already-required conformance job) stays required as PROTO's xfail list shrinks (18 entries now, down from 34; 57/57 fixtures passing on main). No CI change needed — it was already inside a required check; this records the decision not to carve out an exception for it. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01RBPR7iM3YPyxrjWsVtZDPJ
150 lines
9.9 KiB
Markdown
150 lines
9.9 KiB
Markdown
# tests/integration — cross-lane runtime gates
|
||
|
||
Owned by PKG/QA. Real binaries against `tools/mockd` / `tools/testserver`, headless.
|
||
Unit tests live in each lane; this tree is for behaviour that only shows up when the
|
||
pieces run together (throughput, memory over time, reconnect).
|
||
|
||
## Nightly integration run
|
||
|
||
`tests/integration/nightly_run.py`, wired as the `nightly-integration` job in `ci.yml`
|
||
(`schedule: '17 3 * * *'`, plus `workflow_dispatch` for an on-demand run). Real `veloxd`
|
||
+ `tools/testserver`, 50 concurrent downloads mixing hostile testserver modes
|
||
(`flaky-reset`, `throttled`, `no-range`, mostly `plain`), asserting:
|
||
|
||
1. every completed file's SHA-256 matches testserver's own `/<mode>/sha256/<size>` route
|
||
(computed by this script re-hashing the file on disk — never trusting veloxd's own
|
||
claim of success);
|
||
2. every task reaches a terminal state inside the timeout (a task stuck retrying forever
|
||
is a failure, not a hang for CI's `timeout(1)` to paper over);
|
||
3. `veloxd`'s own open-FD count (`/proc/<pid>/fd`) returns to within a small fixed
|
||
tolerance of its pre-run baseline, checked after a settle window.
|
||
|
||
`veloxd` runs isolated: `XDG_RUNTIME_DIR`, `XDG_DATA_HOME` and `XDG_CONFIG_HOME` all
|
||
point into a fresh `mkdtemp()` (not this session's scratch dir — its path is long enough
|
||
to overflow `AF_UNIX`'s ~108-byte `sun_path`; verified live via "File name too long"
|
||
before switching to `tempfile.mkdtemp()`). `saveTo.allowedRoots` is seeded straight into
|
||
`velox.db` after a migrations-only warm-up start: `settings.set` returns `-32603 "not
|
||
implemented in this build"` on the `veloxd` this job builds — verified live, not assumed
|
||
— so direct DB seeding is the only entry point that currently exists, not a workaround
|
||
for a wrong contract. Every task also gets its own `filename` override on `download.add`:
|
||
tasks share `(mode, size)` pairs by design (testserver's content is a pure function of
|
||
path, not of who's asking), and without distinct filenames they raced each other for the
|
||
same destination path — this was caught live on the first real run of this script (48/50
|
||
"passed" with io_errors and checksum mismatches on the collided tasks) before the
|
||
`filename` override was added.
|
||
|
||
### What makes each check go red, proven once
|
||
|
||
Per the standing note in this repo's history (four green checks that didn't look where
|
||
the bug was — an always-false guard, a `ctest` label matching zero tests, `--check`
|
||
validating pkg-config instead of apt names, conformance validating only fixtures): every
|
||
assertion here was forced red once, on purpose, before being trusted.
|
||
|
||
| Assertion | Forced via | Observed |
|
||
|---|---|---|
|
||
| Checksum match | `VDM_NIGHTLY_FORCE_BAD_HASH=1` (substitutes a wrong hash for task 0's comparison only, after the real download and hash succeed) | `FAIL: task 0 (plain, 256K) checksum mismatch: got 6f4c254c…, testserver says 0000…0000`, exit 1 |
|
||
| FD-leak tolerance | `VDM_NIGHTLY_FORCE_FD_LEAK=1` (adds 25 to the post-run FD count) | `FAIL: veloxd leaked file descriptors: 19 -> 49 (tolerance 10)`, exit 1 |
|
||
| A task that can't succeed still fails the run | `VDM_NIGHTLY_FORCE_HOSTILE_STALL=1` (task 0's URL points at 192.0.2.1, TEST-NET-1 — unroutable, so the connection just hangs) | `FAIL: task 0 (plain, 256K) ended in state 'failed', error={'code': 'timeout', ...}`, exit 1 |
|
||
|
||
The first version of the `HOSTILE_STALL` hook used testserver's `416-always` mode,
|
||
expecting it to never yield a 2xx to a Range probe — but `veloxd` correctly falls back
|
||
to a plain full GET on a 416 and the task completed fine, so that attempt proved
|
||
nothing (a real finding in itself: worth knowing the daemon handles this correctly).
|
||
Re-run any of the three whenever the corresponding assertion changes, to re-prove it
|
||
still catches what it claims to — that's the point of the hooks living in the script
|
||
rather than being one-off manual edits.
|
||
|
||
### Proven: the harness cannot leak its daemon
|
||
|
||
A run of this script was itself killed hard (its own harness process, not a graceful
|
||
stop) mid-download and left `veloxd` running under `systemd --user` for 4h40m — the
|
||
`finally:` teardown never got to run, because nothing runs after `SIGKILL`. Fixed by
|
||
giving every child (`veloxd` and `testserver.py`) `PR_SET_PDEATHSIG` (via a `preexec_fn`
|
||
calling `prctl` through `ctypes`) plus its own process group (`start_new_session=True`):
|
||
the kernel now delivers `SIGKILL` to a child the instant its parent dies, by any means,
|
||
without the harness needing to run any code at all. `finally:` still does the graceful
|
||
SIGTERM-then-SIGKILL `killpg` for the normal-exit path; `PR_SET_PDEATHSIG` is what
|
||
covers the path `finally:` cannot reach.
|
||
|
||
Proven live: started the harness, waited for its "baseline FDs" log line (proof the
|
||
real `veloxd` and `testserver.py` were both already up as separate process groups
|
||
under it), then `kill -9`'d the harness itself and confirmed both children were gone
|
||
within 1.5 s — nothing left running, nothing to clean up by hand.
|
||
|
||
### Known limitation: single-instance lock is per-euid, not per-XDG-tree
|
||
|
||
`veloxd`'s single-instance guard binds an abstract-namespace socket keyed only by
|
||
`geteuid()` (`daemon/src/main.cpp`), so `XDG_RUNTIME_DIR` isolation does not let two
|
||
`veloxd` processes for the same Unix user run side by side — confirmed live: a second
|
||
instance exits with "another instance is already running for this user" even with fully
|
||
distinct `XDG_*` dirs. Harmless on a real CI runner (one job, one user, one `veloxd`) but
|
||
means this script cannot run concurrently with another `veloxd` on the same machine —
|
||
worth knowing before parallelizing this job or running it by hand next to another lane's
|
||
manual testing.
|
||
|
||
## GUI M1 definition-of-done gates (R3)
|
||
|
||
`gui/docs/pkg-qa-requests-m1.md` R3: three GUI DoD items are not unit tests and have
|
||
nowhere to run. GUI owns the harness; PKG/QA owns the CI job. This is the wiring contract
|
||
so the two halves meet without another round trip.
|
||
|
||
### What GUI built, and the invocation contract
|
||
|
||
`gui/tests/dod/run.sh <gate> [--json <path>]` (`gate` one of `scroll-60fps` / `rss-flat`
|
||
/ `unhappy-path`), backed by `gui/tests/dod/dod_harness.cpp` (target `gui-dod-harness`).
|
||
Headless by default — `run.sh` sets `QT_QPA_PLATFORM=offscreen` itself, so the CI jobs
|
||
below need no Xvfb.
|
||
|
||
| `<gate>` | Pass / fail condition | Budget |
|
||
|---|---|---|
|
||
| `scroll-60fps` | `mockd --tasks 10000`, scripted fling scroll; **fail** if p99 frame > 16.6 ms (×4 under an ASan/UBSan build — `VELOX_DOD_FRAME_BUDGET_MS` overrides outright) | per-PR |
|
||
| `rss-flat` | `mockd --tasks 10000` + progress events, 10 min; **fail** if RSS growth past warm-up exceeds a 20 MiB slack (`VELOX_DOD_RSS_SLACK_KIB` overrides) | nightly |
|
||
| `unhappy-path` | Three phases (`--slow` / `--flaky <f>` / `--drop-connection <s>`), one `mockd` restart each; **fail** on crash, on the harness's own watchdog firing (75 s), or if connection state never (re)reaches `Connected` | per-PR |
|
||
|
||
Contract, as built: exit `0` pass, non-zero fail; a hang is the harness's own watchdog
|
||
converting itself into a non-zero exit (`3`), never something CI needs `timeout(1)`
|
||
around; `--json` writes one result object per gate (`unhappy-path` merges its three
|
||
phases into one file); no network, no writes outside a tempdir, no leaked child process
|
||
on any exit path (`run.sh`'s `trap cleanup EXIT INT TERM`).
|
||
|
||
### CI jobs — wired
|
||
|
||
`gui-dod` (per-PR: `scroll-60fps` + `unhappy-path`) and `gui-dod-nightly` (`rss-flat`,
|
||
`schedule`/`workflow_dispatch` only) are live in `ci.yml`. Both bootstrap, build
|
||
`gui-dod-harness`, `npm ci` in `tools/mockd`, run `gui/tests/dod/run.sh`, and upload the
|
||
`--json` output as an artifact — no Xvfb step, since `run.sh` already runs offscreen.
|
||
|
||
### Forced red, once per gate, before wiring it required
|
||
|
||
| Gate | Forced via | Observed |
|
||
|---|---|---|
|
||
| `scroll-60fps` | `VELOX_DOD_FRAME_BUDGET_MS=0.01` | `FAIL: p99=50.73 ms mean=31.34 ms max=52.88 ms budget=0.01 ms over 240 steps, 10000 rows`, exit 1 |
|
||
| `rss-flat` | `VELOX_DOD_RSS_SLACK_KIB=-999999999` (guarantees `growthKiB > slack` regardless of actual RSS behavior that run) | `FAIL: growth=13644 KiB slack=-999999999 KiB over 15s (warmup 1s), 10000 rows`, exit 1 |
|
||
| `unhappy-path` | ran `gui-dod-harness` directly (bypassing `run.sh`, which always starts a working `mockd`) against a `--sock` path with nothing listening | 75 s of `Connecting`/`Reconnecting`, then `FAIL: dod_harness watchdog fired — hung past 75s`, exit 3 |
|
||
|
||
The real (non-forced) runs all pass live: `scroll-60fps` p99 51.28 ms against a 66.4 ms
|
||
budget (ASan/UBSan build), `rss-flat` growth 13504 KiB against the 20480 KiB slack over a
|
||
15 s smoke duration, `unhappy-path` all three phases `PASS` with `finalConnected=true`.
|
||
|
||
### Known gap: `unhappy-path`'s drop-connection phase doesn't drop anything
|
||
|
||
Filed by GUI in `gui/docs/proto-requests-m1.md`: `mockd --drop-connection` only works
|
||
over WebSocket — `startUds()` never wires the periodic-drop timer `startWs()` has, so the
|
||
UDS transport (GUI/CLI/nmhost's only one) never sees a connection actually die. GUI's own
|
||
live verification: 45 s observing a `--drop-connection 5` mockd over UDS, `stateChanged`
|
||
never fires. The phase still runs and its JSON honestly records `sawDisruption: false`
|
||
rather than silently passing as if it proved something — that's what the real run above
|
||
shows, `pass: true` alongside `sawDisruption: false`, so a reviewer reading the JSON sees
|
||
exactly how much this phase currently covers.
|
||
|
||
This is PROTO's fix, not PKG/QA's or GUI's to route around locally — coordinating on it
|
||
rather than patching mockd from this lane. Once PROTO lands `dropEverySec` on `startUds`,
|
||
`unhappy-path`'s drop-connection phase starts exercising a real drop and this note comes
|
||
out; until then `gui-dod` stays required as specified (crash/hang/never-reconnects still
|
||
catch real regressions), just not yet catching a swallowed real disconnect.
|
||
|
||
### Status
|
||
|
||
Wired and required: `gui-dod` runs per-PR, `gui-dod-nightly` on schedule. Revisit once
|
||
PROTO's UDS `--drop-connection` fix lands (see above).
|