vdm/tests/integration/README.md

tests/integration — cross-lane runtime gates

Owned by PKG/QA. Real binaries against tools/mockd / tools/testserver, headless. Unit tests live in each lane; this tree is for behaviour that only shows up when the pieces run together (throughput, memory over time, reconnect).

Nightly integration run

tests/integration/nightly_run.py, wired as the nightly-integration job in ci.yml (schedule: '17 3 * * *', plus workflow_dispatch for an on-demand run). Real veloxd

• tools/testserver, 50 concurrent downloads mixing hostile testserver modes (flaky-reset, throttled, no-range, mostly plain), asserting:

1. every completed file's SHA-256 matches testserver's own /<mode>/sha256/<size> route (computed by this script re-hashing the file on disk — never trusting veloxd's own claim of success);
2. every task reaches a terminal state inside the timeout (a task stuck retrying forever is a failure, not a hang for CI's timeout(1) to paper over);
3. veloxd's own open-FD count (/proc/<pid>/fd) returns to within a small fixed tolerance of its pre-run baseline, checked after a settle window.

veloxd runs isolated: XDG_RUNTIME_DIR, XDG_DATA_HOME and XDG_CONFIG_HOME all point into a fresh mkdtemp() (not this session's scratch dir — its path is long enough to overflow AF_UNIX's ~108-byte sun_path; verified live via "File name too long" before switching to tempfile.mkdtemp()). saveTo.allowedRoots is seeded straight into velox.db after a migrations-only warm-up start: settings.set returns -32603 "not implemented in this build" on the veloxd this job builds — verified live, not assumed — so direct DB seeding is the only entry point that currently exists, not a workaround for a wrong contract. Every task also gets its own filename override on download.add: tasks share (mode, size) pairs by design (testserver's content is a pure function of path, not of who's asking), and without distinct filenames they raced each other for the same destination path — this was caught live on the first real run of this script (48/50 "passed" with io_errors and checksum mismatches on the collided tasks) before the filename override was added.

What makes each check go red, proven once

Per the standing note in this repo's history (four green checks that didn't look where the bug was — an always-false guard, a ctest label matching zero tests, --check validating pkg-config instead of apt names, conformance validating only fixtures): every assertion here was forced red once, on purpose, before being trusted.

Assertion	Forced via	Observed
Checksum match	VDM_NIGHTLY_FORCE_BAD_HASH=1 (substitutes a wrong hash for task 0's comparison only, after the real download and hash succeed)	FAIL: task 0 (plain, 256K) checksum mismatch: got 6f4c254c…, testserver says 0000…0000, exit 1
FD-leak tolerance	VDM_NIGHTLY_FORCE_FD_LEAK=1 (adds 25 to the post-run FD count)	FAIL: veloxd leaked file descriptors: 19 -> 49 (tolerance 10), exit 1
A task that can't succeed still fails the run	VDM_NIGHTLY_FORCE_HOSTILE_STALL=1 (task 0's URL points at 192.0.2.1, TEST-NET-1 — unroutable, so the connection just hangs)	FAIL: task 0 (plain, 256K) ended in state 'failed', error={'code': 'timeout', ...}, exit 1

The first version of the HOSTILE_STALL hook used testserver's 416-always mode, expecting it to never yield a 2xx to a Range probe — but veloxd correctly falls back to a plain full GET on a 416 and the task completed fine, so that attempt proved nothing (a real finding in itself: worth knowing the daemon handles this correctly). Re-run any of the three whenever the corresponding assertion changes, to re-prove it still catches what it claims to — that's the point of the hooks living in the script rather than being one-off manual edits.

Proven: the harness cannot leak its daemon

A run of this script was itself killed hard (its own harness process, not a graceful stop) mid-download and left veloxd running under systemd --user for 4h40m — the finally: teardown never got to run, because nothing runs after SIGKILL. Fixed by giving every child (veloxd and testserver.py) PR_SET_PDEATHSIG (via a preexec_fn calling prctl through ctypes) plus its own process group (start_new_session=True): the kernel now delivers SIGKILL to a child the instant its parent dies, by any means, without the harness needing to run any code at all. finally: still does the graceful SIGTERM-then-SIGKILL killpg for the normal-exit path; PR_SET_PDEATHSIG is what covers the path finally: cannot reach.

Proven live: started the harness, waited for its "baseline FDs" log line (proof the real veloxd and testserver.py were both already up as separate process groups under it), then kill -9'd the harness itself and confirmed both children were gone within 1.5 s — nothing left running, nothing to clean up by hand.

Known limitation: single-instance lock is per-euid, not per-XDG-tree

veloxd's single-instance guard binds an abstract-namespace socket keyed only by geteuid() (daemon/src/main.cpp), so XDG_RUNTIME_DIR isolation does not let two veloxd processes for the same Unix user run side by side — confirmed live: a second instance exits with "another instance is already running for this user" even with fully distinct XDG_* dirs. Harmless on a real CI runner (one job, one user, one veloxd) but means this script cannot run concurrently with another veloxd on the same machine — worth knowing before parallelizing this job or running it by hand next to another lane's manual testing.

GUI M1 definition-of-done gates (R3)

gui/docs/pkg-qa-requests-m1.md R3: three GUI DoD items are not unit tests and have nowhere to run. GUI owns the harness; PKG/QA owns the CI job. This is the wiring contract so the two halves meet without another round trip.

What GUI built, and the invocation contract

gui/tests/dod/run.sh <gate> [--json <path>] (gate one of scroll-60fps / rss-flat / unhappy-path), backed by gui/tests/dod/dod_harness.cpp (target gui-dod-harness). Headless by default — run.sh sets QT_QPA_PLATFORM=offscreen itself, so the CI jobs below need no Xvfb.

<gate>	Pass / fail condition	Budget
scroll-60fps	mockd --tasks 10000, scripted fling scroll; fail if p99 frame > 16.6 ms (×4 under an ASan/UBSan build — VELOX_DOD_FRAME_BUDGET_MS overrides outright)	per-PR
rss-flat	mockd --tasks 10000 + progress events, 10 min; fail if RSS growth past warm-up exceeds a 20 MiB slack (VELOX_DOD_RSS_SLACK_KIB overrides)	nightly
unhappy-path	Three phases (--slow / --flaky <f> / --drop-connection <s>), one mockd restart each; fail on crash, on the harness's own watchdog firing (75 s), or if connection state never (re)reaches Connected	per-PR

Contract, as built: exit 0 pass, non-zero fail; a hang is the harness's own watchdog converting itself into a non-zero exit (3), never something CI needs timeout(1) around; --json writes one result object per gate (unhappy-path merges its three phases into one file); no network, no writes outside a tempdir, no leaked child process on any exit path (run.sh's trap cleanup EXIT INT TERM).

CI jobs — wired

gui-dod (per-PR: scroll-60fps + unhappy-path) and gui-dod-nightly (rss-flat, schedule/workflow_dispatch only) are live in ci.yml. Both bootstrap, build gui-dod-harness, npm ci in tools/mockd, run gui/tests/dod/run.sh, and upload the --json output as an artifact — no Xvfb step, since run.sh already runs offscreen.

Forced red, once per gate, before wiring it required

Gate	Forced via	Observed
scroll-60fps	VELOX_DOD_FRAME_BUDGET_MS=0.01	FAIL: p99=50.73 ms mean=31.34 ms max=52.88 ms budget=0.01 ms over 240 steps, 10000 rows, exit 1
rss-flat	VELOX_DOD_RSS_SLACK_KIB=-999999999 (guarantees growthKiB > slack regardless of actual RSS behavior that run)	FAIL: growth=13644 KiB slack=-999999999 KiB over 15s (warmup 1s), 10000 rows, exit 1
unhappy-path	ran gui-dod-harness directly (bypassing run.sh, which always starts a working mockd) against a --sock path with nothing listening	75 s of Connecting/Reconnecting, then FAIL: dod_harness watchdog fired — hung past 75s, exit 3

The real (non-forced) runs all pass live: scroll-60fps p99 51.28 ms against a 66.4 ms budget (ASan/UBSan build), rss-flat growth 13504 KiB against the 20480 KiB slack over a 15 s smoke duration, unhappy-path all three phases PASS with finalConnected=true.

Known gap: unhappy-path's drop-connection phase doesn't drop anything

Filed by GUI in gui/docs/proto-requests-m1.md: mockd --drop-connection only works over WebSocket — startUds() never wires the periodic-drop timer startWs() has, so the UDS transport (GUI/CLI/nmhost's only one) never sees a connection actually die. GUI's own live verification: 45 s observing a --drop-connection 5 mockd over UDS, stateChanged never fires. The phase still runs and its JSON honestly records sawDisruption: false rather than silently passing as if it proved something — that's what the real run above shows, pass: true alongside sawDisruption: false, so a reviewer reading the JSON sees exactly how much this phase currently covers.

This is PROTO's fix, not PKG/QA's or GUI's to route around locally — coordinating on it rather than patching mockd from this lane. Once PROTO lands dropEverySec on startUds, unhappy-path's drop-connection phase starts exercising a real drop and this note comes out; until then gui-dod stays required as specified (crash/hang/never-reconnects still catch real regressions), just not yet catching a swallowed real disconnect.

Status

Wired and required: gui-dod runs per-PR, gui-dod-nightly on schedule. Revisit once PROTO's UDS --drop-connection fix lands (see above).