sami/vdm
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

pkg: nightly integration run — real veloxd + testserver, 50 concurrent

Browse Source 
Adds tests/integration/nightly_run.py, wired as the nightly-integration job in
ci.yml (schedule + workflow_dispatch). Real veloxd + tools/testserver, 50
concurrent downloads mixing flaky-reset/throttled/no-range/plain, asserting:
every completed file's SHA-256 against testserver's own /sha256/ route (never
trusting veloxd's own success claim), every task reaching a terminal state
inside the timeout, and veloxd's own open-FD count settling back to baseline.

veloxd runs isolated (XDG_RUNTIME_DIR/XDG_DATA_HOME/XDG_CONFIG_HOME under a
fresh mkdtemp — not the session scratch dir, whose path overflows AF_UNIX's
sun_path). saveTo.allowedRoots is seeded directly into velox.db after a
migrations-only warm-up start, since settings.set returns -32603 'not
implemented in this build' on the veloxd this job builds (verified live).

Every task gets its own filename override on download.add: tasks sharing
(mode, size) share a URL, and without distinct filenames they raced each
other's rename on the first real run (48/50 'passed' with io_errors and
checksum mismatches on the collided tasks) before this fix.

Every assertion was forced red once on purpose and the transcript recorded in
tests/integration/README.md, per this repo's history of green checks that
didn't look where the bug was.

Every spawned child (veloxd, testserver.py) gets PR_SET_PDEATHSIG plus its own
process group, so a hard-killed harness can't strand a daemon the way a prior
run did (4h40m under systemd --user, because SIGKILL never reaches a
finally: block). Proven by kill -9'ing a running harness mid-download and
confirming both children exit with it.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01RBPR7iM3YPyxrjWsVtZDPJ
This commit is contained in:
sami
2026-09-11 17:07:42 +04:00
co-authored by Claude Sonnet 5
parent 67aefbce4b
commit efe76c319d
3 changed files with 524 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/integration/README.md
+83 -3
View File
@@ -4,6 +4,84 @@ Owned by PKG/QA. Real binaries against `tools/mockd` / `tools/testserver`, headl
Unit tests live in each lane; this tree is for behaviour that only shows up when the
pieces run together (throughput, memory over time, reconnect).
## Nightly integration run
`tests/integration/nightly_run.py`, wired as the `nightly-integration` job in `ci.yml`
(`schedule: '17 3 * * *'`, plus `workflow_dispatch` for an on-demand run). Real `veloxd`
+ `tools/testserver`, 50 concurrent downloads mixing hostile testserver modes
(`flaky-reset`, `throttled`, `no-range`, mostly `plain`), asserting:
1. every completed file's SHA-256 matches testserver's own `/<mode>/sha256/<size>` route
(computed by this script re-hashing the file on disk — never trusting veloxd's own
claim of success);
2. every task reaches a terminal state inside the timeout (a task stuck retrying forever
is a failure, not a hang for CI's `timeout(1)` to paper over);
3. `veloxd`'s own open-FD count (`/proc/<pid>/fd`) returns to within a small fixed
tolerance of its pre-run baseline, checked after a settle window.
`veloxd` runs isolated: `XDG_RUNTIME_DIR`, `XDG_DATA_HOME` and `XDG_CONFIG_HOME` all
point into a fresh `mkdtemp()` (not this session's scratch dir — its path is long enough
to overflow `AF_UNIX`'s ~108-byte `sun_path`; verified live via "File name too long"
before switching to `tempfile.mkdtemp()`). `saveTo.allowedRoots` is seeded straight into
`velox.db` after a migrations-only warm-up start: `settings.set` returns `-32603 "not
implemented in this build"` on the `veloxd` this job builds — verified live, not assumed
— so direct DB seeding is the only entry point that currently exists, not a workaround
for a wrong contract. Every task also gets its own `filename` override on `download.add`:
tasks share `(mode, size)` pairs by design (testserver's content is a pure function of
path, not of who's asking), and without distinct filenames they raced each other for the
same destination path — this was caught live on the first real run of this script (48/50
"passed" with io_errors and checksum mismatches on the collided tasks) before the
`filename` override was added.
### What makes each check go red, proven once
Per the standing note in this repo's history (four green checks that didn't look where
the bug was — an always-false guard, a `ctest` label matching zero tests, `--check`
validating pkg-config instead of apt names, conformance validating only fixtures): every
assertion here was forced red once, on purpose, before being trusted.
| Assertion | Forced via | Observed |
|---|---|---|
| Checksum match | `VDM_NIGHTLY_FORCE_BAD_HASH=1` (substitutes a wrong hash for task 0's comparison only, after the real download and hash succeed) | `FAIL: task 0 (plain, 256K) checksum mismatch: got 6f4c254c…, testserver says 0000…0000`, exit 1 |
| FD-leak tolerance | `VDM_NIGHTLY_FORCE_FD_LEAK=1` (adds 25 to the post-run FD count) | `FAIL: veloxd leaked file descriptors: 19 -> 49 (tolerance 10)`, exit 1 |
| A task that can't succeed still fails the run | `VDM_NIGHTLY_FORCE_HOSTILE_STALL=1` (task 0's URL points at 192.0.2.1, TEST-NET-1 — unroutable, so the connection just hangs) | `FAIL: task 0 (plain, 256K) ended in state 'failed', error={'code': 'timeout', ...}`, exit 1 |
The first version of the `HOSTILE_STALL` hook used testserver's `416-always` mode,
expecting it to never yield a 2xx to a Range probe — but `veloxd` correctly falls back
to a plain full GET on a 416 and the task completed fine, so that attempt proved
nothing (a real finding in itself: worth knowing the daemon handles this correctly).
Re-run any of the three whenever the corresponding assertion changes, to re-prove it
still catches what it claims to — that's the point of the hooks living in the script
rather than being one-off manual edits.
### Proven: the harness cannot leak its daemon
A run of this script was itself killed hard (its own harness process, not a graceful
stop) mid-download and left `veloxd` running under `systemd --user` for 4h40m — the
`finally:` teardown never got to run, because nothing runs after `SIGKILL`. Fixed by
giving every child (`veloxd` and `testserver.py`) `PR_SET_PDEATHSIG` (via a `preexec_fn`
calling `prctl` through `ctypes`) plus its own process group (`start_new_session=True`):
the kernel now delivers `SIGKILL` to a child the instant its parent dies, by any means,
without the harness needing to run any code at all. `finally:` still does the graceful
SIGTERM-then-SIGKILL `killpg` for the normal-exit path; `PR_SET_PDEATHSIG` is what
covers the path `finally:` cannot reach.
Proven live: started the harness, waited for its "baseline FDs" log line (proof the
real `veloxd` and `testserver.py` were both already up as separate process groups
under it), then `kill -9`'d the harness itself and confirmed both children were gone
within 1.5 s — nothing left running, nothing to clean up by hand.
### Known limitation: single-instance lock is per-euid, not per-XDG-tree
`veloxd`'s single-instance guard binds an abstract-namespace socket keyed only by
`geteuid()` (`daemon/src/main.cpp`), so `XDG_RUNTIME_DIR` isolation does not let two
`veloxd` processes for the same Unix user run side by side — confirmed live: a second
instance exits with "another instance is already running for this user" even with fully
distinct `XDG_*` dirs. Harmless on a real CI runner (one job, one user, one `veloxd`) but
means this script cannot run concurrently with another `veloxd` on the same machine —
worth knowing before parallelizing this job or running it by hand next to another lane's
manual testing.
## GUI M1 definition-of-done gates (R3)
`gui/docs/pkg-qa-requests-m1.md` R3: three GUI DoD items are not unit tests and have
@@ -85,11 +163,13 @@ Contract:
path: rss.json
```
`gui-dod-nightly` needs a `schedule:` trigger added to the top of `ci.yml` (there is none
today), or its own workflow file. Decide that when the harness lands.
`gui-dod-nightly` needs `if: github.event_name == 'schedule'` (the `nightly-integration`
job below already added that trigger to `ci.yml` — `cron: '17 3 * * *'` — so this no
longer needs its own).
### Status
Blocked on GUI's harness. Not urgent (GUI M1 DoD, not M0). When GUI files the follow-up
with the real `run.sh` path and the `rss-flat` slack number, PKG/QA drops the `TODO(GUI)`
markers, adds the `schedule:` trigger, and marks `gui-dod` required.
markers and marks `gui-dod` required. The `schedule:` trigger `gui-dod-nightly` needs is
already in `ci.yml`.