vdm/tests/integration/README.md

tests/integration — cross-lane runtime gates

Owned by PKG/QA. Real binaries against tools/mockd / tools/testserver, headless. Unit tests live in each lane; this tree is for behaviour that only shows up when the pieces run together (throughput, memory over time, reconnect).

Nightly integration run

tests/integration/nightly_run.py, wired as the nightly-integration job in ci.yml (schedule: '17 3 * * *', plus workflow_dispatch for an on-demand run). Real veloxd

• tools/testserver, 50 concurrent downloads mixing hostile testserver modes (flaky-reset, throttled, no-range, mostly plain), asserting:

1. every completed file's SHA-256 matches testserver's own /<mode>/sha256/<size> route (computed by this script re-hashing the file on disk — never trusting veloxd's own claim of success);
2. every task reaches a terminal state inside the timeout (a task stuck retrying forever is a failure, not a hang for CI's timeout(1) to paper over);
3. veloxd's own open-FD count (/proc/<pid>/fd) returns to within a small fixed tolerance of its pre-run baseline, checked after a settle window.

veloxd runs isolated: XDG_RUNTIME_DIR, XDG_DATA_HOME and XDG_CONFIG_HOME all point into a fresh mkdtemp() (not this session's scratch dir — its path is long enough to overflow AF_UNIX's ~108-byte sun_path; verified live via "File name too long" before switching to tempfile.mkdtemp()). saveTo.allowedRoots is seeded straight into velox.db after a migrations-only warm-up start: settings.set returns -32603 "not implemented in this build" on the veloxd this job builds — verified live, not assumed — so direct DB seeding is the only entry point that currently exists, not a workaround for a wrong contract. Every task also gets its own filename override on download.add: tasks share (mode, size) pairs by design (testserver's content is a pure function of path, not of who's asking), and without distinct filenames they raced each other for the same destination path — this was caught live on the first real run of this script (48/50 "passed" with io_errors and checksum mismatches on the collided tasks) before the filename override was added.

What makes each check go red, proven once

Per the standing note in this repo's history (four green checks that didn't look where the bug was — an always-false guard, a ctest label matching zero tests, --check validating pkg-config instead of apt names, conformance validating only fixtures): every assertion here was forced red once, on purpose, before being trusted.

Assertion	Forced via	Observed
Checksum match	VDM_NIGHTLY_FORCE_BAD_HASH=1 (substitutes a wrong hash for task 0's comparison only, after the real download and hash succeed)	FAIL: task 0 (plain, 256K) checksum mismatch: got 6f4c254c…, testserver says 0000…0000, exit 1
FD-leak tolerance	VDM_NIGHTLY_FORCE_FD_LEAK=1 (adds 25 to the post-run FD count)	FAIL: veloxd leaked file descriptors: 19 -> 49 (tolerance 10), exit 1
A task that can't succeed still fails the run	VDM_NIGHTLY_FORCE_HOSTILE_STALL=1 (task 0's URL points at 192.0.2.1, TEST-NET-1 — unroutable, so the connection just hangs)	FAIL: task 0 (plain, 256K) ended in state 'failed', error={'code': 'timeout', ...}, exit 1

The first version of the HOSTILE_STALL hook used testserver's 416-always mode, expecting it to never yield a 2xx to a Range probe — but veloxd correctly falls back to a plain full GET on a 416 and the task completed fine, so that attempt proved nothing (a real finding in itself: worth knowing the daemon handles this correctly). Re-run any of the three whenever the corresponding assertion changes, to re-prove it still catches what it claims to — that's the point of the hooks living in the script rather than being one-off manual edits.

Proven: the harness cannot leak its daemon

A run of this script was itself killed hard (its own harness process, not a graceful stop) mid-download and left veloxd running under systemd --user for 4h40m — the finally: teardown never got to run, because nothing runs after SIGKILL. Fixed by giving every child (veloxd and testserver.py) PR_SET_PDEATHSIG (via a preexec_fn calling prctl through ctypes) plus its own process group (start_new_session=True): the kernel now delivers SIGKILL to a child the instant its parent dies, by any means, without the harness needing to run any code at all. finally: still does the graceful SIGTERM-then-SIGKILL killpg for the normal-exit path; PR_SET_PDEATHSIG is what covers the path finally: cannot reach.

Proven live: started the harness, waited for its "baseline FDs" log line (proof the real veloxd and testserver.py were both already up as separate process groups under it), then kill -9'd the harness itself and confirmed both children were gone within 1.5 s — nothing left running, nothing to clean up by hand.

Known limitation: single-instance lock is per-euid, not per-XDG-tree

veloxd's single-instance guard binds an abstract-namespace socket keyed only by geteuid() (daemon/src/main.cpp), so XDG_RUNTIME_DIR isolation does not let two veloxd processes for the same Unix user run side by side — confirmed live: a second instance exits with "another instance is already running for this user" even with fully distinct XDG_* dirs. Harmless on a real CI runner (one job, one user, one veloxd) but means this script cannot run concurrently with another veloxd on the same machine — worth knowing before parallelizing this job or running it by hand next to another lane's manual testing.

GUI M1 definition-of-done gates (R3)

gui/docs/pkg-qa-requests-m1.md R3: three GUI DoD items are not unit tests and have nowhere to run. GUI owns the harness; PKG/QA owns the CI job. This is the wiring contract so the two halves meet without another round trip.

What PKG/QA needs from GUI

A driver invoked as gui/tests/dod/run.sh <gate> [--json <path>] (exact path TBD by GUI), headless-capable (Xvfb or offscreen QT_QPA_PLATFORM), with:

<gate>	Pass / fail condition	Budget
scroll-60fps	mockd --tasks 10000, scripted fling scroll; fail if p99 frame > 16.6 ms	per-PR
rss-flat	mockd --tasks 10000 + progress events, 10 min; fail if RSS growth > a fixed slack (GUI picks the number, states it)	nightly
unhappy-path	mockd --slow / --flaky <f> / --drop-connection <s>; fail on crash, on watchdog-detected hang, or if connection state never returns to Connected	per-PR

Contract:

• exit 0 pass, non-zero fail; a hang is the harness's own watchdog to catch and turn into a non-zero exit, not something CI should have to timeout(1) around.
• --json writes one machine-readable result file (measured p99, RSS series, recovery time) so the job can upload it as an artifact and a regression is a diff, not a re-run.
• no network, no writes outside a tempdir, no leaked child processes on failure.

CI job — pre-drafted, add once the harness path is fixed

  gui-dod:
    # Per-PR GUI gates. The 10-minute rss-flat gate is in gui-dod-nightly, not here.
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Bootstrap toolchain
        run: sudo ./tools/bootstrap.sh
      - uses: actions/setup-node@v4
        with:
          node-version: '22'          # tools/mockd
      - name: Configure + build
        run: |
          cmake --preset dev
          cmake --build --preset dev --target velox-gui
      - name: Install mockd
        run: cd tools/mockd && npm ci
      - name: Xvfb + gates
        run: |
          sudo apt-get install -y --no-install-recommends xvfb
          xvfb-run -a gui/tests/dod/run.sh scroll-60fps   --json scroll.json    # TODO(GUI): path
          xvfb-run -a gui/tests/dod/run.sh unhappy-path    --json unhappy.json
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: gui-dod-${{ github.run_id }}
          path: "*.json"

  gui-dod-nightly:
    if: github.event_name == 'schedule'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Bootstrap toolchain
        run: sudo ./tools/bootstrap.sh
      - uses: actions/setup-node@v4
        with:
          node-version: '22'
      - name: Configure + build
        run: |
          cmake --preset dev
          cmake --build --preset dev --target velox-gui
      - run: cd tools/mockd && npm ci
      - name: RSS soak (10 min)
        run: |
          sudo apt-get install -y --no-install-recommends xvfb
          xvfb-run -a gui/tests/dod/run.sh rss-flat --json rss.json
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: gui-dod-rss-${{ github.run_id }}
          path: rss.json

gui-dod-nightly needs if: github.event_name == 'schedule' (the nightly-integration job below already added that trigger to ci.yml — cron: '17 3 * * *' — so this no longer needs its own).

Status

Blocked on GUI's harness. Not urgent (GUI M1 DoD, not M0). When GUI files the follow-up with the real run.sh path and the rss-flat slack number, PKG/QA drops the TODO(GUI) markers and marks gui-dod required. The schedule: trigger gui-dod-nightly needs is already in ci.yml.