vdm/tests/integration/README.md

# tests/integration — cross-lane runtime gates

Owned by PKG/QA. Real binaries against `tools/mockd` / `tools/testserver`, headless.
Unit tests live in each lane; this tree is for behaviour that only shows up when the
pieces run together (throughput, memory over time, reconnect).

## Nightly integration run

`tests/integration/nightly_run.py`, wired as the `nightly-integration` job in `ci.yml`
(`schedule: '17 3 * * *'`, plus `workflow_dispatch` for an on-demand run). Real `veloxd`
+ `tools/testserver`, 50 concurrent downloads mixing hostile testserver modes
(`flaky-reset`, `throttled`, `no-range`, mostly `plain`), asserting:

1. every completed file's SHA-256 matches testserver's own `/<mode>/sha256/<size>` route
   (computed by this script re-hashing the file on disk — never trusting veloxd's own
   claim of success);
2. every task reaches a terminal state inside the timeout (a task stuck retrying forever
   is a failure, not a hang for CI's `timeout(1)` to paper over);
3. `veloxd`'s own open-FD count (`/proc/<pid>/fd`) returns to within a small fixed
   tolerance of its pre-run baseline, checked after a settle window.

`veloxd` runs isolated: `XDG_RUNTIME_DIR`, `XDG_DATA_HOME` and `XDG_CONFIG_HOME` all
point into a fresh `mkdtemp()` (not this session's scratch dir — its path is long enough
to overflow `AF_UNIX`'s ~108-byte `sun_path`; verified live via "File name too long"
before switching to `tempfile.mkdtemp()`). `saveTo.allowedRoots` is seeded straight into
`velox.db` after a migrations-only warm-up start: `settings.set` returns `-32603 "not
implemented in this build"` on the `veloxd` this job builds — verified live, not assumed
— so direct DB seeding is the only entry point that currently exists, not a workaround
for a wrong contract. Every task also gets its own `filename` override on `download.add`:
tasks share `(mode, size)` pairs by design (testserver's content is a pure function of
path, not of who's asking), and without distinct filenames they raced each other for the
same destination path — this was caught live on the first real run of this script (48/50
"passed" with io_errors and checksum mismatches on the collided tasks) before the
`filename` override was added.

### What makes each check go red, proven once

Per the standing note in this repo's history (four green checks that didn't look where
the bug was — an always-false guard, a `ctest` label matching zero tests, `--check`
validating pkg-config instead of apt names, conformance validating only fixtures): every
assertion here was forced red once, on purpose, before being trusted.

| Assertion | Forced via | Observed |
|---|---|---|
| Checksum match | `VDM_NIGHTLY_FORCE_BAD_HASH=1` (substitutes a wrong hash for task 0's comparison only, after the real download and hash succeed) | `FAIL: task 0 (plain, 256K) checksum mismatch: got 6f4c254c…, testserver says 0000…0000`, exit 1 |
| FD-leak tolerance | `VDM_NIGHTLY_FORCE_FD_LEAK=1` (adds 25 to the post-run FD count) | `FAIL: veloxd leaked file descriptors: 19 -> 49 (tolerance 10)`, exit 1 |
| A task that can't succeed still fails the run | `VDM_NIGHTLY_FORCE_HOSTILE_STALL=1` (task 0's URL points at 192.0.2.1, TEST-NET-1 — unroutable, so the connection just hangs) | `FAIL: task 0 (plain, 256K) ended in state 'failed', error={'code': 'timeout', ...}`, exit 1 |

The first version of the `HOSTILE_STALL` hook used testserver's `416-always` mode,
expecting it to never yield a 2xx to a Range probe — but `veloxd` correctly falls back
to a plain full GET on a 416 and the task completed fine, so that attempt proved
nothing (a real finding in itself: worth knowing the daemon handles this correctly).
Re-run any of the three whenever the corresponding assertion changes, to re-prove it
still catches what it claims to — that's the point of the hooks living in the script
rather than being one-off manual edits.

### Proven: the harness cannot leak its daemon

A run of this script was itself killed hard (its own harness process, not a graceful
stop) mid-download and left `veloxd` running under `systemd --user` for 4h40m — the
`finally:` teardown never got to run, because nothing runs after `SIGKILL`. Fixed by
giving every child (`veloxd` and `testserver.py`) `PR_SET_PDEATHSIG` (via a `preexec_fn`
calling `prctl` through `ctypes`) plus its own process group (`start_new_session=True`):
the kernel now delivers `SIGKILL` to a child the instant its parent dies, by any means,
without the harness needing to run any code at all. `finally:` still does the graceful
SIGTERM-then-SIGKILL `killpg` for the normal-exit path; `PR_SET_PDEATHSIG` is what
covers the path `finally:` cannot reach.

Proven live: started the harness, waited for its "baseline FDs" log line (proof the
real `veloxd` and `testserver.py` were both already up as separate process groups
under it), then `kill -9`'d the harness itself and confirmed both children were gone
within 1.5 s — nothing left running, nothing to clean up by hand.

### Known limitation: single-instance lock is per-euid, not per-XDG-tree

`veloxd`'s single-instance guard binds an abstract-namespace socket keyed only by
`geteuid()` (`daemon/src/main.cpp`), so `XDG_RUNTIME_DIR` isolation does not let two
`veloxd` processes for the same Unix user run side by side — confirmed live: a second
instance exits with "another instance is already running for this user" even with fully
distinct `XDG_*` dirs. Harmless on a real CI runner (one job, one user, one `veloxd`) but
means this script cannot run concurrently with another `veloxd` on the same machine —
worth knowing before parallelizing this job or running it by hand next to another lane's
manual testing.

## GUI M1 definition-of-done gates (R3)

`gui/docs/pkg-qa-requests-m1.md` R3: three GUI DoD items are not unit tests and have
nowhere to run. GUI owns the harness; PKG/QA owns the CI job. This is the wiring contract
so the two halves meet without another round trip.

### What PKG/QA needs from GUI

A driver invoked as `gui/tests/dod/run.sh <gate> [--json <path>]` (exact path TBD by GUI),
headless-capable (Xvfb or offscreen `QT_QPA_PLATFORM`), with:

| `<gate>` | Pass / fail condition | Budget |
|---|---|---|
| `scroll-60fps` | `mockd --tasks 10000`, scripted fling scroll; **fail** if p99 frame > 16.6 ms | per-PR |
| `rss-flat` | `mockd --tasks 10000` + progress events, 10 min; **fail** if RSS growth > a fixed slack (GUI picks the number, states it) | nightly |
| `unhappy-path` | `mockd --slow` / `--flaky <f>` / `--drop-connection <s>`; **fail** on crash, on watchdog-detected hang, or if connection state never returns to `Connected` | per-PR |

Contract:

* exit `0` pass, non-zero fail; a hang is the harness's own watchdog to catch and turn
  into a non-zero exit, not something CI should have to `timeout(1)` around.
* `--json` writes one machine-readable result file (measured p99, RSS series, recovery
  time) so the job can upload it as an artifact and a regression is a diff, not a re-run.
* no network, no writes outside a tempdir, no leaked child processes on failure.

### CI job — pre-drafted, add once the harness path is fixed

```yaml
  gui-dod:
    # Per-PR GUI gates. The 10-minute rss-flat gate is in gui-dod-nightly, not here.
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Bootstrap toolchain
        run: sudo ./tools/bootstrap.sh
      - uses: actions/setup-node@v4
        with:
          node-version: '22'          # tools/mockd
      - name: Configure + build
        run: |
          cmake --preset dev
          cmake --build --preset dev --target velox-gui
      - name: Install mockd
        run: cd tools/mockd && npm ci
      - name: Xvfb + gates
        run: |
          sudo apt-get install -y --no-install-recommends xvfb
          xvfb-run -a gui/tests/dod/run.sh scroll-60fps   --json scroll.json    # TODO(GUI): path
          xvfb-run -a gui/tests/dod/run.sh unhappy-path    --json unhappy.json
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: gui-dod-${{ github.run_id }}
          path: "*.json"

  gui-dod-nightly:
    if: github.event_name == 'schedule'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Bootstrap toolchain
        run: sudo ./tools/bootstrap.sh
      - uses: actions/setup-node@v4
        with:
          node-version: '22'
      - name: Configure + build
        run: |
          cmake --preset dev
          cmake --build --preset dev --target velox-gui
      - run: cd tools/mockd && npm ci
      - name: RSS soak (10 min)
        run: |
          sudo apt-get install -y --no-install-recommends xvfb
          xvfb-run -a gui/tests/dod/run.sh rss-flat --json rss.json
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: gui-dod-rss-${{ github.run_id }}
          path: rss.json
```

`gui-dod-nightly` needs `if: github.event_name == 'schedule'` (the `nightly-integration`
job below already added that trigger to `ci.yml` — `cron: '17 3 * * *'` — so this no
longer needs its own).

### Status

Blocked on GUI's harness. Not urgent (GUI M1 DoD, not M0). When GUI files the follow-up
with the real `run.sh` path and the `rss-flat` slack number, PKG/QA drops the `TODO(GUI)`
markers and marks `gui-dod` required. The `schedule:` trigger `gui-dod-nightly` needs is
already in `ci.yml`.