diff --git a/.github/BRANCH_PROTECTION.md b/.github/BRANCH_PROTECTION.md index 6e5ca5d..1d533ac 100644 --- a/.github/BRANCH_PROTECTION.md +++ b/.github/BRANCH_PROTECTION.md @@ -18,8 +18,9 @@ policy so it can be re-applied or audited. | `bootstrap-script-2604` | now — real `--with-clang` install in a 26.04 container; the release the project ships on | | `build (gcc)` / `build (clang)` | now — core, daemon and gui have merged | | `sanitizers (dev)` / `sanitizers (tsan)` | now — core, daemon and gui have merged | - | `conformance` | **now — `tests/conformance/` has landed; this is the M0 exit gate** | + | `conformance` | **now — `tests/conformance/` has landed; this is the M0 exit gate.** Includes the live-`veloxd` runner (step 3b of `run.sh`), unconditional in the script — see `docs/adr/0019-live-veloxd-conformance-is-required.md`. | | `extension-lint` | now — `extension/` has merged (MV3 manifest + esbuild build) | + | `gui-dod` | now — `gui/tests/dod/` has landed (GUI M1 DoD gates R3: `scroll-60fps`, `unhappy-path`); see `tests/integration/README.md#gui-m1-definition-of-done-gates-r3`. `gui-dod-nightly` (`rss-flat`) is schedule-only and cannot be a required PR check. | `clang-tidy` is intentionally **not** required through M1 (`continue-on-error: true`, `.clang-tidy` has `WarningsAsErrors: ''`). Make it required at M2. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c68674b..0ad4a17 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -248,3 +248,56 @@ jobs: run: cmake --build --preset dev --target veloxd - name: Nightly integration run run: python3 tests/integration/nightly_run.py --veloxd build/dev/bin/veloxd --tasks 50 --timeout 180 + + gui-dod: + # Per-PR GUI M1 DoD gates (gui/docs/pkg-qa-requests-m1.md R3): scroll-60fps and + # unhappy-path. The 10-minute rss-flat gate is gui-dod-nightly, not here. GUI's + # harness defaults QT_QPA_PLATFORM=offscreen itself, so no Xvfb/compositor needed. + # See tests/integration/README.md#gui-m1-definition-of-done-gates-r3 for what each + # gate catches and the forced-failure transcript proving it isn't vacuous. + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Bootstrap toolchain + run: sudo ./tools/bootstrap.sh + - uses: actions/setup-node@v4 + with: + node-version: '22' # tools/mockd + - name: Configure + build + run: | + cmake --preset dev + cmake --build --preset dev --target gui-dod-harness + - name: Install mockd + run: cd tools/mockd && npm ci + - name: Gates + run: | + gui/tests/dod/run.sh scroll-60fps --json scroll.json + gui/tests/dod/run.sh unhappy-path --json unhappy.json + - uses: actions/upload-artifact@v4 + if: always() + with: + name: gui-dod-${{ github.run_id }} + path: "*.json" + + gui-dod-nightly: + if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Bootstrap toolchain + run: sudo ./tools/bootstrap.sh + - uses: actions/setup-node@v4 + with: + node-version: '22' + - name: Configure + build + run: | + cmake --preset dev + cmake --build --preset dev --target gui-dod-harness + - run: cd tools/mockd && npm ci + - name: RSS soak (10 min) + run: gui/tests/dod/run.sh rss-flat --json rss.json + - uses: actions/upload-artifact@v4 + if: always() + with: + name: gui-dod-rss-${{ github.run_id }} + path: rss.json diff --git a/docs/adr/0019-live-veloxd-conformance-is-required.md b/docs/adr/0019-live-veloxd-conformance-is-required.md new file mode 100644 index 0000000..5565959 --- /dev/null +++ b/docs/adr/0019-live-veloxd-conformance-is-required.md @@ -0,0 +1,45 @@ +# ADR 0019 — The live-`veloxd` conformance runner is a required check, as-is + +**Status:** accepted · **Date:** 2026-09-12 · **Lane:** PKG/QA + +## Context + +`tests/conformance/run.sh` step 3b (ADR 0014's `conformance` ctest, already required +per `.github/BRANCH_PROTECTION.md`) replays every fixture against a real, isolated +`veloxd` it builds and starts — not just mockd, which only proves the TS client and the +fixtures agree with each other. This is the runner that can catch `veloxd` disagreeing +with its own contract, and it is unconditional in `run.sh` (`set -euo pipefail`, no +skip flag): it already runs, and already blocks, inside the `conformance` job. + +What was open was whether to treat that as a settled, defended gate or as something +still provisional while `daemon/docs/deferrals.md`'s D1-D4b stub handlers were excused +via `veloxd-xfail.json`. PROTO's update: 57/57 fixtures pass on `main`, and the xfail +allowlist is down to 18 entries from 34 as DAEMON lands the deferred handlers behind +them. + +## Decision + +The live-`veloxd` conformance runner stays required — no change to CI is needed, since +it already runs inside the already-required `conformance` job (ADR 0014). What this ADR +records is the standing: PKG/QA is not carving out an exception, a `continue-on-error`, +or a separate advisory job for it while the xfail list shrinks. A regression here fails +the same required check a schema mismatch would. + +Verified, not assumed: `tests/conformance/veloxd-xfail.json` has 18 entries as of this +ADR (`python3 -c "import json; print(len(json.load(open('tests/conformance/veloxd-xfail.json'))))"`). +Each remaining entry excuses one still-stubbed handler on `deferrals.md`'s D-list, not a +real disagreement between `veloxd` and its contract — `tests/conformance/README.md` +already draws that line (an entry for anything else is a `run.sh`-detected "xfail entry +unexpectedly passed" or a straight failure, not a quiet pass). + +## Consequences + +- No `ci.yml` or `BRANCH_PROTECTION.md` change: `conformance` was already listed + required, and this runner was already inside it. +- The xfail list is a visible, shrinking number, not a static allowance — as DAEMON + clears more of `deferrals.md`'s D-list, entries come out of + `tests/conformance/veloxd-xfail.json`, and `replay.ts` fails loudly (per + `applyXfail`'s "unexpectedly passed" check) if one is left in after its handler ships. +- Nothing here changes who owns what: `veloxd-xfail.json` and `run.sh` stay PROTO's; + PKG/QA's role is the branch-protection policy this ADR confirms, not the runner + itself. diff --git a/tests/integration/README.md b/tests/integration/README.md index 2c0ecc7..6940548 100644 --- a/tests/integration/README.md +++ b/tests/integration/README.md @@ -88,88 +88,62 @@ manual testing. nowhere to run. GUI owns the harness; PKG/QA owns the CI job. This is the wiring contract so the two halves meet without another round trip. -### What PKG/QA needs from GUI +### What GUI built, and the invocation contract -A driver invoked as `gui/tests/dod/run.sh [--json ]` (exact path TBD by GUI), -headless-capable (Xvfb or offscreen `QT_QPA_PLATFORM`), with: +`gui/tests/dod/run.sh [--json ]` (`gate` one of `scroll-60fps` / `rss-flat` +/ `unhappy-path`), backed by `gui/tests/dod/dod_harness.cpp` (target `gui-dod-harness`). +Headless by default — `run.sh` sets `QT_QPA_PLATFORM=offscreen` itself, so the CI jobs +below need no Xvfb. | `` | Pass / fail condition | Budget | |---|---|---| -| `scroll-60fps` | `mockd --tasks 10000`, scripted fling scroll; **fail** if p99 frame > 16.6 ms | per-PR | -| `rss-flat` | `mockd --tasks 10000` + progress events, 10 min; **fail** if RSS growth > a fixed slack (GUI picks the number, states it) | nightly | -| `unhappy-path` | `mockd --slow` / `--flaky ` / `--drop-connection `; **fail** on crash, on watchdog-detected hang, or if connection state never returns to `Connected` | per-PR | +| `scroll-60fps` | `mockd --tasks 10000`, scripted fling scroll; **fail** if p99 frame > 16.6 ms (×4 under an ASan/UBSan build — `VELOX_DOD_FRAME_BUDGET_MS` overrides outright) | per-PR | +| `rss-flat` | `mockd --tasks 10000` + progress events, 10 min; **fail** if RSS growth past warm-up exceeds a 20 MiB slack (`VELOX_DOD_RSS_SLACK_KIB` overrides) | nightly | +| `unhappy-path` | Three phases (`--slow` / `--flaky ` / `--drop-connection `), one `mockd` restart each; **fail** on crash, on the harness's own watchdog firing (75 s), or if connection state never (re)reaches `Connected` | per-PR | -Contract: +Contract, as built: exit `0` pass, non-zero fail; a hang is the harness's own watchdog +converting itself into a non-zero exit (`3`), never something CI needs `timeout(1)` +around; `--json` writes one result object per gate (`unhappy-path` merges its three +phases into one file); no network, no writes outside a tempdir, no leaked child process +on any exit path (`run.sh`'s `trap cleanup EXIT INT TERM`). -* exit `0` pass, non-zero fail; a hang is the harness's own watchdog to catch and turn - into a non-zero exit, not something CI should have to `timeout(1)` around. -* `--json` writes one machine-readable result file (measured p99, RSS series, recovery - time) so the job can upload it as an artifact and a regression is a diff, not a re-run. -* no network, no writes outside a tempdir, no leaked child processes on failure. +### CI jobs — wired -### CI job — pre-drafted, add once the harness path is fixed +`gui-dod` (per-PR: `scroll-60fps` + `unhappy-path`) and `gui-dod-nightly` (`rss-flat`, +`schedule`/`workflow_dispatch` only) are live in `ci.yml`. Both bootstrap, build +`gui-dod-harness`, `npm ci` in `tools/mockd`, run `gui/tests/dod/run.sh`, and upload the +`--json` output as an artifact — no Xvfb step, since `run.sh` already runs offscreen. -```yaml - gui-dod: - # Per-PR GUI gates. The 10-minute rss-flat gate is in gui-dod-nightly, not here. - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - name: Bootstrap toolchain - run: sudo ./tools/bootstrap.sh - - uses: actions/setup-node@v4 - with: - node-version: '22' # tools/mockd - - name: Configure + build - run: | - cmake --preset dev - cmake --build --preset dev --target velox-gui - - name: Install mockd - run: cd tools/mockd && npm ci - - name: Xvfb + gates - run: | - sudo apt-get install -y --no-install-recommends xvfb - xvfb-run -a gui/tests/dod/run.sh scroll-60fps --json scroll.json # TODO(GUI): path - xvfb-run -a gui/tests/dod/run.sh unhappy-path --json unhappy.json - - uses: actions/upload-artifact@v4 - if: always() - with: - name: gui-dod-${{ github.run_id }} - path: "*.json" +### Forced red, once per gate, before wiring it required - gui-dod-nightly: - if: github.event_name == 'schedule' - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - name: Bootstrap toolchain - run: sudo ./tools/bootstrap.sh - - uses: actions/setup-node@v4 - with: - node-version: '22' - - name: Configure + build - run: | - cmake --preset dev - cmake --build --preset dev --target velox-gui - - run: cd tools/mockd && npm ci - - name: RSS soak (10 min) - run: | - sudo apt-get install -y --no-install-recommends xvfb - xvfb-run -a gui/tests/dod/run.sh rss-flat --json rss.json - - uses: actions/upload-artifact@v4 - if: always() - with: - name: gui-dod-rss-${{ github.run_id }} - path: rss.json -``` +| Gate | Forced via | Observed | +|---|---|---| +| `scroll-60fps` | `VELOX_DOD_FRAME_BUDGET_MS=0.01` | `FAIL: p99=50.73 ms mean=31.34 ms max=52.88 ms budget=0.01 ms over 240 steps, 10000 rows`, exit 1 | +| `rss-flat` | `VELOX_DOD_RSS_SLACK_KIB=-999999999` (guarantees `growthKiB > slack` regardless of actual RSS behavior that run) | `FAIL: growth=13644 KiB slack=-999999999 KiB over 15s (warmup 1s), 10000 rows`, exit 1 | +| `unhappy-path` | ran `gui-dod-harness` directly (bypassing `run.sh`, which always starts a working `mockd`) against a `--sock` path with nothing listening | 75 s of `Connecting`/`Reconnecting`, then `FAIL: dod_harness watchdog fired — hung past 75s`, exit 3 | -`gui-dod-nightly` needs `if: github.event_name == 'schedule'` (the `nightly-integration` -job below already added that trigger to `ci.yml` — `cron: '17 3 * * *'` — so this no -longer needs its own). +The real (non-forced) runs all pass live: `scroll-60fps` p99 51.28 ms against a 66.4 ms +budget (ASan/UBSan build), `rss-flat` growth 13504 KiB against the 20480 KiB slack over a +15 s smoke duration, `unhappy-path` all three phases `PASS` with `finalConnected=true`. + +### Known gap: `unhappy-path`'s drop-connection phase doesn't drop anything + +Filed by GUI in `gui/docs/proto-requests-m1.md`: `mockd --drop-connection` only works +over WebSocket — `startUds()` never wires the periodic-drop timer `startWs()` has, so the +UDS transport (GUI/CLI/nmhost's only one) never sees a connection actually die. GUI's own +live verification: 45 s observing a `--drop-connection 5` mockd over UDS, `stateChanged` +never fires. The phase still runs and its JSON honestly records `sawDisruption: false` +rather than silently passing as if it proved something — that's what the real run above +shows, `pass: true` alongside `sawDisruption: false`, so a reviewer reading the JSON sees +exactly how much this phase currently covers. + +This is PROTO's fix, not PKG/QA's or GUI's to route around locally — coordinating on it +rather than patching mockd from this lane. Once PROTO lands `dropEverySec` on `startUds`, +`unhappy-path`'s drop-connection phase starts exercising a real drop and this note comes +out; until then `gui-dod` stays required as specified (crash/hang/never-reconnects still +catch real regressions), just not yet catching a swallowed real disconnect. ### Status -Blocked on GUI's harness. Not urgent (GUI M1 DoD, not M0). When GUI files the follow-up -with the real `run.sh` path and the `rss-flat` slack number, PKG/QA drops the `TODO(GUI)` -markers and marks `gui-dod` required. The `schedule:` trigger `gui-dod-nightly` needs is -already in `ci.yml`. +Wired and required: `gui-dod` runs per-PR, `gui-dod-nightly` on schedule. Revisit once +PROTO's UDS `--drop-connection` fix lands (see above).