merge: lane/pkg-qa — GUI DoD gates wired, conformance required
CI / clang-format (push) Waiting to run
CI / testserver (push) Waiting to run
CI / bootstrap-script (push) Waiting to run
CI / bootstrap-script-2604 (push) Waiting to run
CI / extension-lint (push) Waiting to run
CI / build (clang) (push) Waiting to run
CI / build (gcc) (push) Waiting to run
CI / sanitizers (dev) (push) Waiting to run
CI / sanitizers (tsan) (push) Waiting to run
CI / clang-tidy (push) Waiting to run
CI / conformance (push) Waiting to run
CI / nightly-integration (push) Waiting to run
CI / gui-dod (push) Waiting to run
CI / gui-dod-nightly (push) Waiting to run
CI / clang-format (push) Waiting to run
CI / testserver (push) Waiting to run
CI / bootstrap-script (push) Waiting to run
CI / bootstrap-script-2604 (push) Waiting to run
CI / extension-lint (push) Waiting to run
CI / build (clang) (push) Waiting to run
CI / build (gcc) (push) Waiting to run
CI / sanitizers (dev) (push) Waiting to run
CI / sanitizers (tsan) (push) Waiting to run
CI / clang-tidy (push) Waiting to run
CI / conformance (push) Waiting to run
CI / nightly-integration (push) Waiting to run
CI / gui-dod (push) Waiting to run
CI / gui-dod-nightly (push) Waiting to run
This commit is contained in:
@@ -18,8 +18,9 @@ policy so it can be re-applied or audited.
|
||||
| `bootstrap-script-2604` | now — real `--with-clang` install in a 26.04 container; the release the project ships on |
|
||||
| `build (gcc)` / `build (clang)` | now — core, daemon and gui have merged |
|
||||
| `sanitizers (dev)` / `sanitizers (tsan)` | now — core, daemon and gui have merged |
|
||||
| `conformance` | **now — `tests/conformance/` has landed; this is the M0 exit gate** |
|
||||
| `conformance` | **now — `tests/conformance/` has landed; this is the M0 exit gate.** Includes the live-`veloxd` runner (step 3b of `run.sh`), unconditional in the script — see `docs/adr/0019-live-veloxd-conformance-is-required.md`. |
|
||||
| `extension-lint` | now — `extension/` has merged (MV3 manifest + esbuild build) |
|
||||
| `gui-dod` | now — `gui/tests/dod/` has landed (GUI M1 DoD gates R3: `scroll-60fps`, `unhappy-path`); see `tests/integration/README.md#gui-m1-definition-of-done-gates-r3`. `gui-dod-nightly` (`rss-flat`) is schedule-only and cannot be a required PR check. |
|
||||
|
||||
`clang-tidy` is intentionally **not** required through M1 (`continue-on-error: true`,
|
||||
`.clang-tidy` has `WarningsAsErrors: ''`). Make it required at M2.
|
||||
|
||||
@@ -248,3 +248,56 @@ jobs:
|
||||
run: cmake --build --preset dev --target veloxd
|
||||
- name: Nightly integration run
|
||||
run: python3 tests/integration/nightly_run.py --veloxd build/dev/bin/veloxd --tasks 50 --timeout 180
|
||||
|
||||
gui-dod:
|
||||
# Per-PR GUI M1 DoD gates (gui/docs/pkg-qa-requests-m1.md R3): scroll-60fps and
|
||||
# unhappy-path. The 10-minute rss-flat gate is gui-dod-nightly, not here. GUI's
|
||||
# harness defaults QT_QPA_PLATFORM=offscreen itself, so no Xvfb/compositor needed.
|
||||
# See tests/integration/README.md#gui-m1-definition-of-done-gates-r3 for what each
|
||||
# gate catches and the forced-failure transcript proving it isn't vacuous.
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Bootstrap toolchain
|
||||
run: sudo ./tools/bootstrap.sh
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22' # tools/mockd
|
||||
- name: Configure + build
|
||||
run: |
|
||||
cmake --preset dev
|
||||
cmake --build --preset dev --target gui-dod-harness
|
||||
- name: Install mockd
|
||||
run: cd tools/mockd && npm ci
|
||||
- name: Gates
|
||||
run: |
|
||||
gui/tests/dod/run.sh scroll-60fps --json scroll.json
|
||||
gui/tests/dod/run.sh unhappy-path --json unhappy.json
|
||||
- uses: actions/upload-artifact@v4
|
||||
if: always()
|
||||
with:
|
||||
name: gui-dod-${{ github.run_id }}
|
||||
path: "*.json"
|
||||
|
||||
gui-dod-nightly:
|
||||
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Bootstrap toolchain
|
||||
run: sudo ./tools/bootstrap.sh
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22'
|
||||
- name: Configure + build
|
||||
run: |
|
||||
cmake --preset dev
|
||||
cmake --build --preset dev --target gui-dod-harness
|
||||
- run: cd tools/mockd && npm ci
|
||||
- name: RSS soak (10 min)
|
||||
run: gui/tests/dod/run.sh rss-flat --json rss.json
|
||||
- uses: actions/upload-artifact@v4
|
||||
if: always()
|
||||
with:
|
||||
name: gui-dod-rss-${{ github.run_id }}
|
||||
path: rss.json
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
# ADR 0019 — The live-`veloxd` conformance runner is a required check, as-is
|
||||
|
||||
**Status:** accepted · **Date:** 2026-09-12 · **Lane:** PKG/QA
|
||||
|
||||
## Context
|
||||
|
||||
`tests/conformance/run.sh` step 3b (ADR 0014's `conformance` ctest, already required
|
||||
per `.github/BRANCH_PROTECTION.md`) replays every fixture against a real, isolated
|
||||
`veloxd` it builds and starts — not just mockd, which only proves the TS client and the
|
||||
fixtures agree with each other. This is the runner that can catch `veloxd` disagreeing
|
||||
with its own contract, and it is unconditional in `run.sh` (`set -euo pipefail`, no
|
||||
skip flag): it already runs, and already blocks, inside the `conformance` job.
|
||||
|
||||
What was open was whether to treat that as a settled, defended gate or as something
|
||||
still provisional while `daemon/docs/deferrals.md`'s D1-D4b stub handlers were excused
|
||||
via `veloxd-xfail.json`. PROTO's update: 57/57 fixtures pass on `main`, and the xfail
|
||||
allowlist is down to 18 entries from 34 as DAEMON lands the deferred handlers behind
|
||||
them.
|
||||
|
||||
## Decision
|
||||
|
||||
The live-`veloxd` conformance runner stays required — no change to CI is needed, since
|
||||
it already runs inside the already-required `conformance` job (ADR 0014). What this ADR
|
||||
records is the standing: PKG/QA is not carving out an exception, a `continue-on-error`,
|
||||
or a separate advisory job for it while the xfail list shrinks. A regression here fails
|
||||
the same required check a schema mismatch would.
|
||||
|
||||
Verified, not assumed: `tests/conformance/veloxd-xfail.json` has 18 entries as of this
|
||||
ADR (`python3 -c "import json; print(len(json.load(open('tests/conformance/veloxd-xfail.json'))))"`).
|
||||
Each remaining entry excuses one still-stubbed handler on `deferrals.md`'s D-list, not a
|
||||
real disagreement between `veloxd` and its contract — `tests/conformance/README.md`
|
||||
already draws that line (an entry for anything else is a `run.sh`-detected "xfail entry
|
||||
unexpectedly passed" or a straight failure, not a quiet pass).
|
||||
|
||||
## Consequences
|
||||
|
||||
- No `ci.yml` or `BRANCH_PROTECTION.md` change: `conformance` was already listed
|
||||
required, and this runner was already inside it.
|
||||
- The xfail list is a visible, shrinking number, not a static allowance — as DAEMON
|
||||
clears more of `deferrals.md`'s D-list, entries come out of
|
||||
`tests/conformance/veloxd-xfail.json`, and `replay.ts` fails loudly (per
|
||||
`applyXfail`'s "unexpectedly passed" check) if one is left in after its handler ships.
|
||||
- Nothing here changes who owns what: `veloxd-xfail.json` and `run.sh` stay PROTO's;
|
||||
PKG/QA's role is the branch-protection policy this ADR confirms, not the runner
|
||||
itself.
|
||||
+46
-72
@@ -88,88 +88,62 @@ manual testing.
|
||||
nowhere to run. GUI owns the harness; PKG/QA owns the CI job. This is the wiring contract
|
||||
so the two halves meet without another round trip.
|
||||
|
||||
### What PKG/QA needs from GUI
|
||||
### What GUI built, and the invocation contract
|
||||
|
||||
A driver invoked as `gui/tests/dod/run.sh <gate> [--json <path>]` (exact path TBD by GUI),
|
||||
headless-capable (Xvfb or offscreen `QT_QPA_PLATFORM`), with:
|
||||
`gui/tests/dod/run.sh <gate> [--json <path>]` (`gate` one of `scroll-60fps` / `rss-flat`
|
||||
/ `unhappy-path`), backed by `gui/tests/dod/dod_harness.cpp` (target `gui-dod-harness`).
|
||||
Headless by default — `run.sh` sets `QT_QPA_PLATFORM=offscreen` itself, so the CI jobs
|
||||
below need no Xvfb.
|
||||
|
||||
| `<gate>` | Pass / fail condition | Budget |
|
||||
|---|---|---|
|
||||
| `scroll-60fps` | `mockd --tasks 10000`, scripted fling scroll; **fail** if p99 frame > 16.6 ms | per-PR |
|
||||
| `rss-flat` | `mockd --tasks 10000` + progress events, 10 min; **fail** if RSS growth > a fixed slack (GUI picks the number, states it) | nightly |
|
||||
| `unhappy-path` | `mockd --slow` / `--flaky <f>` / `--drop-connection <s>`; **fail** on crash, on watchdog-detected hang, or if connection state never returns to `Connected` | per-PR |
|
||||
| `scroll-60fps` | `mockd --tasks 10000`, scripted fling scroll; **fail** if p99 frame > 16.6 ms (×4 under an ASan/UBSan build — `VELOX_DOD_FRAME_BUDGET_MS` overrides outright) | per-PR |
|
||||
| `rss-flat` | `mockd --tasks 10000` + progress events, 10 min; **fail** if RSS growth past warm-up exceeds a 20 MiB slack (`VELOX_DOD_RSS_SLACK_KIB` overrides) | nightly |
|
||||
| `unhappy-path` | Three phases (`--slow` / `--flaky <f>` / `--drop-connection <s>`), one `mockd` restart each; **fail** on crash, on the harness's own watchdog firing (75 s), or if connection state never (re)reaches `Connected` | per-PR |
|
||||
|
||||
Contract:
|
||||
Contract, as built: exit `0` pass, non-zero fail; a hang is the harness's own watchdog
|
||||
converting itself into a non-zero exit (`3`), never something CI needs `timeout(1)`
|
||||
around; `--json` writes one result object per gate (`unhappy-path` merges its three
|
||||
phases into one file); no network, no writes outside a tempdir, no leaked child process
|
||||
on any exit path (`run.sh`'s `trap cleanup EXIT INT TERM`).
|
||||
|
||||
* exit `0` pass, non-zero fail; a hang is the harness's own watchdog to catch and turn
|
||||
into a non-zero exit, not something CI should have to `timeout(1)` around.
|
||||
* `--json` writes one machine-readable result file (measured p99, RSS series, recovery
|
||||
time) so the job can upload it as an artifact and a regression is a diff, not a re-run.
|
||||
* no network, no writes outside a tempdir, no leaked child processes on failure.
|
||||
### CI jobs — wired
|
||||
|
||||
### CI job — pre-drafted, add once the harness path is fixed
|
||||
`gui-dod` (per-PR: `scroll-60fps` + `unhappy-path`) and `gui-dod-nightly` (`rss-flat`,
|
||||
`schedule`/`workflow_dispatch` only) are live in `ci.yml`. Both bootstrap, build
|
||||
`gui-dod-harness`, `npm ci` in `tools/mockd`, run `gui/tests/dod/run.sh`, and upload the
|
||||
`--json` output as an artifact — no Xvfb step, since `run.sh` already runs offscreen.
|
||||
|
||||
```yaml
|
||||
gui-dod:
|
||||
# Per-PR GUI gates. The 10-minute rss-flat gate is in gui-dod-nightly, not here.
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Bootstrap toolchain
|
||||
run: sudo ./tools/bootstrap.sh
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22' # tools/mockd
|
||||
- name: Configure + build
|
||||
run: |
|
||||
cmake --preset dev
|
||||
cmake --build --preset dev --target velox-gui
|
||||
- name: Install mockd
|
||||
run: cd tools/mockd && npm ci
|
||||
- name: Xvfb + gates
|
||||
run: |
|
||||
sudo apt-get install -y --no-install-recommends xvfb
|
||||
xvfb-run -a gui/tests/dod/run.sh scroll-60fps --json scroll.json # TODO(GUI): path
|
||||
xvfb-run -a gui/tests/dod/run.sh unhappy-path --json unhappy.json
|
||||
- uses: actions/upload-artifact@v4
|
||||
if: always()
|
||||
with:
|
||||
name: gui-dod-${{ github.run_id }}
|
||||
path: "*.json"
|
||||
### Forced red, once per gate, before wiring it required
|
||||
|
||||
gui-dod-nightly:
|
||||
if: github.event_name == 'schedule'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Bootstrap toolchain
|
||||
run: sudo ./tools/bootstrap.sh
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22'
|
||||
- name: Configure + build
|
||||
run: |
|
||||
cmake --preset dev
|
||||
cmake --build --preset dev --target velox-gui
|
||||
- run: cd tools/mockd && npm ci
|
||||
- name: RSS soak (10 min)
|
||||
run: |
|
||||
sudo apt-get install -y --no-install-recommends xvfb
|
||||
xvfb-run -a gui/tests/dod/run.sh rss-flat --json rss.json
|
||||
- uses: actions/upload-artifact@v4
|
||||
if: always()
|
||||
with:
|
||||
name: gui-dod-rss-${{ github.run_id }}
|
||||
path: rss.json
|
||||
```
|
||||
| Gate | Forced via | Observed |
|
||||
|---|---|---|
|
||||
| `scroll-60fps` | `VELOX_DOD_FRAME_BUDGET_MS=0.01` | `FAIL: p99=50.73 ms mean=31.34 ms max=52.88 ms budget=0.01 ms over 240 steps, 10000 rows`, exit 1 |
|
||||
| `rss-flat` | `VELOX_DOD_RSS_SLACK_KIB=-999999999` (guarantees `growthKiB > slack` regardless of actual RSS behavior that run) | `FAIL: growth=13644 KiB slack=-999999999 KiB over 15s (warmup 1s), 10000 rows`, exit 1 |
|
||||
| `unhappy-path` | ran `gui-dod-harness` directly (bypassing `run.sh`, which always starts a working `mockd`) against a `--sock` path with nothing listening | 75 s of `Connecting`/`Reconnecting`, then `FAIL: dod_harness watchdog fired — hung past 75s`, exit 3 |
|
||||
|
||||
`gui-dod-nightly` needs `if: github.event_name == 'schedule'` (the `nightly-integration`
|
||||
job below already added that trigger to `ci.yml` — `cron: '17 3 * * *'` — so this no
|
||||
longer needs its own).
|
||||
The real (non-forced) runs all pass live: `scroll-60fps` p99 51.28 ms against a 66.4 ms
|
||||
budget (ASan/UBSan build), `rss-flat` growth 13504 KiB against the 20480 KiB slack over a
|
||||
15 s smoke duration, `unhappy-path` all three phases `PASS` with `finalConnected=true`.
|
||||
|
||||
### Known gap: `unhappy-path`'s drop-connection phase doesn't drop anything
|
||||
|
||||
Filed by GUI in `gui/docs/proto-requests-m1.md`: `mockd --drop-connection` only works
|
||||
over WebSocket — `startUds()` never wires the periodic-drop timer `startWs()` has, so the
|
||||
UDS transport (GUI/CLI/nmhost's only one) never sees a connection actually die. GUI's own
|
||||
live verification: 45 s observing a `--drop-connection 5` mockd over UDS, `stateChanged`
|
||||
never fires. The phase still runs and its JSON honestly records `sawDisruption: false`
|
||||
rather than silently passing as if it proved something — that's what the real run above
|
||||
shows, `pass: true` alongside `sawDisruption: false`, so a reviewer reading the JSON sees
|
||||
exactly how much this phase currently covers.
|
||||
|
||||
This is PROTO's fix, not PKG/QA's or GUI's to route around locally — coordinating on it
|
||||
rather than patching mockd from this lane. Once PROTO lands `dropEverySec` on `startUds`,
|
||||
`unhappy-path`'s drop-connection phase starts exercising a real drop and this note comes
|
||||
out; until then `gui-dod` stays required as specified (crash/hang/never-reconnects still
|
||||
catch real regressions), just not yet catching a swallowed real disconnect.
|
||||
|
||||
### Status
|
||||
|
||||
Blocked on GUI's harness. Not urgent (GUI M1 DoD, not M0). When GUI files the follow-up
|
||||
with the real `run.sh` path and the `rss-flat` slack number, PKG/QA drops the `TODO(GUI)`
|
||||
markers and marks `gui-dod` required. The `schedule:` trigger `gui-dod-nightly` needs is
|
||||
already in `ci.yml`.
|
||||
Wired and required: `gui-dod` runs per-PR, `gui-dod-nightly` on schedule. Revisit once
|
||||
PROTO's UDS `--drop-connection` fix lands (see above).
|
||||
|
||||
Reference in New Issue
Block a user