merge: lane/pkg-qa — GUI DoD gates wired, conformance required
CI / clang-format (push) Waiting to run
CI / testserver (push) Waiting to run
CI / bootstrap-script (push) Waiting to run
CI / bootstrap-script-2604 (push) Waiting to run
CI / extension-lint (push) Waiting to run
CI / build (clang) (push) Waiting to run
CI / build (gcc) (push) Waiting to run
CI / sanitizers (dev) (push) Waiting to run
CI / sanitizers (tsan) (push) Waiting to run
CI / clang-tidy (push) Waiting to run
CI / conformance (push) Waiting to run
CI / nightly-integration (push) Waiting to run
CI / gui-dod (push) Waiting to run
CI / gui-dod-nightly (push) Waiting to run
CI / clang-format (push) Waiting to run
CI / testserver (push) Waiting to run
CI / bootstrap-script (push) Waiting to run
CI / bootstrap-script-2604 (push) Waiting to run
CI / extension-lint (push) Waiting to run
CI / build (clang) (push) Waiting to run
CI / build (gcc) (push) Waiting to run
CI / sanitizers (dev) (push) Waiting to run
CI / sanitizers (tsan) (push) Waiting to run
CI / clang-tidy (push) Waiting to run
CI / conformance (push) Waiting to run
CI / nightly-integration (push) Waiting to run
CI / gui-dod (push) Waiting to run
CI / gui-dod-nightly (push) Waiting to run
This commit is contained in:
@@ -18,8 +18,9 @@ policy so it can be re-applied or audited.
|
|||||||
| `bootstrap-script-2604` | now — real `--with-clang` install in a 26.04 container; the release the project ships on |
|
| `bootstrap-script-2604` | now — real `--with-clang` install in a 26.04 container; the release the project ships on |
|
||||||
| `build (gcc)` / `build (clang)` | now — core, daemon and gui have merged |
|
| `build (gcc)` / `build (clang)` | now — core, daemon and gui have merged |
|
||||||
| `sanitizers (dev)` / `sanitizers (tsan)` | now — core, daemon and gui have merged |
|
| `sanitizers (dev)` / `sanitizers (tsan)` | now — core, daemon and gui have merged |
|
||||||
| `conformance` | **now — `tests/conformance/` has landed; this is the M0 exit gate** |
|
| `conformance` | **now — `tests/conformance/` has landed; this is the M0 exit gate.** Includes the live-`veloxd` runner (step 3b of `run.sh`), unconditional in the script — see `docs/adr/0019-live-veloxd-conformance-is-required.md`. |
|
||||||
| `extension-lint` | now — `extension/` has merged (MV3 manifest + esbuild build) |
|
| `extension-lint` | now — `extension/` has merged (MV3 manifest + esbuild build) |
|
||||||
|
| `gui-dod` | now — `gui/tests/dod/` has landed (GUI M1 DoD gates R3: `scroll-60fps`, `unhappy-path`); see `tests/integration/README.md#gui-m1-definition-of-done-gates-r3`. `gui-dod-nightly` (`rss-flat`) is schedule-only and cannot be a required PR check. |
|
||||||
|
|
||||||
`clang-tidy` is intentionally **not** required through M1 (`continue-on-error: true`,
|
`clang-tidy` is intentionally **not** required through M1 (`continue-on-error: true`,
|
||||||
`.clang-tidy` has `WarningsAsErrors: ''`). Make it required at M2.
|
`.clang-tidy` has `WarningsAsErrors: ''`). Make it required at M2.
|
||||||
|
|||||||
@@ -248,3 +248,56 @@ jobs:
|
|||||||
run: cmake --build --preset dev --target veloxd
|
run: cmake --build --preset dev --target veloxd
|
||||||
- name: Nightly integration run
|
- name: Nightly integration run
|
||||||
run: python3 tests/integration/nightly_run.py --veloxd build/dev/bin/veloxd --tasks 50 --timeout 180
|
run: python3 tests/integration/nightly_run.py --veloxd build/dev/bin/veloxd --tasks 50 --timeout 180
|
||||||
|
|
||||||
|
gui-dod:
|
||||||
|
# Per-PR GUI M1 DoD gates (gui/docs/pkg-qa-requests-m1.md R3): scroll-60fps and
|
||||||
|
# unhappy-path. The 10-minute rss-flat gate is gui-dod-nightly, not here. GUI's
|
||||||
|
# harness defaults QT_QPA_PLATFORM=offscreen itself, so no Xvfb/compositor needed.
|
||||||
|
# See tests/integration/README.md#gui-m1-definition-of-done-gates-r3 for what each
|
||||||
|
# gate catches and the forced-failure transcript proving it isn't vacuous.
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- name: Bootstrap toolchain
|
||||||
|
run: sudo ./tools/bootstrap.sh
|
||||||
|
- uses: actions/setup-node@v4
|
||||||
|
with:
|
||||||
|
node-version: '22' # tools/mockd
|
||||||
|
- name: Configure + build
|
||||||
|
run: |
|
||||||
|
cmake --preset dev
|
||||||
|
cmake --build --preset dev --target gui-dod-harness
|
||||||
|
- name: Install mockd
|
||||||
|
run: cd tools/mockd && npm ci
|
||||||
|
- name: Gates
|
||||||
|
run: |
|
||||||
|
gui/tests/dod/run.sh scroll-60fps --json scroll.json
|
||||||
|
gui/tests/dod/run.sh unhappy-path --json unhappy.json
|
||||||
|
- uses: actions/upload-artifact@v4
|
||||||
|
if: always()
|
||||||
|
with:
|
||||||
|
name: gui-dod-${{ github.run_id }}
|
||||||
|
path: "*.json"
|
||||||
|
|
||||||
|
gui-dod-nightly:
|
||||||
|
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- name: Bootstrap toolchain
|
||||||
|
run: sudo ./tools/bootstrap.sh
|
||||||
|
- uses: actions/setup-node@v4
|
||||||
|
with:
|
||||||
|
node-version: '22'
|
||||||
|
- name: Configure + build
|
||||||
|
run: |
|
||||||
|
cmake --preset dev
|
||||||
|
cmake --build --preset dev --target gui-dod-harness
|
||||||
|
- run: cd tools/mockd && npm ci
|
||||||
|
- name: RSS soak (10 min)
|
||||||
|
run: gui/tests/dod/run.sh rss-flat --json rss.json
|
||||||
|
- uses: actions/upload-artifact@v4
|
||||||
|
if: always()
|
||||||
|
with:
|
||||||
|
name: gui-dod-rss-${{ github.run_id }}
|
||||||
|
path: rss.json
|
||||||
|
|||||||
@@ -0,0 +1,45 @@
|
|||||||
|
# ADR 0019 — The live-`veloxd` conformance runner is a required check, as-is
|
||||||
|
|
||||||
|
**Status:** accepted · **Date:** 2026-09-12 · **Lane:** PKG/QA
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
`tests/conformance/run.sh` step 3b (ADR 0014's `conformance` ctest, already required
|
||||||
|
per `.github/BRANCH_PROTECTION.md`) replays every fixture against a real, isolated
|
||||||
|
`veloxd` it builds and starts — not just mockd, which only proves the TS client and the
|
||||||
|
fixtures agree with each other. This is the runner that can catch `veloxd` disagreeing
|
||||||
|
with its own contract, and it is unconditional in `run.sh` (`set -euo pipefail`, no
|
||||||
|
skip flag): it already runs, and already blocks, inside the `conformance` job.
|
||||||
|
|
||||||
|
What was open was whether to treat that as a settled, defended gate or as something
|
||||||
|
still provisional while `daemon/docs/deferrals.md`'s D1-D4b stub handlers were excused
|
||||||
|
via `veloxd-xfail.json`. PROTO's update: 57/57 fixtures pass on `main`, and the xfail
|
||||||
|
allowlist is down to 18 entries from 34 as DAEMON lands the deferred handlers behind
|
||||||
|
them.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
The live-`veloxd` conformance runner stays required — no change to CI is needed, since
|
||||||
|
it already runs inside the already-required `conformance` job (ADR 0014). What this ADR
|
||||||
|
records is the standing: PKG/QA is not carving out an exception, a `continue-on-error`,
|
||||||
|
or a separate advisory job for it while the xfail list shrinks. A regression here fails
|
||||||
|
the same required check a schema mismatch would.
|
||||||
|
|
||||||
|
Verified, not assumed: `tests/conformance/veloxd-xfail.json` has 18 entries as of this
|
||||||
|
ADR (`python3 -c "import json; print(len(json.load(open('tests/conformance/veloxd-xfail.json'))))"`).
|
||||||
|
Each remaining entry excuses one still-stubbed handler on `deferrals.md`'s D-list, not a
|
||||||
|
real disagreement between `veloxd` and its contract — `tests/conformance/README.md`
|
||||||
|
already draws that line (an entry for anything else is a `run.sh`-detected "xfail entry
|
||||||
|
unexpectedly passed" or a straight failure, not a quiet pass).
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- No `ci.yml` or `BRANCH_PROTECTION.md` change: `conformance` was already listed
|
||||||
|
required, and this runner was already inside it.
|
||||||
|
- The xfail list is a visible, shrinking number, not a static allowance — as DAEMON
|
||||||
|
clears more of `deferrals.md`'s D-list, entries come out of
|
||||||
|
`tests/conformance/veloxd-xfail.json`, and `replay.ts` fails loudly (per
|
||||||
|
`applyXfail`'s "unexpectedly passed" check) if one is left in after its handler ships.
|
||||||
|
- Nothing here changes who owns what: `veloxd-xfail.json` and `run.sh` stay PROTO's;
|
||||||
|
PKG/QA's role is the branch-protection policy this ADR confirms, not the runner
|
||||||
|
itself.
|
||||||
+46
-72
@@ -88,88 +88,62 @@ manual testing.
|
|||||||
nowhere to run. GUI owns the harness; PKG/QA owns the CI job. This is the wiring contract
|
nowhere to run. GUI owns the harness; PKG/QA owns the CI job. This is the wiring contract
|
||||||
so the two halves meet without another round trip.
|
so the two halves meet without another round trip.
|
||||||
|
|
||||||
### What PKG/QA needs from GUI
|
### What GUI built, and the invocation contract
|
||||||
|
|
||||||
A driver invoked as `gui/tests/dod/run.sh <gate> [--json <path>]` (exact path TBD by GUI),
|
`gui/tests/dod/run.sh <gate> [--json <path>]` (`gate` one of `scroll-60fps` / `rss-flat`
|
||||||
headless-capable (Xvfb or offscreen `QT_QPA_PLATFORM`), with:
|
/ `unhappy-path`), backed by `gui/tests/dod/dod_harness.cpp` (target `gui-dod-harness`).
|
||||||
|
Headless by default — `run.sh` sets `QT_QPA_PLATFORM=offscreen` itself, so the CI jobs
|
||||||
|
below need no Xvfb.
|
||||||
|
|
||||||
| `<gate>` | Pass / fail condition | Budget |
|
| `<gate>` | Pass / fail condition | Budget |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `scroll-60fps` | `mockd --tasks 10000`, scripted fling scroll; **fail** if p99 frame > 16.6 ms | per-PR |
|
| `scroll-60fps` | `mockd --tasks 10000`, scripted fling scroll; **fail** if p99 frame > 16.6 ms (×4 under an ASan/UBSan build — `VELOX_DOD_FRAME_BUDGET_MS` overrides outright) | per-PR |
|
||||||
| `rss-flat` | `mockd --tasks 10000` + progress events, 10 min; **fail** if RSS growth > a fixed slack (GUI picks the number, states it) | nightly |
|
| `rss-flat` | `mockd --tasks 10000` + progress events, 10 min; **fail** if RSS growth past warm-up exceeds a 20 MiB slack (`VELOX_DOD_RSS_SLACK_KIB` overrides) | nightly |
|
||||||
| `unhappy-path` | `mockd --slow` / `--flaky <f>` / `--drop-connection <s>`; **fail** on crash, on watchdog-detected hang, or if connection state never returns to `Connected` | per-PR |
|
| `unhappy-path` | Three phases (`--slow` / `--flaky <f>` / `--drop-connection <s>`), one `mockd` restart each; **fail** on crash, on the harness's own watchdog firing (75 s), or if connection state never (re)reaches `Connected` | per-PR |
|
||||||
|
|
||||||
Contract:
|
Contract, as built: exit `0` pass, non-zero fail; a hang is the harness's own watchdog
|
||||||
|
converting itself into a non-zero exit (`3`), never something CI needs `timeout(1)`
|
||||||
|
around; `--json` writes one result object per gate (`unhappy-path` merges its three
|
||||||
|
phases into one file); no network, no writes outside a tempdir, no leaked child process
|
||||||
|
on any exit path (`run.sh`'s `trap cleanup EXIT INT TERM`).
|
||||||
|
|
||||||
* exit `0` pass, non-zero fail; a hang is the harness's own watchdog to catch and turn
|
### CI jobs — wired
|
||||||
into a non-zero exit, not something CI should have to `timeout(1)` around.
|
|
||||||
* `--json` writes one machine-readable result file (measured p99, RSS series, recovery
|
|
||||||
time) so the job can upload it as an artifact and a regression is a diff, not a re-run.
|
|
||||||
* no network, no writes outside a tempdir, no leaked child processes on failure.
|
|
||||||
|
|
||||||
### CI job — pre-drafted, add once the harness path is fixed
|
`gui-dod` (per-PR: `scroll-60fps` + `unhappy-path`) and `gui-dod-nightly` (`rss-flat`,
|
||||||
|
`schedule`/`workflow_dispatch` only) are live in `ci.yml`. Both bootstrap, build
|
||||||
|
`gui-dod-harness`, `npm ci` in `tools/mockd`, run `gui/tests/dod/run.sh`, and upload the
|
||||||
|
`--json` output as an artifact — no Xvfb step, since `run.sh` already runs offscreen.
|
||||||
|
|
||||||
```yaml
|
### Forced red, once per gate, before wiring it required
|
||||||
gui-dod:
|
|
||||||
# Per-PR GUI gates. The 10-minute rss-flat gate is in gui-dod-nightly, not here.
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
steps:
|
|
||||||
- uses: actions/checkout@v4
|
|
||||||
- name: Bootstrap toolchain
|
|
||||||
run: sudo ./tools/bootstrap.sh
|
|
||||||
- uses: actions/setup-node@v4
|
|
||||||
with:
|
|
||||||
node-version: '22' # tools/mockd
|
|
||||||
- name: Configure + build
|
|
||||||
run: |
|
|
||||||
cmake --preset dev
|
|
||||||
cmake --build --preset dev --target velox-gui
|
|
||||||
- name: Install mockd
|
|
||||||
run: cd tools/mockd && npm ci
|
|
||||||
- name: Xvfb + gates
|
|
||||||
run: |
|
|
||||||
sudo apt-get install -y --no-install-recommends xvfb
|
|
||||||
xvfb-run -a gui/tests/dod/run.sh scroll-60fps --json scroll.json # TODO(GUI): path
|
|
||||||
xvfb-run -a gui/tests/dod/run.sh unhappy-path --json unhappy.json
|
|
||||||
- uses: actions/upload-artifact@v4
|
|
||||||
if: always()
|
|
||||||
with:
|
|
||||||
name: gui-dod-${{ github.run_id }}
|
|
||||||
path: "*.json"
|
|
||||||
|
|
||||||
gui-dod-nightly:
|
| Gate | Forced via | Observed |
|
||||||
if: github.event_name == 'schedule'
|
|---|---|---|
|
||||||
runs-on: ubuntu-latest
|
| `scroll-60fps` | `VELOX_DOD_FRAME_BUDGET_MS=0.01` | `FAIL: p99=50.73 ms mean=31.34 ms max=52.88 ms budget=0.01 ms over 240 steps, 10000 rows`, exit 1 |
|
||||||
steps:
|
| `rss-flat` | `VELOX_DOD_RSS_SLACK_KIB=-999999999` (guarantees `growthKiB > slack` regardless of actual RSS behavior that run) | `FAIL: growth=13644 KiB slack=-999999999 KiB over 15s (warmup 1s), 10000 rows`, exit 1 |
|
||||||
- uses: actions/checkout@v4
|
| `unhappy-path` | ran `gui-dod-harness` directly (bypassing `run.sh`, which always starts a working `mockd`) against a `--sock` path with nothing listening | 75 s of `Connecting`/`Reconnecting`, then `FAIL: dod_harness watchdog fired — hung past 75s`, exit 3 |
|
||||||
- name: Bootstrap toolchain
|
|
||||||
run: sudo ./tools/bootstrap.sh
|
|
||||||
- uses: actions/setup-node@v4
|
|
||||||
with:
|
|
||||||
node-version: '22'
|
|
||||||
- name: Configure + build
|
|
||||||
run: |
|
|
||||||
cmake --preset dev
|
|
||||||
cmake --build --preset dev --target velox-gui
|
|
||||||
- run: cd tools/mockd && npm ci
|
|
||||||
- name: RSS soak (10 min)
|
|
||||||
run: |
|
|
||||||
sudo apt-get install -y --no-install-recommends xvfb
|
|
||||||
xvfb-run -a gui/tests/dod/run.sh rss-flat --json rss.json
|
|
||||||
- uses: actions/upload-artifact@v4
|
|
||||||
if: always()
|
|
||||||
with:
|
|
||||||
name: gui-dod-rss-${{ github.run_id }}
|
|
||||||
path: rss.json
|
|
||||||
```
|
|
||||||
|
|
||||||
`gui-dod-nightly` needs `if: github.event_name == 'schedule'` (the `nightly-integration`
|
The real (non-forced) runs all pass live: `scroll-60fps` p99 51.28 ms against a 66.4 ms
|
||||||
job below already added that trigger to `ci.yml` — `cron: '17 3 * * *'` — so this no
|
budget (ASan/UBSan build), `rss-flat` growth 13504 KiB against the 20480 KiB slack over a
|
||||||
longer needs its own).
|
15 s smoke duration, `unhappy-path` all three phases `PASS` with `finalConnected=true`.
|
||||||
|
|
||||||
|
### Known gap: `unhappy-path`'s drop-connection phase doesn't drop anything
|
||||||
|
|
||||||
|
Filed by GUI in `gui/docs/proto-requests-m1.md`: `mockd --drop-connection` only works
|
||||||
|
over WebSocket — `startUds()` never wires the periodic-drop timer `startWs()` has, so the
|
||||||
|
UDS transport (GUI/CLI/nmhost's only one) never sees a connection actually die. GUI's own
|
||||||
|
live verification: 45 s observing a `--drop-connection 5` mockd over UDS, `stateChanged`
|
||||||
|
never fires. The phase still runs and its JSON honestly records `sawDisruption: false`
|
||||||
|
rather than silently passing as if it proved something — that's what the real run above
|
||||||
|
shows, `pass: true` alongside `sawDisruption: false`, so a reviewer reading the JSON sees
|
||||||
|
exactly how much this phase currently covers.
|
||||||
|
|
||||||
|
This is PROTO's fix, not PKG/QA's or GUI's to route around locally — coordinating on it
|
||||||
|
rather than patching mockd from this lane. Once PROTO lands `dropEverySec` on `startUds`,
|
||||||
|
`unhappy-path`'s drop-connection phase starts exercising a real drop and this note comes
|
||||||
|
out; until then `gui-dod` stays required as specified (crash/hang/never-reconnects still
|
||||||
|
catch real regressions), just not yet catching a swallowed real disconnect.
|
||||||
|
|
||||||
### Status
|
### Status
|
||||||
|
|
||||||
Blocked on GUI's harness. Not urgent (GUI M1 DoD, not M0). When GUI files the follow-up
|
Wired and required: `gui-dod` runs per-PR, `gui-dod-nightly` on schedule. Revisit once
|
||||||
with the real `run.sh` path and the `rss-flat` slack number, PKG/QA drops the `TODO(GUI)`
|
PROTO's UDS `--drop-connection` fix lands (see above).
|
||||||
markers and marks `gui-dod` required. The `schedule:` trigger `gui-dod-nightly` needs is
|
|
||||||
already in `ci.yml`.
|
|
||||||
|
|||||||
Reference in New Issue
Block a user