samiandClaude Sonnet 5
203d4a662f
Verified PKG's landed CI (.github/workflows/ci.yml, db7650b/8f45815) against this lane's actual tree, per the standing instruction that conformance as a required check is this lane's DoD to verify, not PKG's. It wasn't running: the `conformance` job's presence-check looks for tests/conformance/CMakeLists.txt or tests/conformance/package.json, and neither existed -- the job was silently short-circuiting to a green "skipped" on every PR, forever. The M0 exit gate was not gating anything. tests/conformance/CMakeLists.txt registers one ctest entry, labeled "conformance", that shells out to run.sh -- the exact command tests/conformance/README.md tells a human to run locally, so there is one definition of "the suite passed", not a CMake-flavoured near-duplicate of it. cpp/CMakeLists.txt's existing conformance_cpp test gets the same label, for a lane iterating on core/generated/ who wants the fast native-only path. Fixed a second landmine found while wiring this: the root CMakeLists.txt only find_package(nlohmann_json)'s when daemon/CMakeLists.txt exists, since daemon is its real consumer -- but daemon hasn't landed yet, so add_subdirectory(tests/conformance) would have failed to configure the moment this file existed, on every machine, until daemon merges. Fixed inside tests/conformance/cpp/CMakeLists.txt with an if(NOT TARGET) guard rather than widening the root file's condition, which is PKG's to change. run.sh now installs its own Python deps (jsonschema, referencing) on demand: they aren't in tools/bootstrap.sh's apt list -- that's PKG's script, these are this suite's own dependency -- so a bare CI image would otherwise fail check_contract.py with an ImportError before this suite even started. Verified end to end: `cmake --preset dev && ctest --test-dir build/dev -R '^conformance$'` passes in 23.8s, exercising the exact command and label the CI job uses. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_012fgjnqFCS5h5L7gZTZo3rV
tests/conformance — one suite, three runners
This is a required check on every lane's PR. It is the mechanism that makes four parallel lanes safe: the C++ daemon and the TypeScript extension are proved compatible without either having run against the other.
./tests/conformance/run.sh # starts its own mockd
./tests/conformance/run.sh --uds /run/user/1000/velox/velox.sock --ws-port 52000
The runners
| Runner | Needs | Asserts |
|---|---|---|
check_contract.py |
python3, jsonschema | schemas parse and resolve; the documented surface matches the schema surface both ways; every method has a success fixture; every fixture validates; SettingKey and Settings agree; committed generated code is not stale |
cpp/ |
a C++23 compiler, nlohmann | every golden payload parses into the generated structs, serialises back stably, and goes through the real dispatch(); privileged methods are refused -32003 over the WebSocket |
ts/replay.ts |
node ≥ 20 | a live server answers every fixture over every transport the contract allows, and the reply passes the generated validator |
run.sh also runs one scenario that cannot be shown against a healthy server: with the
daemon answering slower than capture.offer's 750 ms deadline, the client must give up and
let Firefox take the download. That is the fail-open guarantee, and it is checked here.
What "passing" means
The runners check the contract, not the implementation's opinions. Results are compared by
shape and validated against the generated validators; error codes are compared exactly.
Byte-equality with a golden file is deliberately not asserted, because a live daemon
returns its own ids and its own clock — see contracts/fixtures/README.md.
Adding a method without a fixture fails check_contract.py. Regenerating and forgetting to
commit the output fails it too.
Request to lane PKG/QA
.github/ belongs to PKG/QA, so this suite is not wired into CI by lane PROTO. Please add
it as a required status check on every branch, roughly:
conformance:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version: '22' }
- run: sudo apt-get update && sudo apt-get install -y nlohmann-json3-dev
- run: pip install jsonschema referencing
- run: ./tests/conformance/run.sh
The suite needs: python3 with jsonschema, a C++23 compiler, nlohmann-json, and Node
≥ 20. It starts and stops its own mockd; nothing else needs to be running.