samiandClaude Sonnet 5
dad88fce3f
PROTO wired the suite into ctest (label "conformance": the end-to-end `conformance` test that shells to run.sh, plus the native `conformance_cpp`); the CI job called run.sh directly. Two entry points, and the required M0 gate exercised only one of them, so the ctest registration could rot. The conformance job now configures, builds velox_conformance_cpp, and runs `ctest --preset dev -L conformance`. The dev test preset's noTestsAction: error is the rot guard: an empty label match exits non-zero instead of the old silent pass. build/sanitizers exclude the heavy e2e test with -E '^conformance$' (the dedicated job owns that run; conformance_cpp still runs under every sanitizer). ADR 0014 records the decision. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_0143aKiohmDiyefJBwHDJJqw
2.7 KiB
ADR 0014 — The conformance suite has one entry point: ctest -L conformance
Status: accepted · Date: 2026-09-10 · Lane: PKG/QA
Context
The conformance suite could be started two ways, and both were live:
tests/conformance/run.sh— the script a developer runs by hand; the CIconformancejob called it directly.ctest -L conformance— PROTO wiredtests/conformance/CMakeLists.txtto registerconformance(which shells out torun.sh) andconformance_cpp(the finer-grained native runner) under that label.
Nothing was wrong with either, but the CI conformance job — the one named as the M0
exit gate in .github/BRANCH_PROTECTION.md — exercised only the script. The ctest
registration was reached only incidentally, by the unfiltered ctest run in the build
and sanitizers jobs. A change that broke the registration (a deleted add_test, a
renamed label, a WORKING_DIRECTORY regression) would not fail the check that is supposed
to be about conformance; it would fail a build job, or nothing.
Two entry points also means two answers to "did the suite pass", free to drift.
Decision
ctest -L conformance is the canonical entry point. run.sh remains the thing it runs —
tests/conformance/CMakeLists.txt registers it verbatim — but CI reaches it only through
ctest, the same way a developer does.
- The
conformancejob runscmake --preset dev, builds just thevelox_conformance_cpptarget, thenctest --preset dev -L conformance --output-on-failure. This puts PROTO's registration on the exercised path. - The rot guard is
noTestsAction: errorin thedevtest preset: if the label ever matches zero tests, ctest exits non-zero instead of the old silent "0 tests, exit 0". - The
buildandsanitizersjobs exclude the end-to-end test with-E '^conformance$'. It installs two npm trees and runs a node process against a self-compiled, unsanitizedg++binary — running that once, in the dedicated job, is enough; sanitizing it buys nothing.conformance_cppis not excluded and still runs on every matrix leg, which is where ASan/UBSan/TSan coverage of the generated wire code is worth having.
Consequences
- One definition of "the suite passed".
run.shstill works standalone and unchanged for local use and forrun.sh --uds … --ws-port …against a real daemon; that path is out of scope for this ADR, which is only about what CI treats as authoritative.- If PROTO ever wants the two labelled tests separated (e.g. a
conformance-e2elabel so jobs can select them independently without a name regex), that is atests/conformance/change on their side; the-E '^conformance$'here is the PKG/QA-side stopgap until then.