pkg: run conformance through ctest as the one canonical path
PROTO wired the suite into ctest (label "conformance": the end-to-end `conformance` test that shells to run.sh, plus the native `conformance_cpp`); the CI job called run.sh directly. Two entry points, and the required M0 gate exercised only one of them, so the ctest registration could rot. The conformance job now configures, builds velox_conformance_cpp, and runs `ctest --preset dev -L conformance`. The dev test preset's noTestsAction: error is the rot guard: an empty label match exits non-zero instead of the old silent pass. build/sanitizers exclude the heavy e2e test with -E '^conformance$' (the dedicated job owns that run; conformance_cpp still runs under every sanitizer). ADR 0014 records the decision. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_0143aKiohmDiyefJBwHDJJqw
This commit is contained in:
@@ -0,0 +1,50 @@
|
||||
# ADR 0014 — The conformance suite has one entry point: `ctest -L conformance`
|
||||
|
||||
**Status:** accepted · **Date:** 2026-09-10 · **Lane:** PKG/QA
|
||||
|
||||
## Context
|
||||
|
||||
The conformance suite could be started two ways, and both were live:
|
||||
|
||||
* `tests/conformance/run.sh` — the script a developer runs by hand; the CI `conformance`
|
||||
job called it directly.
|
||||
* `ctest -L conformance` — PROTO wired `tests/conformance/CMakeLists.txt` to register
|
||||
`conformance` (which shells out to `run.sh`) and `conformance_cpp` (the finer-grained
|
||||
native runner) under that label.
|
||||
|
||||
Nothing was wrong with either, but the CI `conformance` job — the one named as the M0
|
||||
exit gate in `.github/BRANCH_PROTECTION.md` — exercised only the script. The ctest
|
||||
registration was reached only incidentally, by the unfiltered `ctest` run in the `build`
|
||||
and `sanitizers` jobs. A change that broke the registration (a deleted `add_test`, a
|
||||
renamed label, a `WORKING_DIRECTORY` regression) would not fail the check that is supposed
|
||||
to be about conformance; it would fail a build job, or nothing.
|
||||
|
||||
Two entry points also means two answers to "did the suite pass", free to drift.
|
||||
|
||||
## Decision
|
||||
|
||||
`ctest -L conformance` is the canonical entry point. `run.sh` remains the thing it runs —
|
||||
`tests/conformance/CMakeLists.txt` registers it verbatim — but CI reaches it only through
|
||||
ctest, the same way a developer does.
|
||||
|
||||
* The `conformance` job runs `cmake --preset dev`, builds just the `velox_conformance_cpp`
|
||||
target, then `ctest --preset dev -L conformance --output-on-failure`. This puts PROTO's
|
||||
registration on the exercised path.
|
||||
* The rot guard is `noTestsAction: error` in the `dev` test preset: if the label ever
|
||||
matches zero tests, ctest exits non-zero instead of the old silent "0 tests, exit 0".
|
||||
* The `build` and `sanitizers` jobs exclude the end-to-end test with `-E '^conformance$'`.
|
||||
It installs two npm trees and runs a node process against a self-compiled, unsanitized
|
||||
`g++` binary — running that once, in the dedicated job, is enough; sanitizing it buys
|
||||
nothing. `conformance_cpp` is *not* excluded and still runs on every matrix leg, which
|
||||
is where ASan/UBSan/TSan coverage of the generated wire code is worth having.
|
||||
|
||||
## Consequences
|
||||
|
||||
* One definition of "the suite passed".
|
||||
* `run.sh` still works standalone and unchanged for local use and for
|
||||
`run.sh --uds … --ws-port …` against a real daemon; that path is out of scope for this
|
||||
ADR, which is only about what CI treats as authoritative.
|
||||
* If PROTO ever wants the two labelled tests separated (e.g. a `conformance-e2e` label so
|
||||
jobs can select them independently without a name regex), that is a `tests/conformance/`
|
||||
change on their side; the `-E '^conformance$'` here is the PKG/QA-side stopgap until
|
||||
then.
|
||||
Reference in New Issue
Block a user