sami/vdm
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

proto: wire veloxd into the conformance suite's canonical entry point

Browse Source 
run.sh's TS runner only ever started mockd; veloxd existed but nothing in
the ctest -L conformance path touched it, so mockd's always-valid fixtures
were the only thing ts/replay.ts ever saw. That let a real bug through:
veloxd's download.get can return segments: 0, which TaskSummary.segments
forbids (minimum 1, required) — nothing caught it.

Add a 3b step to run.sh (still the one canonical entry, per ADR 0014):
builds veloxd, starts it isolated (its own XDG_RUNTIME_DIR/XDG_DATA_HOME/
XDG_CONFIG_HOME), seeds saveTo.allowedRoots/defaultDir directly into the
isolated velox.db (settings.set is itself a stub, and the default
~/Downloads root doesn't isolate download.add's writes), then replays
every fixture against it over both transports.

Most handlers are still stubs (daemon/docs/deferrals.md D1-D4b). Fixtures
that hit them get an expected-failure entry in the new veloxd-xfail.json,
loaded by replay.ts's new --xfail flag. This is a maintained allowlist,
not a snapshot: a listed fixture that unexpectedly *passes* is flipped
back to a failure (applyXfail), so the list can only shrink as DAEMON
lands handlers, never rot into a list nobody rechecks. download.get's
segments: 0 is deliberately *not* on it — that's the regression this
step exists to catch.

Also hardened setupBindings: a server that can't even complete fixture
binding used to take the whole runner down with an uncaught exception
before a single fixture was checked. It's now a reported Outcome instead,
so the run still produces a coherent report. That robustness fix earned
its keep immediately: veloxd's download.add crashes on startMode
"later" (a valid, documented StartMode — "the File Info dialog's
Download Later button") with a SQLite CHECK constraint violation, because
migrations/0001_initial.sql's start_mode CHECK never had 'later' added to
it (and includes 'manual'/'auto', neither a contract value). That's a
second, more severe bug this wiring found, unrelated to segments: 0 and
currently blocking most of the veloxd run — filed for DAEMON in
tests/conformance/README.md, not fixed here (out of lane). capture.offer
and capture.getRules are also stubs but missing from deferrals.md's
D-list; xfailed with a note asking DAEMON to add the row.

Verified live once against a real, isolated veloxd before this session's
sandbox became persistently contended for veloxd's single-instance lock
(UID-scoped, not namespaced by XDG_RUNTIME_DIR — daemon/src/main.cpp;
documented as a caveat in the README): it built, started isolated, seeded
settings, connected over both transports, and surfaced the startMode bug
above as a real, non-xfailed failure — confirming the whole pipeline
including --xfail end to end. segments: 0 is confirmed by direct reading
of daemon/src/store/tasks.{hpp,cpp} (TaskRow::eff_segments defaults to 0,
copied verbatim into TaskSummary.segments) rather than by a second live
run reaching that specific fixture, since setup itself fails first on the
startMode bug above. mockd path re-verified green after these changes
(200/200, up from 196/196 — the new setup/$taskId outcomes are visible
and passing).

Recommendation for PKG: don't flip this required yet. The existing
`conformance` ctest entry is already a required check, and right now the
startMode bug fails most of the veloxd run, not just the one expected
segments: 0 case — merging as-is would block every lane's PRs on two
DAEMON bugs at once, one of them unrelated to what this task set out to
catch. Required once DAEMON lands a fix for startMode "later" at minimum;
segments: 0 can stay red for a while by design, same as any other tracked
regression.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01SFeUKLbdHizrJjLBeK7ffz
This commit is contained in:
sami
2026-09-11 12:52:13 +04:00
co-authored by Claude Sonnet 5
parent 0a38867579
commit 5b03eff926
4 changed files with 270 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/conformance/ts/replay.ts
+79 -9
View File
@@ -222,15 +222,29 @@ function bind(value: unknown, bindings: Record<string, string>): unknown {
/**
* Create the tasks the task-referencing fixtures bind to. Doing this per connection is
* what lets the same suite run against an empty veloxd and against a seeded mockd.
*
* A server that cannot even complete this setup (not a fixture, so nothing above can
* report it) still needs to be visible as a failure rather than an uncaught exception
* that takes the whole runner down before a single fixture is checked — so a failure
* here becomes an Outcome and setup moves on, leaving that binding unresolved (its
* fixtures will then fail on the placeholder, individually, same as any other bad value).
*/
async function setupBindings(conn: Conn): Promise<Record<string, string>> {
async function setupBindings(conn: Conn): Promise<{ bindings: Record<string, string>; setup: Outcome[] }> {
const bindings: Record<string, string> = {};
const setup: Outcome[] = [];
for (const [key, url] of [['$taskId', 'https://example.org/conformance-a.bin'],
['$taskId2', 'https://example.org/conformance-b.bin']] as const) {
const added = await conn.call('download.add', { url, startMode: 'later' });
bindings[key] = added.taskId;
try {
const added = await conn.call('download.add', { url, startMode: 'later' });
bindings[key] = added.taskId;
setup.push({ fixture: `setup/${key}`, transport: conn.transport, ok: true,
detail: 'download.add for the binding succeeded' });
} catch (err) {
setup.push({ fixture: `setup/${key}`, transport: conn.transport, ok: false,
detail: `download.add for the binding failed: ${String(err)}` });
}
}
return bindings;
return { bindings, setup };
}
async function replay(conn: Conn, fixtures: readonly Fixture[],
@@ -319,6 +333,56 @@ async function privilegeChecks(conn: Conn): Promise<Outcome[]> {
return out;
}
// ----------------------------------------------------------- expected failure
/**
* A fixture this runner is allowed to fail against the target server, with why. Used
* against veloxd, which still has stub handlers (daemon/docs/deferrals.md D1-D4b) that
* mockd does not: mockd always answers every fixture correctly, so this list is empty
* there and the mechanism does not apply.
*
* `fixture` matches Outcome.fixture exactly (the path printed in a FAIL line, e.g.
* "contracts/fixtures/download.pause.json"); `transport`, if given, narrows to one
* transport. This is a maintained allowlist, not a captured snapshot: an entry that no
* longer fails is a bug in the list, not a pass, so `applyXfail` turns that back into a
* failure rather than silently dropping the entry. That is what keeps the list shrinking
* as DAEMON lands handlers instead of quietly becoming a list nobody rechecks.
*/
interface XfailEntry {
fixture: string;
transport?: TransportName;
reason: string;
}
function loadXfail(path: string): XfailEntry[] {
const parsed = JSON.parse(readFileSync(path, 'utf8')) as unknown;
if (!Array.isArray(parsed)) throw new Error(`${path}: expected a JSON array`);
return parsed as XfailEntry[];
}
/**
* Reconciles outcomes against the allowlist. A listed fixture that failed is downgraded
* to a pass (its detail says why). A listed fixture that *passed* is flipped to a
* failure: the entry is stale and must be deleted from the list, not left to rot.
*/
function applyXfail(results: readonly Outcome[], xfail: readonly XfailEntry[]): Outcome[] {
const matches = (e: XfailEntry, r: Outcome): boolean =>
e.fixture === r.fixture && (e.transport === undefined || e.transport === r.transport);
return results.map((r) => {
const entry = xfail.find((e) => matches(e, r));
if (!entry) return r;
if (!r.ok) {
return { ...r, ok: true, detail: `xfail (${entry.reason}): ${r.detail}` };
}
return {
...r, ok: false,
detail: `xfail entry unexpectedly passed — delete it from the allowlist ` +
`(was: ${entry.reason})`,
};
});
}
// ---------------------------------------------------------------------- main
async function main(): Promise<void> {
@@ -348,7 +412,8 @@ async function main(): Promise<void> {
if (udsPath) {
const conn = await connectUds(udsPath);
await conn.call('session.hello', { clientType: 'test', clientName: 'conformance', protocolVersion: '1.0.0' });
results.push(...(await replay(conn, fixtures, await setupBindings(conn), includeRequires)));
const { bindings, setup } = await setupBindings(conn);
results.push(...setup, ...(await replay(conn, fixtures, bindings, includeRequires)));
conn.close();
}
if (wsPort) {
@@ -362,7 +427,8 @@ async function main(): Promise<void> {
if (token === undefined) throw new Error('pairing failed: no token issued');
await conn.request('session.hello',
{ clientType: 'test', clientName: 'conformance', protocolVersion: '1.0.0', token }, 5000);
results.push(...(await replay(conn, fixtures, await setupBindings(conn), includeRequires)));
const { bindings, setup } = await setupBindings(conn);
results.push(...setup, ...(await replay(conn, fixtures, bindings, includeRequires)));
results.push(...(await privilegeChecks(conn)));
conn.close();
}
@@ -370,14 +436,18 @@ async function main(): Promise<void> {
process.stdout.write('no --uds or --ws-port given: ran static checks only\n');
}
const failed = results.filter((r) => !r.ok);
const xfailPath = arg('--xfail');
const finalResults = xfailPath ? applyXfail(results, loadXfail(xfailPath)) : results;
const failed = finalResults.filter((r) => !r.ok);
for (const r of failed) {
process.stdout.write(`FAIL [${r.transport}] ${r.fixture}\n ${r.detail}\n`);
}
const byTransport = new Map<string, number>();
for (const r of results) byTransport.set(r.transport, (byTransport.get(r.transport) ?? 0) + 1);
for (const r of finalResults) byTransport.set(r.transport, (byTransport.get(r.transport) ?? 0) + 1);
const summary = [...byTransport].map(([k, v]) => `${k}:${v}`).join(' ');
process.stdout.write(`\n${results.length - failed.length}/${results.length} checks passed (${summary})\n`);
process.stdout.write(
`\n${finalResults.length - failed.length}/${finalResults.length} checks passed (${summary})\n`);
process.exit(failed.length === 0 ? 0 : 1);
}