proto: stop conformance from downloading real files, ADR the null-clearing gap
1. download.add.json had startMode "now" against a real, large (~6 GB)
Ubuntu ISO with saveDir hardcoded to /home/sami/Downloads/Programs.
Against a real veloxd (tests/conformance/run.sh) that's a real
download into the real user's real home, every single run — it had
already happened twice. startMode -> "later" (exercises the add path,
hands nothing to the engine) and saveDir is dropped entirely (resolves
to saveTo.defaultDir instead, checked against allowedRoots the same
way). Documented the rule this fixture was breaking in
contracts/fixtures/README.md so it doesn't happen a third time.
Auditing the rest for the same shape (now real: capture.offer, D7)
found a second, subtler instance: capture.offer.take.json's "take"
admits a real, immediately-started task the same way download.add
does, and the "Programs" category's saveDir is a migration-seeded
builtin (~/Downloads/Programs) that no isolated test setup can
redirect -- so even after pointing the URL at example.org (RFC 2606),
a real ~6 GB sparse .veloxpart still landed in the real home on the
declared Content-Length alone. Shrunk to a plausible-but-small 5 MiB.
Also scoped to "transport": "uds" -- a real "take" persists an active
task, so replaying the same fixture again on the second live transport
against the same shared daemon was hitting capture.offer's own
dedupe-by-URL and failing on a missing taskId, not a bug.
download.add's other real-URL siblings (errors/*.invalid-path,
*.invalid-params, *.disk-full) all fail before admission or are
requires-gated; left alone.
2. ADR 0018: DAEMON can set a nullable field through download.update /
settings.set but never clear it back to null, because the generated
C++ parser collapses "absent" and "explicit null" to the same
std::nullopt for every optional field (contracts/codegen/gen_cpp.py,
on purpose, and correct for create-style params -- just wrong for
patch-style ones, which is the only place the schema documents
"explicit null clears"). Decision: an opt-in x-clearable schema
annotation makes just those fields std::optional<std::optional<T>> in
C++ (TS already round-trips this natively); not a blanket rule
(would retype response fields like TaskSummary.effectiveUrl that have
no clear-vs-absent distinction to make), not an explicit clear-list
field (would redesign a wire contract DAEMON already built against
just to route around a generator gap). Recorded, not implemented here
-- that's its own PROTO PR (schema annotations + gen_cpp.py + gen_ts.py
+ regeneration + a minor VERSION bump per ADR 0015), not bundled into
a fixture-safety pass. Left a pointer to the ADR at the generator
comment it concerns.
3. Re-verified every xfail entry against current deferrals.md rather
than trust the reasons already on file: D7/D8 (capture.offer/
getRules), D3d/e/f/g/h/i (rules, queue.reorder, schedule, limiter,
download.update/refreshUrl) and D9 (settings) have all closed since
the list was last pruned, so most of it was stale. Removed everything
that now cleanly passes; kept and re-reasoned everything that doesn't:
- errors/download.provideAuth.not-found.json stays, as asked: real
bug, on_download_provideAuth never checks the task exists.
- category.list.json (mimeTypes -- documented D3a gap), schedule.set.json
(nextRunAt -- documented D3f gap): unchanged in substance, reason
text was already accurate.
- download.probe/get/list/update.json, session.hello.json,
queue.start/reorder.json, category.remove.json: not bugs -- each
golden depicts a richer lifecycle/config state (a probed download,
real queue or category membership, media/grabber capabilities) than
this harness's fresh, never-started bound tasks and empty isolated
DB can produce.
- limiter.get.json: real fixture bug, not a daemon one -- applyToRunning
is a write-only instruction on limiter.set, on_limiter_get never
returns it; the golden shouldn't have had it either. Fixed the
fixture and tools/mockd's own limiter.get, which had the same field
hardcoded into its in-memory state independent of the fixture file.
- grabber.*/media.*: still genuinely stub (M4 territory).
Only remaining unexpected-pass surfaced while re-verifying
(errors/capture.offer.ignore.json, always "take" instead of "ignore")
traced to capture.minSizeBytes defaulting to 0 on a fresh daemon,
making its below-minimum-size scenario unreachable -- not a bug, so
raised the setting in run.sh's isolated seeding instead of xfailing it.
ctest -L conformance: green, 100% (2/2), ~87s.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01SFeUKLbdHizrJjLBeK7ffz
This commit is contained in:
Claude Sonnet 5
@@ -506,7 +506,12 @@ def emit_field_parse(f: Field, indent: str) -> list[str]:
|
||||
f'{i} const auto it = j.find("{f.name}");']
|
||||
if f.optional:
|
||||
# Absent and null mean the same thing: the field is not set. A client that omits
|
||||
# a nullable field and one that sends null are treated identically on purpose.
|
||||
# a nullable field and one that sends null are treated identically on purpose --
|
||||
# correct for create-style params, where there is no existing value to distinguish
|
||||
# "never set" from "explicitly cleared". Patch-style fields need the distinction
|
||||
# (download.update's patch: "an explicit null clears a nullable field") and get an
|
||||
# opt-in exception via x-clearable per ADR 0018 (not implemented yet: this is the
|
||||
# decision record, not the generator change).
|
||||
o.append(f"{i} if (it != j.end() && !it->is_null()) {{")
|
||||
o += emit_value_parse(f.type, "(*it)", "val", "fp", i + " ")
|
||||
o.append(f"{i} out.{m} = std::move(val);")
|
||||
|
||||
@@ -21,7 +21,7 @@ fixtures/
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"name": "download.add — start an ISO now, into the Programs category",
|
||||
"name": "download.add — add an ISO for later, into the Programs category",
|
||||
"description": "Why this case is worth pinning.",
|
||||
"transport": "uds", // optional: replay only on this transport
|
||||
"requires": "...", // optional: a condition a plain server cannot produce
|
||||
@@ -81,3 +81,27 @@ cases in `tests/integration/`.
|
||||
correct response is *no response*: past 750 ms the extension must abandon the offer and let
|
||||
Firefox download normally. A download manager that eats downloads when its daemon is down
|
||||
is worse than no download manager.
|
||||
|
||||
## No fixture may pair a real external URL with `startMode: "now"`
|
||||
|
||||
This suite replays every fixture against a real, live `veloxd` (`tests/conformance/run.sh`),
|
||||
not just `mockd`. `mockd` never actually fetches anything, so it hid this for a while: a
|
||||
fixture with `startMode: "now"` (or `"queue"` into a running queue — anything that gets
|
||||
admitted to the scheduler right away) and a real, resolvable URL makes a **real** daemon
|
||||
actually start downloading it, for real, onto whatever machine runs the suite. This
|
||||
happened — twice, with `download.add.json` pointed at a ~6 GB Ubuntu ISO, straight into the
|
||||
developer's real `~/Downloads`.
|
||||
|
||||
The fix in each case is one of:
|
||||
- `startMode: "later"` — exercises the add path (validation, category assignment, the
|
||||
event) without ever handing the task to the engine;
|
||||
- a URL under `example.org`/`example.com` (IANA-reserved for exactly this, RFC 2606) —
|
||||
resolvable enough to validate as a URL, never a real download source;
|
||||
- `requires`, if the fixture's entire point needs a real transfer to fail in a specific way
|
||||
(see `errors/download.add.disk-full.json`) — skipped by default, so it only ever runs
|
||||
where the condition has actually been arranged.
|
||||
|
||||
A real `saveDir` gets the same treatment for the same reason: an absolute path like
|
||||
`/home/sami/Downloads/...` only means anything on the machine that fixture was written on.
|
||||
Omit `saveDir` and let `saveTo.defaultDir` apply, or use a relative-feeling path under a
|
||||
root the runner controls.
|
||||
|
||||
@@ -1,22 +1,23 @@
|
||||
{
|
||||
"name": "capture.offer — attachment on a monitored type is taken",
|
||||
"description": "Golden fixture. tests/conformance replays this against the real daemon AND the TS client. If either side drifts, this goes red before the lanes ever integrate.",
|
||||
"description": "Golden fixture. tests/conformance replays this against the real daemon AND the TS client. If either side drifts, this goes red before the lanes ever integrate. url is example.org (RFC 2606), not a real download source: 'take' against a real veloxd (tests/conformance/run.sh) admits a real task and hands it to the engine for real, and no fixture may do that against a real external URL. contentLength is a plausible-but-small 5 MiB rather than a real ISO's size: the 'Programs' category's saveDir is a migration-seeded builtin (~/Downloads/Programs, daemon/src/store/migrations/0001_initial.sql), not something an isolated test run's settings can redirect, so 'take' always sparse-preallocates into that real path on whatever machine runs this suite -- keeping the declared size small keeps that footprint trivial instead of a real ISO's worth of disk. transport is uds only: a real 'take' persists an active task, so replaying this same fixture again on a second live transport against the same daemon would correctly dedupe against it (capture.offer dedupes by exact URL) and get 'ignore' instead -- an artifact of replaying one fixture against one shared daemon over two transports, not a behaviour to golden.",
|
||||
"transport": "uds",
|
||||
"request": {
|
||||
"jsonrpc": "2.0",
|
||||
"id": 42,
|
||||
"method": "capture.offer",
|
||||
"params": {
|
||||
"url": "https://releases.ubuntu.com/26.04/ubuntu-26.04-desktop-amd64.iso",
|
||||
"url": "https://example.org/dl/ubuntu-26.04-desktop-amd64.iso",
|
||||
"method": "GET",
|
||||
"tabUrl": "https://releases.ubuntu.com/26.04/",
|
||||
"tabUrl": "https://example.org/26.04/",
|
||||
"headers": {
|
||||
"User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:154.0) Gecko/20100101 Firefox/154.0",
|
||||
"Referer": "https://releases.ubuntu.com/26.04/",
|
||||
"Referer": "https://example.org/26.04/",
|
||||
"Accept": "*/*"
|
||||
},
|
||||
"cookies": [],
|
||||
"contentType": "application/octet-stream",
|
||||
"contentLength": 6228541440,
|
||||
"contentLength": 5242880,
|
||||
"contentDisposition": "attachment; filename=\"ubuntu-26.04-desktop-amd64.iso\"",
|
||||
"filename": "ubuntu-26.04-desktop-amd64.iso",
|
||||
"origin": "moz-extension://11111111-2222-3333-4444-555555555555"
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "download.add \u2014 start an ISO now, into the Programs category",
|
||||
"description": "The ordinary add path. saveDir is canonicalized and checked against the allowed roots before anything is written.",
|
||||
"name": "download.add — add an ISO for later, into the Programs category",
|
||||
"description": "The ordinary add path. saveDir is canonicalized and checked against the allowed roots before anything is written. startMode is 'later' deliberately: this suite replays against a real veloxd (tests/conformance/run.sh), and a real daemon given startMode 'now' would actually start fetching url for real. No fixture may pair a real external URL with startMode 'now' -- see contracts/fixtures/README.md.",
|
||||
"request": {
|
||||
"jsonrpc": "2.0",
|
||||
"id": 11,
|
||||
@@ -8,10 +8,9 @@
|
||||
"params": {
|
||||
"url": "https://releases.ubuntu.com/26.04/ubuntu-26.04-desktop-amd64.iso",
|
||||
"filename": "ubuntu-26.04-desktop-amd64.iso",
|
||||
"saveDir": "/home/sami/Downloads/Programs",
|
||||
"categoryId": "programs",
|
||||
"segments": 8,
|
||||
"startMode": "now"
|
||||
"startMode": "later"
|
||||
}
|
||||
},
|
||||
"response": {
|
||||
@@ -19,13 +18,13 @@
|
||||
"id": 11,
|
||||
"result": {
|
||||
"taskId": "$uuid",
|
||||
"state": "connecting",
|
||||
"state": "paused",
|
||||
"duplicate": null
|
||||
}
|
||||
},
|
||||
"assertions": [
|
||||
"the .veloxpart file is created sparse and preallocated at the final size",
|
||||
"saveDir resolves inside saveTo.allowedRoots, or the call fails -32011 having written nothing",
|
||||
"saveDir is omitted here on purpose: it resolves to saveTo.defaultDir, which is itself checked against saveTo.allowedRoots the same way an explicit saveDir would be -- see errors/download.add.invalid-path.json for the -32011 case",
|
||||
"startMode 'later' lands the task in 'paused' and never hands it to the engine, so nothing is fetched and no .veloxpart is created yet -- that only happens once the task is actually started (download.start.json, or startMode 'now'/'queue' against a source this suite controls)",
|
||||
"event.task.added is emitted to every subscriber before this reply is sent"
|
||||
]
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "limiter.get \u2014 the limiter is off",
|
||||
"description": "globalBps still carries the last configured value so the GUI can restore it when the user re-enables the limit.",
|
||||
"description": "globalBps still carries the last configured value so the GUI can restore it when the user re-enables the limit. applyToRunning is a write-only instruction on limiter.set (\"retune already-running transfers now\", not a persisted setting), so it never comes back from get.",
|
||||
"request": {
|
||||
"jsonrpc": "2.0",
|
||||
"id": 52,
|
||||
@@ -12,11 +12,11 @@
|
||||
"id": 52,
|
||||
"result": {
|
||||
"enabled": false,
|
||||
"globalBps": 2097152,
|
||||
"applyToRunning": false
|
||||
"globalBps": 2097152
|
||||
}
|
||||
},
|
||||
"assertions": [
|
||||
"enabled false means no throttling regardless of globalBps"
|
||||
"enabled false means no throttling regardless of globalBps",
|
||||
"applyToRunning is absent, not false: it's meaningless outside a limiter.set call"
|
||||
]
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user