Was 7/16 of tools/testserver/README.md's mode table covered by
engine_test.cpp. Adds the rest:
- engine_expiring_signed_url_recovers_via_refresh_url: an expired signed
URL 403s, the engine asks (paused, decision_calls >= 1) rather than
failing terminally, and DownloadHandle::refresh_url() with a freshly
signed URL completes it -- exercises both do_refresh_url() fixes and
the probe-level referrer retry's second-403 path from the previous
commit.
- engine_403_without_referer_retries_with_origin: no spec.referrer set,
the automatic single retry (previous commit) recovers with zero
decisions asked.
- engine_redirect_chain_follows_to_completion: 5 hops of a plain 302.
No core-side change needed -- documents that CURLOPT_FOLLOWLOCATION/
MAXREDIRS (already on, RequestOptions::follow_redirects) cover both
the probe's and every worker's own request, not just one of the two.
- engine_slow_loris_stall_timeout_fires: proves curl's stall detector
(CURLOPT_LOW_SPEED_LIMIT/_TIME, download_task.cpp's hardcoded 1024 B/s
for 30s) actually fires rather than hanging. Needed a real fix, not
just a test: every other test in this file relies on TestServer's
short 1s loris dribble to keep runtime down, but 1s of trickle
followed by full-speed streaming never accumulates curl's required 30
CONSECUTIVE seconds under the floor, so it would never actually abort
-- a test built on the default dribble would pass by the download
merely finishing a bit late, not by observing the stall timeout fire.
testserver_fixture.hpp's TestServer gained an explicit-loris-seconds
constructor (default ctor unchanged, still 1s) so this one test can
ask for a dribble (40s) that genuinely outlasts the threshold.
- engine_401_digest_then_provide_auth_completes: same shape as the
existing 401-basic test: http_client.cpp already asks libcurl for
CURLAUTH_ANY regardless of net::AuthScheme, so this needed no core
change -- it passed on the first run and is here to prove that's true
end-to-end, not just at the http_client unit level.
- engine_chunked_no_length_completes_single_segment: Transfer-Encoding:
chunked, no Content-Length anywhere (including HEAD). No core change
needed -- takes the same size-agnostic "unknown size, one plain-GET
segment" path as the existing no-range test.
- utf8/legacy-content-disposition: already covered end-to-end by
probe_reads_utf8_content_disposition and
probe_reads_legacy_content_disposition in probe_test.cpp (probe-level,
as these modes only affect the initial request) -- verified passing,
no new test needed.
All 20 engine_test.cpp cases and all 10 probe_test.cpp cases pass. Every
testserver.py spawned while writing and running this was reaped by
TestServer's destructor; verified no stragglers with `ps aux` after each
run.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Q3QrF7rCt21bkAjt9BCDFQ
docs/04-engine-design.md §7's failure policy table has said "403 after
redirect: retry once with the original referrer; many CDNs require it"
since it was written, and Error::forbidden's own enum comment says the
same -- but grepping download_task.cpp and http_client.cpp for 403 turned
up nothing. It was never built.
Implemented at both points a 403 can surface:
- The probe (net::Prober, a separate request path from segment workers):
on_probe_result() now retries once via restart_probe(false), with
effective_referrer set to the download URL's own origin (origin_of(),
via net::split_url()), when the failure is Error::forbidden and this is
the first retry. A second 403 asks rather than fails outright --
auto_pause_locked(..., false, true), the same "ask, don't just fail"
path 416/etag-mismatch already use -- specifically so DownloadHandle::
refresh_url() stays usable afterward (its own contract requires a
non-terminal task); this is what makes the expiring-signed-url mode's
README-documented refresh_url() recovery actually reachable.
- Each segment worker (SegWorker::forbidden, set in seg_head() on a 403
HEAD): the same one-shot referrer retry via retry_worker(), landing on
auto_pause_locked() on a second 403 for the same reason.
Both paths route the retry's Referer through a new effective_referrer
field rather than spec.referrer directly, since the origin-retry must not
overwrite what the caller actually asked for -- start_worker_locked() and
restart_probe() were switched to send effective_referrer instead.
do_refresh_url() had two latent bugs surfaced by actually exercising the
expiring-signed-url recovery path end-to-end:
1. It unconditionally proceeded to resume even when the refresh probe
itself failed -- a bad refresh URL would silently un-pause a task with
nothing behind it. Now returns (stays paused) on !r.has_value().
2. It only handled "already probed once, just refreshing a few fields" --
for a task whose first-ever probe never succeeded (every hostile mode
this commit adds a test for that pauses at the initial probe, not
mid-download), s->registered was never true, so the existing
`if (s->registered) set_want()` never fired and nothing happened. Now
detects !s->have_probe and calls finish_probe_locked() directly, the
actual first-time registration/segmenter-construction path.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Q3QrF7rCt21bkAjt9BCDFQ
main.cpp's single-instance lock is now keyed to the runtime directory
instead of the euid, so an isolated XDG_RUNTIME_DIR/XDG_DATA_HOME/
XDG_CONFIG_HOME/HOME gets its own veloxd alongside anyone else's.
tests/live/real-veloxd.test.ts spawns one (VELOX_PAIR_AUTO=1 standing
in for the GUI's Allow click) plus tools/testserver/testserver.py, and
exercises the real WebSocketTransport end to end: session.hello,
pairing, the token surviving a reconnect, a wrong token rejected and
then rate-limiting the next pairing attempt, download.add reaching a
real running task, the real capture.offer path (rules table + category
folder + dedupe) taking a monitored download and ignoring its own
duplicate, and fail-open proven by SIGKILLing the daemon mid-offer —
the hook still resolves to {} inside its 750ms budget. A last case
proves fail-open at the transport layer too: a call against a closed
socket rejects instead of hanging.
Guarded behind so it skips itself (with a clear message)
when no daemon binary is around — npm test and CI are unaffected;
run it with VELOXD_BIN=/path/to/veloxd npx vitest run tests/live.
Real-daemon testing found one actual bug, fixed here: background/
index.ts never called session.subscribe, so event.task.progress and
friends never reached this connection at all — FakeDaemon's tests
never caught it because FakeDaemon broadcasts regardless of
subscription state. Now subscribed to the full event set on every
connect (first connect and every reconnect), which is what the popup's
live-progress path actually depends on against a real daemon.
Per instructions: ctest -L conformance was not run (pending PROTO's
fixture fix).
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Ed8KEmAW48v4YHdxLtqsMB
rules.list/rules.upsert: new store/rules.{hpp,cpp} (list in priority order; apply()
upserts+removes atomically, generating an id when absent, per the schema's own "never
leaves the table in a half-valid state"). Found and fixed along the way: migration
0001's rules table had no column for Rule.name at all — every rules.list/.upsert call
failed outright ("no such column: name"), unit tests included, since :memory: migrates
through the same path. Migration 0004 adds it.
queue.reorder: new store::Queues::reorder — taskIds must be an exact permutation of
the queue's current membership (compared as sorted sets) or nothing is written and
-32602 names the queue; a valid permutation rewrites every member's queue_position in
one transaction.
schedule.get/schedule.set: a thin wrapper over the queues.schedule column (already
read since D3b, never independently settable). nextRunAt is deliberately left unset —
computing it needs the same local-time, DST-aware window logic
sched/schedule_window.hpp's window_open() only has half of; called out rather than
approximated, and the field is optional.
limiter.get/limiter.set: backed by the same downloads.speedLimitEnabled/
downloads.speedLimitBps settings keys D9 already wired — one bag of truth, not two.
The new part is reaching the engine: EnginePort/TaskActionPort gain
set_global_speed_limit(bps) (0 = unlimited, TokenBucket's own convention), wired to
Engine::rate_limiter().set_global_limit(). Pushed live on every limiter.set *and* on
Scheduler::reload_config() so a limit from a previous run isn't silently unlimited
again after a restart. applyToRunning is accepted but has no lever to pull
differently — a single shared global bucket has no "next task only" variant. Also
found, not chased further: the schema's "globalBps:0 with enabled:true means 'stop
everything'" is the opposite of what TokenBucket does with rate_bps==0 (unlimited) —
a real discrepancy, but the schema says the GUI must not offer that combination.
download.update: "moving saveDir or filename moves the file on disk in the same
operation" — resolved and root-checked like download.add's destination, then the
.veloxpart/.veloxpart.meta pair (or the finished file, if complete) is moved via
rename, falling back to copy+remove across filesystems, only when the resolved
location actually differs. categoryId/queueId(appended to the new queue's run
order)/description/segments/bufferBytes/checksum apply through new
store::Tasks::apply_update.
download.refreshUrl: same async server-layer special-case as download.probe (a real
network round trip, same 30s deadline). Re-probes, flags contentChanged only when
size or validator are both known and actually differ, persists the new URL and probe
result, and swaps the URL on a live engine handle via a newly-widened
EnginePort::refresh_url (now takes headers too, matching DownloadHandle's real
signature — the seam had silently dropped them).
Found and documented, not fixed: the generated parser collapses "field absent" and
"field explicitly null" to the same nullopt for every optional<T> patch field
(DownloadUpdateParamsPatch, Settings) — both schemas document "an explicit null
clears the field" but neither handler can act on it because the wire distinction is
already gone by the time either sees the parsed struct. A generator-level gap
(PROTO's), not something to hand-route around locally.
Verified against real veloxd + tools/testserver: rules create/list, limiter.set
takes effect and reads back, schedule.set/get round-trips, queue.reorder against real
membership (and rejects a non-permutation), download.update renames+recategorizes a
task, download.refreshUrl swaps a paused task's URL and reports contentChanged
correctly. Full ctest: 55/55 (excluding the pre-existing, unrelated conformance
failure noted two commits back).
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GRDjHGgpYmMoPE2UFbe7pP
Two run.sh fixes plus the xfail prune, all requested together:
1. VELOX_PAIR_AUTO=1 for the isolated veloxd. Pairing is the D1 dev stub
(EnvAutoApprover) and denies without it, so session.pair never issued
a token and the WS half of the veloxd step could never even connect.
2. WS_PORT was hardcoded to 52080 with no free-port search, so one leaked
mockd made every future run fail EADDRINUSE. free_port() binds :0 and
asks the kernel instead. The EXIT trap's stop() used `pkill -P "$pid"`,
which only reaps direct children — tsx's actual listener is often a
grandchild, which that missed and left holding the port. Every server
(mockd, slow mockd, veloxd) now launches under `setsid`, making it the
leader of its own process group, so stop() does `kill -TERM -"$pid"`
(a process-group kill) and reaches everything it spawned in one shot.
3. Pruned the xfail list now that D2, D4b and most of D3 have landed.
Pruning surfaced two more bugs than expected, both in the test harness
itself, not veloxd — worth recording since they were indistinguishable
from real daemon hangs until isolated:
- errors/session.hello.version-mismatch.json documents that the *server*
closes the connection after replying (correct, intended behavior). The
harness replays every fixture on one shared connection per transport,
so once this fixture ran, every later UDS fixture sent into the dead
socket and just sat there until its own timeout — including ones still
on the xfail list, which applyXfail waved through as "expected -32603"
regardless of the real reason. Fixed with a `closesConnection` fixture
flag: replay() reconnects (fresh session.hello) right after such a
fixture instead of leaving the rest of the run to time out one by one.
This is what was actually behind queue.*/session.*/download.remove
appearing to hang — none of them do; verified individually and via a
raw probe script before finding the real cause.
- category.remove.json (deletes the "firmware" category) sorted before
category.upsert.json (creates it) alphabetically, so it was failing
-32602 "no such category" against a fresh DB — never a daemon bug.
Added it to DESTRUCTIVE so it now replays after every other fixture.
Also fixed while verifying "confirm each really passes": download.addBatch.json's
`defaults.categoryId` was "compressed", a category nothing ever creates —
real veloxd correctly enforces the FK on tasks.category_id, so all three
batch items failed instead of the two expected. Changed to "programs" (a
migration-seeded builtin).
Of the 15 fixtures named for pruning, 10 turned out to cleanly pass and
are gone from the list entirely: download.pause/resume/start/cancel,
download.remove, download.addBatch, queue.upsert/stop, download.probe's
success path (D2, including errors/download.probe.probe-failed.json),
and category.upsert. Two do NOT cleanly pass and are kept, with reasons
rewritten to match what's actually happening now instead of the stale D3
text: download.probe.json (see below) and errors/download.provideAuth.not-found.json,
a real bug — on_download_provideAuth never checks the task exists, so an
unknown taskId gets a normal `{ok:false}` result instead of -32010.
Five more fixtures newly needed xfail entries to reach green, none of
them stubs:
- category.list.json — documented gap (deferrals.md's D3a note): the
categories table has no mimeTypes/sortOrder columns.
- download.probe.json, download.get.json, download.list.json,
session.hello.json — not bugs. Each golden depicts a richer lifecycle
state (a probed/in-progress download, a daemon with media/grabber/
Secret Service implemented) than this harness's bound tasks, which are
always fresh and never started, can produce. Optional/omit-if-absent
fields (effectiveUrl, requiresAuth, capabilities) are correctly absent;
the mismatch is against the golden's illustrative values, not the
contract.
- queue.start.json, category.remove.json — same class: startedTaskIds /
reassignedTaskIds are correctly empty because this run's queue/category
have no real membership.
`ctest -L conformance` is green: 100% (2/2), 81.7s (down from ~240s now
that pairing and the port/reconnect fixes remove the retries and the
5-10s timeouts the connection-death bug was producing).
One thing NOT fixed here, flagged for a follow-up decision rather than
touched mid-task: download.add.json's fixture is `startMode: "now"`
against a real, large (~6GB) Ubuntu ISO on the real internet, with
saveDir hardcoded to /home/sami/Downloads/Programs. Every run against a
real veloxd writes a real multi-GB file into that path — confirmed by
running this repeatedly during verification. Isolating the daemon's XDG
dirs doesn't isolate this. Worth its own change (startMode: "later"
would still exercise the add path without the transfer) but out of scope
for a fixture I wasn't asked to touch beyond what blocked this task.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01SFeUKLbdHizrJjLBeK7ffz
capture.offer applies capture.enabled/excludedHosts/monitoredExtensions/
monitoredMimeTypes/minSizeBytes from settings, then the rules table (new
store::Rules + CORE's vdm::rules::match_rules/glob_match — DAEMON only converts its
own stored proto::Rule JSON into CORE's plain vocabulary, per that header's own
layering note), resolves the category folder (a rule's explicit categoryId/saveDir,
else store::Categories::guess_by_extension — the same extension guess
download.probe's suggestedCategoryId already used, now shared instead of
duplicated), dedupes against active (non-terminal) tasks by exact URL, and on `take`
calls add_one() — the same path download.add itself uses, so a captured download is
a real, admitted, persisted task, not a special case.
The 750ms deadline (CLAUDE.md §4 / AGENT-DAEMON.md build step 6) is checked
cooperatively between every step via a new rpc::CaptureDataSource seam: the real
implementation wraps store::Settings/Categories/Rules/Tasks; a test fake can jump its
own injected clock forward to simulate "the store was slow just now" with zero real
sleep. This catches the realistic failure mode (several slow steps adding up) though
it cannot preempt a single pathologically stuck call mid-flight — a true preemptive
guarantee would need the same async/background-thread treatment as download.probe,
which isn't safe to do against the same sqlite3 connection (opened SQLITE_OPEN_NOMUTEX,
explicitly not for concurrent use) without a second connection; left as a known,
documented limit of this pass rather than adding that plumbing speculatively.
capture.getRules returns the same settings-backed fields capture.offer itself reads,
so the two can never drift. rulesVersion is a placeholder constant (1) — no persisted
revision counter exists yet, and the extension already re-fetches on
event.settings.changed regardless.
New store/rules.{hpp,cpp}: rules table CRUD (list, and an atomic upsert+remove for
rules.upsert later). store::Categories::guess_by_extension replaces a duplicate copy
that used to live in sched/scheduler.cpp. store::Tasks::has_active_duplicate for the
dedupe check.
Verified against real veloxd + tools/testserver: a monitored-type offer answers in
~5ms and actually creates + downloads the task, correctly categorized; an
unmonitored type, an excluded host, a rule-vetoed host, and a second offer for a
still-active URL all answer ignore with the right reason; a bad category save dir
surfaces its real -32011. New capture_offer_test covers all of the above plus the
deadline itself (two cases, one per "slow" checkpoint), asserting real wall-clock
time barely moves even though the fake clock jumped 2 simulated seconds. Full ctest:
54/54 (excluding the pre-existing, unrelated conformance failure noted in the
previous commit).
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GRDjHGgpYmMoPE2UFbe7pP
Rebased lane/daemon onto main first — fast-forward, no conflicts (lane/daemon was
already fully merged; main has since taken lane/gui, lane/pkg-qa, lane/proto).
1. capture.offer and capture.getRules were stubs, buried inside D3's generic stub list
with no flag for how load-bearing capture.offer is: it's a CLAUDE.md §4
non-negotiable (750ms deadline, fails open) and AGENT-DAEMON.md build step 6, and
against the real daemon it 500s to -32603 every time — the extension captures
nothing, not "falls through to Firefox on a slow path." Split into their own D7/D8
rows so they stop being invisible.
2. settings.get/settings.set (D9, closed). Four pointer-to-member tables in
dispatcher.cpp (bool / ranged int / plain string / string array), five enum-typed
keys handled individually since parse_XXX already validates those — covers all 43
SettingKeys without ~40 repetitive hand blocks. store::Settings::kDefaults grew from
14 entries to 43 (the schema itself carries no "default" keyword anywhere, so these
are hand-chosen — conservative for the ones ADR 0012 doesn't speak to;
capture.monitoredExtensions defaults to the union of every builtin category's
extensions rather than an arbitrary list of its own).
settings.set validates every field before writing any of them: numeric min/max
(hand-checked — the generated parser only checks JSON type, not schema
constraints) and saveTo.* paths via fs::resolve_target/canonicalize_root (-32011) —
allowedRoots entries checked as roots in their own right, defaultDir/tempDir checked
as paths resolving inside the (possibly just-updated, same call) root list. Reports
exactly the keys whose effective value actually changed, publishes
event.settings.changed with that list, and calls the scheduler's reload_config()
when a connection.* key took effect — live, not on next restart.
Verified against real veloxd: all 43 keys round-trip with defaults, a keys subset
filters, an out-of-range value rejects the whole call, a bad saveTo.defaultDir is
-32011, allowedRoots+defaultDir set together cross-validate against the new roots,
event.settings.changed fires over a live subscription. New dispatcher_settings_test
covers the same ground without a socket. Full ctest: 53/53 (excluding a pre-existing,
unrelated conformance failure — see below).
Also found, not fixed (not this lane): tests/conformance/run.sh's live-veloxd leg fails
"pairing failed: no token issued" reproducibly, isolated, on main before any of this
session's changes — it launches the isolated veloxd without VELOX_PAIR_AUTO=1, so
EnvAutoApprover denies every session.pair and the WS leg's session.hello never gets a
token. tests/conformance/ is PROTO/QA-owned; flagging rather than editing across the
lane boundary.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GRDjHGgpYmMoPE2UFbe7pP
Per-task engine state, downloaded/effective_segments, every segment's own
state, and the engine's SegmentBudget snapshot -- printed once, when a task
times out, instead of needing to re-run the bench under a debugger or add
throwaway instrumentation to find out why. This is what actually diagnosed
the SegmentBudget over-admission bug fixed in the previous commit: the dump
showed budget.active pinned at max_active_segments while multiple tasks sat
starved, which is what pointed straight at confirm_slot()'s missing
engine-wide check rather than a per-task target bug.
Also updates the vdm_bench_load20 ctest registration's comment: the
straggler under --preset tsan that motivated running it at reduced
concurrency (docs/adr/0016's postscript) is now suspected to have been the
same SegmentBudget bug (docs/adr/0017), not the TSan-timing artifact first
guessed -- not reverified at the DoD's full shape under TSan in this change,
so the reduced-concurrency registration stays for now.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Q3QrF7rCt21bkAjt9BCDFQ
Root cause of the M7 RSS gap (core/docs/m7-baseline.md: ~70 MB measured
against a 60 MB target): SegmentBudget::confirm_slot() only checked a task's
own held count against its own target -- never the engine-wide active_ sum.
reallocate_locked()'s two-pass fairness allocation does bound
sum(target) <= max_active_ at the moment it computes a plan, but that bound
says nothing about sum(held): a task can be legitimately holding more than
its own just-lowered target for a while (yield is deferred to a segment
boundary, never mid-segment -- ADR 0011 A1), and another task's target can
correctly rise to claim that capacity before the first task has physically
released it. Both confirm_slot() calls could then succeed against their own,
individually-correct targets while sum(held) exceeded max_active_ --
tools/bench heap-profile caught this directly: budget.active reading 56-86
against a total of 32.
confirm_slot() now also checks active_ < max_active_, unconditionally, as a
backstop that doesn't depend on any task's target bookkeeping being in sync
with what every other task holds. That creates a liveness question the
original design never answered: a task denied only by this new check has a
target that's already correct, so it never changes again and
reallocate_locked()'s plain "fire a callback when a task's target changes"
mechanism never revisits it. Task gained a waiting_for_slot flag, set on
exactly this denial; release_slot()/deregister_task() (the only two places
that free real capacity) now hand a freed slot directly to the
highest-priority waiting task via wake_one_waiter_locked(), if
reallocate_locked()'s own plan didn't already produce a callback for anyone.
download_task.cpp's fill_slots_locked() needed a matching fix: a woken
task's stalled segments (SegState::stalled -- backed off mid-retry, its own
release_slot() already called) have no live worker and never surface
through Segmenter::assign_slot(), which only hands out unassigned or fresh
ranges. fill_slots_locked() now restarts any stalled segment with no live
worker directly (bounded by slot_target, same as its assign_slot() loop)
before looking for new work; a segment it doesn't get to keeps its own
scheduled retry_worker() timer as a second chance.
Also fixes a real TSan-caught data race this work surfaced: SegWorker::
speed_bps was written only by its own segment's curl callback and, before
Progress.speed_bps's polled-path fix, only ever read from that same thread
-- safe without synchronization. snapshot_progress() reading it from
whatever thread calls DownloadHandle::progress() broke that invariant
(workers_mu's shared_lock protects the workers map's structure, not an
individual SegWorker's fields). Now std::atomic<double> with relaxed
ordering -- an informational EMA, nothing synchronizes real state on it --
rather than adding a lock to the write side.
core/tests/segment/budget_test.cpp adds two tests reproducing the actual
gap (budget_active_never_exceeds_max_active_segments_under_concurrent_load,
budget_wait_list_wakes_a_task_whose_target_never_changed) plus a sanity
baseline (budget_release_wakes_a_denied_waiter), and introduces AsyncFakeTask
+ TestTimer for the one existing test that drives the budget from multiple
concurrent threads -- mirroring production's real dispatch (register_task()'s
on_target lambda posts through host.schedule(), download_task.cpp, never a
synchronous call) rather than adding reentrancy-guarding machinery to
SegmentBudget itself to compensate for a synchronous test double being
unlike production. See docs/adr/0017 for the full writeup, including what an
earlier version of this fix got wrong chasing a same-thread reentrancy
hazard that doesn't actually exist in production.
core/docs/m7-baseline.md updated: the RSS number now clears the DoD line
(45.41 MiB via heap-profile), root-caused rather than just re-measured.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Q3QrF7rCt21bkAjt9BCDFQ
For the M7 RSS gap (core/docs/m7-baseline.md: ~70 MB measured vs ADR 0012's
~45-50 MB estimate) -- "find where before tuning anything". No root here to
install massif or heaptrack, so this is a small in-process equivalent:
operator new/delete (already overridden for alloc-check's counter) now also
track, per call site (one return address via __builtin_return_address(0),
cheap enough to run for a whole scenario), live (not-yet-freed) bytes. Runs
the same 20-task concurrent scenario as `load`, waits for peak RSS to stop
growing, then prints the top sites by live bytes -- resolved via one batched
addr2line invocation against /proc/self/exe (dladdr alone only resolves
dynamic-symbol-table entries, which misses most of this codebase's
internal-linkage call sites), with dladdr as a per-site fallback.
Also adds the nothrow operator new/delete overloads alongside the existing
plain ones: without them, anything that allocates via the nothrow form (e.g.
std::stable_sort's std::get_temporary_buffer, hit once while investigating
the RSS gap) falls through to ASan's own default nothrow new while still
being freed through this file's plain delete override -- an
alloc-dealloc-mismatch ASan correctly flags as a real ABI-level bug in an
allocator override that claims to intercept "everything".
Using this tool, sum(effective_segments) across all 20 tasks vs.
Engine::segment_budget().budget() (now cross-checked and printed together)
showed the budget hands out well more than max_active_segments -- the actual
root cause, in core/src/segment/budget.cpp, not covered by this commit.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Q3QrF7rCt21bkAjt9BCDFQ
DAEMON reported Progress.speed_bps reading 0 for the whole life of a live
throttled download while downloaded bytes visibly advanced. DAEMON reads
progress by polling DownloadHandle::progress() (engine_port_core.hpp), not
the on_progress push callback.
DownloadTaskState::snapshot_progress() -- the body behind progress() -- never
set speed_bps, per-segment speed_bps, or eta_seconds at all; only the
event-driven emit_progress_if_due() (which drives the on_progress callback)
computed them, from the same live SegWorker::speed_bps EMA seg_data()
maintains. snapshot_progress() now reads that same per-worker speed while
building its segment list, so a segment with no live worker (idle, paused,
complete, failed) correctly reports 0 and a segment with an active transfer
reports its real EMA, matching emit_progress_if_due()'s math including the
eta_seconds derivation.
engine_polled_progress_reports_nonzero_speed reproduces the bug (fails
without the fix, confirmed) by polling .progress() -- the same path DAEMON
uses -- during a throttled download and asserting speed_bps > 0 once real
progress has accumulated.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Q3QrF7rCt21bkAjt9BCDFQ
libdbus-1-dev / libsystemd-dev have no headers installed in this build environment
(only the runtime .so's — apt-cache policy confirms libdbus-1-dev is available but not
installed). A real org.freedesktop.Notifications-backed PairingApprover needs one of
those linked into veloxd, which is a new build dependency for daemon/CMakeLists.txt
and, since packaging manifests would need to know about it too, a decision to surface
rather than reach for silently mid-session.
PairingApprover::approve() is also still synchronous by shape — its own doc comment
already says the real approver "will run async and is not this shape." The async
pattern this session built for download.probe (rpc::TaskActionPort + the server-layer
deferred-reply special-case in uds_server.cpp/ws_server.cpp) is the right shape to
reuse once there's a real implementation to justify reshaping the interface; doing
that with nothing behind it yet would just be churn.
Left EnvAutoApprover in place rather than hand-roll a D-Bus wire client to route
around the missing headers — a broken pairing approver is a worse outcome than an
honest, already-documented stub. Findings recorded in deferrals.md for whoever picks
this up once the dependency is available and approved.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GRDjHGgpYmMoPE2UFbe7pP
Closes a bounded, high-value slice of the remaining D3 stubs. settings.*/rules.*/
limiter.*/schedule.*/grabber.*/media.*/capture.*/queue.reorder/download.refreshUrl/
download.update stay deferred — reasons noted individually in deferrals.md (settings.*
specifically: a real, large field<->key<->JSON-type mapping table across ~43 fields /
~50 SettingKeys, not started rather than rushed).
category.upsert/remove: store/categories.hpp gains get/upsert/remove. upsert generates
an id when absent and always ignores the payload's `builtin` (preserved from the
existing row on replace, false on create); saveDir goes through the same
fs::resolve_target canonicalize-and-root-check as download.add. remove refuses a
builtin at both layers (dispatcher's -32602 pre-check; the store's own
"DELETE ... AND builtin = 0" as defense in depth) and reassigns member tasks to
reassignTo (default "general") inside one transaction before deleting the row.
queue.upsert: store/queues.hpp gains get/upsert (set_state already existed from D4b).
Same create-generates-id pattern; a create always starts 'stopped', a replace keeps
the queue's current run state (upsert edits config, not run state — that's
queue.start/stop). Also fixed in passing: on_complete has been a real column since
migration 0001 but Queues::list/get never projected it onto Queue.onComplete.
download.remove: cancels with discard_partial=true (always drops the .veloxpart pair —
unlike download.cancel, which keeps them, the row is gone either way), deletes the
finished file only when deleteFile is true and the task was complete (best-effort),
deletes the row (segments cascade via the FK), and publishes event.task.removed
(closing the last open note under D5).
download.addBatch: on_download_add's body is now a shared add_one(), called once per
item after merging each item's unset fields against params.defaults.
download.provideAuth: forwards to EnginePort::provide_auth via a new
TaskActionPort::provide_auth. `remember`/persisting to the Secret Service is accepted
but not acted on — nothing in this build talks to libsecret yet.
Verified against real veloxd + tools/testserver: category create/replace/
remove-with-reassignment, queue create/replace-keeps-run-state, a batch add sharing
defaults.saveDir, and download.remove with deleteFile actually deleting the file and
the task then 404ing download.get with -32010. Full ctest: 39/39.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GRDjHGgpYmMoPE2UFbe7pP
download.probe was a stub (-32603). It needs an HTTP round trip on the engine's probe
pool (up to the schema's 30s x-deadlineMs), which cannot fit VeloxDispatcher's
synchronous on_download_probe -> HandlerResult<T> return without blocking the RPC
loop for the duration — a hard no per CLAUDE.md ("never block the RPC loop") and
AGENT-DAEMON.md build step 1.
uds_server.cpp and ws_server.cpp special-case "download.probe" before the generic
dispatch(), exactly the way they already special-case session.hello/session.subscribe:
parse the params, call the port, and queue the reply whenever the callback fires
(dropped silently if the connection is gone by then).
rpc::TaskActionPort gains probe_now(DownloadProbeParams, callback) — kept in proto/std
terms, no vdm::net::* in the signature, so veloxd_rpc never needs core/include's vdm
headers just to declare this. sched::Scheduler::probe_now is the implementation:
builds a vdm::net::ProbeRequest, runs it on the engine's probe pool, marshals the
engine-thread callback back onto the loop (deps_.post_to_loop, same as every other
engine callback here), maps a probe failure to -32013 ProbeFailed (data.httpStatus set
when there was an HTTP response), and fills suggestedCategoryId/suggestedSaveDir with a
plain extension match against the categories table — not the real rules engine, which
is still D3; noted in a comment.
Verified against real veloxd + tools/testserver, not just unit tests: a real probe
answers in ~5ms with size/resumable/etag/redirect chain; a bad host maps to -32013;
and — the actual point of the async design — a connection running a 10s slow-loris
probe does not block a second connection's download.list, which answers in ~1ms while
the probe is still outstanding.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GRDjHGgpYmMoPE2UFbe7pP
download.pause/resume/start/cancel and queue.start/stop were stubs; now they call into
the scheduler and take effect immediately, not on the next 1s tick — pausing, resuming
or cancelling a live transfer can't wait, and per ADR 0013 §3 the governor never
touches a user-owned pause on its own.
New rpc::TaskActionPort interface (owned by rpc/, implemented by sched::Scheduler) is
what dispatcher.hpp depends on instead of sched/scheduler.hpp directly. Needed because
veloxd_sched already links veloxd_rpc (for EventHub); dispatcher.hpp pulling in
sched/scheduler.hpp directly would make it a real circular library dependency, breaking
anything that links veloxd_rpc alone (cli's tests, as it turned out — hit and fixed
during this change).
Scheduler::user_pause/user_resume/user_start/user_cancel + pause_queue follow tick()'s
existing to_pause pattern: call the engine (async, no synchronous effect) and transition
the store eagerly so download.get/list are correct the instant the RPC call returns.
Fixed a real bug surfaced while building this: transition() always overwrote
pause_reason to NULL when the engine's own delayed pause-ack callback (on_state to
paused, no error) arrived after whoever actually initiated the pause had already
written the real reason — now it preserves the stored reason when the callback supplies
none, instead of clobbering it. Covered by a regression check in sched_scheduler_test.
store/queues gets get() and set_state() (was list()-only) for queue.start/stop.
Verified against real veloxd + tools/testserver, not just unit tests: pausing a live
single-segment throttled transfer freezes downloadedBytes, resume continues it from
that point, cancel stops it; a bad taskId comes back in BulkTaskResult.failed with
-32010, not a crash; queue.stop(pauseRunning:true) pauses the queue's running task
immediately and queue.start resumes admission.
Known gap: download.start's contract "a task in 'queued' jumps its queue" (priority
bump) is not implemented — admission is still plain FIFO by created_at. Noted in
deferrals.md.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GRDjHGgpYmMoPE2UFbe7pP
Continues the build order past the Add-URL/File-Info/Progress dialog flow.
- OptionsDialog: General/Save To/Connection/Downloads/Proxy/Sounds tabs, every
control bound to a real settings.* key (contracts/schema/types/Settings.
schema.json). The spec's File Types and Site Logins tabs have no settings.*
backing (categories go through category.upsert, credentials through the
Secret Service) so they don't exist here — a tab either binds to a real key
or isn't shipped. diffChanged() sends only what actually changed, matching
settings.set's "changed[] names exactly what took effect" contract.
- SchedulerDialog: per-queue schedule (schedule.get/set) plus maxConcurrent/
onComplete (queue.upsert), Start Now/Stop. Queue.schema.json already carries
the schedule so queue.list alone seeds the window.
- SpeedLimiterDialog: the live global limiter (limiter.get/set) — a different
thing from Options' downloads.speedLimit* default. buildParams() enforces
the schema's "0 with enabled true must not be offered".
- BatchDialog: clipboard-blob and {start..end}-wildcard tabs sharing one
category/queue/start-mode footer into download.addBatch.
- GrabberWizard: 4-step QWizard (project label -> start URL/depth/filters ->
file-type filter -> review), grabber.start feeding a poll+event.grabber.
progress-driven review page, Finish = grabber.harvest for the checked files.
- TrayIcon: active-count tooltip, Show/Add URL/Pause All/Resume All/Speed
Limiter submenu/Quit. Quit only closes the GUI — there is no RPC to stop
veloxd itself, filed as a new gap in daemon-requests-m1.md. MainWindow now
also hides to tray instead of closing when general.minimizeToTray is set.
Every dialog's non-widget logic (diffChanged, buildSchedule, buildParams,
parseUrlBlob/expandWildcard/buildAddBatchParams, buildFileTypes/
buildStartParams/buildHarvestParams) is a static pure function with its own
test, same shape as FileInfoDialog::buildSpec from the previous round.
Verified end-to-end against a running mockd under ASan+UBSan: all five
surfaces render real data (settings.get values, queue.list's two seeded
queues, limiter.get, a live grabber.start/status crawl returning 3 files) with
no sanitizer reports. gui-check (non-ASan) and dev (ASan+UBSan) presets both
build the whole repo clean; all gui-labeled ctest targets pass.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01NSCdCWFXBTSBBK3MzWtJiC
Protocol.hpp gains method-name constants for every RPC the next batch of dialogs
needs (settings.*, limiter.*, schedule.*, queue.*, download.addBatch, grabber.*)
and RpcClient re-broadcasts the two events nothing consumed yet:
event.settings.changed and event.grabber.progress. No behaviour change on its
own — OptionsDialog, SchedulerDialog and GrabberWizard are what actually call
these, landing next.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01NSCdCWFXBTSBBK3MzWtJiC
Adds tests/integration/nightly_run.py, wired as the nightly-integration job in
ci.yml (schedule + workflow_dispatch). Real veloxd + tools/testserver, 50
concurrent downloads mixing flaky-reset/throttled/no-range/plain, asserting:
every completed file's SHA-256 against testserver's own /sha256/ route (never
trusting veloxd's own success claim), every task reaching a terminal state
inside the timeout, and veloxd's own open-FD count settling back to baseline.
veloxd runs isolated (XDG_RUNTIME_DIR/XDG_DATA_HOME/XDG_CONFIG_HOME under a
fresh mkdtemp — not the session scratch dir, whose path overflows AF_UNIX's
sun_path). saveTo.allowedRoots is seeded directly into velox.db after a
migrations-only warm-up start, since settings.set returns -32603 'not
implemented in this build' on the veloxd this job builds (verified live).
Every task gets its own filename override on download.add: tasks sharing
(mode, size) share a URL, and without distinct filenames they raced each
other's rename on the first real run (48/50 'passed' with io_errors and
checksum mismatches on the collided tasks) before this fix.
Every assertion was forced red once on purpose and the transcript recorded in
tests/integration/README.md, per this repo's history of green checks that
didn't look where the bug was.
Every spawned child (veloxd, testserver.py) gets PR_SET_PDEATHSIG plus its own
process group, so a hard-killed harness can't strand a daemon the way a prior
run did (4h40m under systemd --user, because SIGKILL never reaches a
finally: block). Proven by kill -9'ing a running harness mid-download and
confirming both children exit with it.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01RBPR7iM3YPyxrjWsVtZDPJ
Two bugs blocking PROTO's live-veloxd conformance check.
1. tasks.start_mode's CHECK was ('auto','now','queue','manual') — not the
contract's StartMode enum (['now','later','queue']) at all. 'later', a real
documented value (the File Info dialog's Download Later button), hit the CHECK
on every insert and surfaced as an unhandled -32603; 'auto'/'manual' were
never contract values to begin with.
Migration 0003 rebuilds tasks (SQLite can't ALTER a CHECK) with the contract's
values, remapping existing rows by what they actually meant: 'auto' -> 'now'
(eligible for the scheduler immediately), 'manual' -> 'later' (parked, matching
StartMode's own "lands the task in paused" description). store_migrations_test
covers the remap and that 'later' inserts clean while the retired spellings
are rejected.
dispatcher.cpp's on_download_add matched: default (absent startMode) is now
'now' instead of the invented 'auto'; 'later' actually lands the task in
`paused` (pause_reason 'user') instead of a dead 'manual' -> `new` branch that
spec.startMode (typed as the 3-value enum) could never even reach.
TaskRow::start_mode's in-memory default followed suit ('now').
2. main.cpp's single-instance guard bound an abstract socket named
"velox-daemon-<euid>" — one name per user, system-wide. XDG_RUNTIME_DIR
isolation never reached it: a leaked test veloxd held the lock for 4h40m and
locked out every other isolated instance with the same euid (PROTO, EXT, the
orchestrator), real daemon included.
Extracted rpc/single_instance.{hpp,cpp} (was a static in main.cpp, untestable)
and derived the abstract-socket name from a hash of the resolved runtime dir
path instead of euid alone. The real per-user daemon is still unique (its
runtime dir is unique to it); isolated instances pointed at their own runtime
dirs now coexist. main() resolves the runtime dir before acquiring the lock
(was the other way around). New single_instance_test covers same-dir refusal,
different-dir coexistence, and release-on-close.
Verified against real veloxd binaries, not just unit tests: startMode: "later"
via a live download.add lands in `paused`; two veloxd with different runtime
dirs run concurrently, two with the same one and the second refuses with the
runtime dir named in the error. Full ctest: 39/39.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GRDjHGgpYmMoPE2UFbe7pP
Pure functions only, per AGENT-CORE.md's build order: no I/O, no JSON, no SQL,
no notion of the wire Rule type — DAEMON decodes its own stored/wire
representation into these plain structs and calls in.
- rules/filename.hpp: sanitize_filename() turns a raw candidate (from
net::parse_content_disposition or net::url_filename — neither is
filesystem-safe by design; both headers say so and point here) into one
safe to create on ext4/APFS/NTFS: strips separators and control bytes,
folds NTFS-illegal characters, neutralizes reserved Windows device names,
clamps length on a UTF-8 boundary. Total on hostile input; never empty.
Not the path-traversal security boundary — that's daemon/fs/safepath,
downstream of this and the one that actually matters adversarially.
- rules/collision.hpp: resolve_collision() finds the next free name
Explorer/Finder-style ("name (1).ext", ...) given an existence predicate,
or returns the desired name unchanged under an overwrite policy. Never
fabricates a guaranteed-unique name past its attempt bound — hands back
the last candidate tried rather than hiding a persistent collision.
- rules/match.hpp: match_rules() is the evaluation half of
contracts/schema/types/Rule.schema.json — priority order, first rule
whose present match clauses (extensions/mimeTypes/host & url glob/size
bounds) all hold, wins; a size clause never matches speculatively before
the probe fills in size_bytes. glob_match() is the iterative (not
recursive — bounded work on an all-'*' pattern) matcher both host_pattern
and url_pattern use.
Every header compiles standalone; tests (39 cases) pass under ASan+UBSan and
TSan. core/include/vdm/README.md documents the new public surface.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Q3QrF7rCt21bkAjt9BCDFQ
Three subcommands in one binary, driving vdm::Engine directly (docs/04 §8):
- throughput: a single download against a fast local origin
(support/local_server.hpp, busybox httpd), reporting Mbps/CPU%/RSS. Gates
on --require-mbps/--max-cpu-pct only when passed, so the ctest smoke
registration stays a correctness check, not a hardware-dependent
perf gate -- the real 1-Gbit-link sign-off is a manual/CI job (see the
file's header comment).
- load: N concurrent tasks against tools/testserver's `throttled` mode
(support/testserver_client.hpp), reporting peak RSS via getrusage(). Paced
externally rather than through the engine's own rate::RateLimiter or
busybox: the limiter's pause/resume path allocates on every throttle event
(would contaminate alloc-check's measurement) and under heavy segment
contention was found to starve individual tasks indefinitely (see
docs/adr/0016, added here); busybox couldn't sustain the DoD's ~160
concurrent connections (20 tasks * default_segments=8) reliably. The
ctest registration runs at reduced concurrency under sanitizer presets --
see the CMakeLists.txt comment and the ADR's postscript.
- alloc-check: operator new/delete overridden process-wide, sampling the
allocation count across a steady mid-transfer window against a paced
tools/testserver origin. Caught a real bug in the same change (see the
http_client.cpp commit) and, by dropping its Engine mid-download to end
cleanly, also surfaced the quiesce() use-after-free (see that commit).
core/docs/m7-baseline.md records actual measured numbers against the M1/M7
DoD lines, including where they don't clear yet (RSS ~70 MB vs a 60 MB
target; throughput/CPU only measured on loopback, no 1 Gbit link available
here) rather than rounding them away.
docs/adr/0016 documents a rate::RateLimiter fairness gap found building the
load subcommand: a single shared TokenBucket under heavy segment contention
has no fairness ordering across its peek/commit race and can starve a
waiter well past what its configured rate implies. Filed as a follow-up
(it's a core/src/rate design question, not a tools/bench one) rather than
fixed here, along with a related TSan-only load-test straggler that could
not be root-caused in this environment.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Q3QrF7rCt21bkAjt9BCDFQ
HttpClient::Impl::drain_commands() default-constructed a std::deque<Command>
every call, on every iteration of the worker thread's event loop (once per
curl_multi_poll wake -- i.e. once per socket-readiness event on the transfer
hot path), then swapped the (usually empty) command queue into it. In
libstdc++, an empty std::deque still allocates its map array on construction,
so this was a real allocation on the hot path regardless of whether any
command (add/pause/resume/cancel) was actually pending -- which is the
common case, since those are rare next to data arriving.
Found via tools/bench's alloc-check, which is built in this change and
exists specifically to catch this class of bug (AGENT-CORE.md: "no
allocation in the curl write callback... checked in review and by a bench
assertion"): before this fix it reported thousands of allocations/sec under
a sustained transfer; after, single digits.
Fixed by checking `w.queue.empty()` under the lock before touching `local`
at all, so the deque is only constructed when there's actually something to
swap into it.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Q3QrF7rCt21bkAjt9BCDFQ
DownloadTaskState::quiesce() (Engine shutdown / ~Engine, via quiesce_task())
cancelled every live worker's transfer, then immediately cleared `workers` on
the calling thread. transfer.cancel() only *requests* the HttpClient worker
thread stop the transfer -- it does not wait for that to happen. If that
thread was mid write-callback (seg_data -> WriteBuffer::append ->
SparseFile::write_at), clearing the map destroyed the SegWorker (and its
ring buffer) it was still writing through: a heap-use-after-free, caught by
ASan via tools/bench alloc-check, which by design drops its Engine while a
download is still active mid-sample.
Every other exit path (verify/fail/auto_pause/demote, via begin_drain_locked)
already gets this right: cancel, then let each worker remove and flush
itself through seg_finished once HttpClient actually confirms the transfer
stopped, on the correct thread. quiesce() now does the same instead of
tearing the map down itself -- wait on a condition variable, notified from
seg_finished right after it erases, until `workers` is empty.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Q3QrF7rCt21bkAjt9BCDFQ
Resumes work left mid-session on the M1 hostile-mode matrix. download_task.cpp:
- Replace the old cancel-then-clear teardown (cancel_all_transfers_locked /
start_assembly_locked) with a single begin_drain_locked()/PendingAction
mechanism: cancel every live worker, remember what to do (verify / fail /
auto_pause / demote), and let whichever worker's seg_finished finds the
worker map empty carry it out. Every sibling still flushes its buffer on
the way out, so no buffered-but-unflushed tail is lost when a download
finishes or fails while other segments are still mid-transfer.
- A 200 where 206 was expected (wrong_status) now checks the response's
ETag/Last-Modified against the probe's: a real mismatch asks the user
(server_file_changed, "ask, never silently corrupt" -- docs/04 §5); a match
means the server just stopped honouring Range for this connection, so
demote to one segment and keep going without a round trip (docs/04 §7).
- 416 mid-download (stale range metadata) now surfaces as a decision instead
of retrying the same now-invalid range to exhaustion.
- do_decide's abort path surfaces the actual reason a decision was asked
for (last_error) instead of hardcoding server_file_changed, which was
mislabeling a 416 abort.
engine_test.cpp adds the four hostile modes where a bug means silent
corruption rather than a visible failure: etag-changes, 416-always,
lies-about-accept-ranges, content-length-mismatch.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Q3QrF7rCt21bkAjt9BCDFQ
run.sh's TS runner only ever started mockd; veloxd existed but nothing in
the ctest -L conformance path touched it, so mockd's always-valid fixtures
were the only thing ts/replay.ts ever saw. That let a real bug through:
veloxd's download.get can return segments: 0, which TaskSummary.segments
forbids (minimum 1, required) — nothing caught it.
Add a 3b step to run.sh (still the one canonical entry, per ADR 0014):
builds veloxd, starts it isolated (its own XDG_RUNTIME_DIR/XDG_DATA_HOME/
XDG_CONFIG_HOME), seeds saveTo.allowedRoots/defaultDir directly into the
isolated velox.db (settings.set is itself a stub, and the default
~/Downloads root doesn't isolate download.add's writes), then replays
every fixture against it over both transports.
Most handlers are still stubs (daemon/docs/deferrals.md D1-D4b). Fixtures
that hit them get an expected-failure entry in the new veloxd-xfail.json,
loaded by replay.ts's new --xfail flag. This is a maintained allowlist,
not a snapshot: a listed fixture that unexpectedly *passes* is flipped
back to a failure (applyXfail), so the list can only shrink as DAEMON
lands handlers, never rot into a list nobody rechecks. download.get's
segments: 0 is deliberately *not* on it — that's the regression this
step exists to catch.
Also hardened setupBindings: a server that can't even complete fixture
binding used to take the whole runner down with an uncaught exception
before a single fixture was checked. It's now a reported Outcome instead,
so the run still produces a coherent report. That robustness fix earned
its keep immediately: veloxd's download.add crashes on startMode
"later" (a valid, documented StartMode — "the File Info dialog's
Download Later button") with a SQLite CHECK constraint violation, because
migrations/0001_initial.sql's start_mode CHECK never had 'later' added to
it (and includes 'manual'/'auto', neither a contract value). That's a
second, more severe bug this wiring found, unrelated to segments: 0 and
currently blocking most of the veloxd run — filed for DAEMON in
tests/conformance/README.md, not fixed here (out of lane). capture.offer
and capture.getRules are also stubs but missing from deferrals.md's
D-list; xfailed with a note asking DAEMON to add the row.
Verified live once against a real, isolated veloxd before this session's
sandbox became persistently contended for veloxd's single-instance lock
(UID-scoped, not namespaced by XDG_RUNTIME_DIR — daemon/src/main.cpp;
documented as a caveat in the README): it built, started isolated, seeded
settings, connected over both transports, and surfaced the startMode bug
above as a real, non-xfailed failure — confirming the whole pipeline
including --xfail end to end. segments: 0 is confirmed by direct reading
of daemon/src/store/tasks.{hpp,cpp} (TaskRow::eff_segments defaults to 0,
copied verbatim into TaskSummary.segments) rather than by a second live
run reaching that specific fixture, since setup itself fails first on the
startMode bug above. mockd path re-verified green after these changes
(200/200, up from 196/196 — the new setup/$taskId outcomes are visible
and passing).
Recommendation for PKG: don't flip this required yet. The existing
`conformance` ctest entry is already a required check, and right now the
startMode bug fails most of the veloxd run, not just the one expected
segments: 0 case — merging as-is would block every lane's PRs on two
DAEMON bugs at once, one of them unrelated to what this task set out to
catch. Required once DAEMON lands a fix for startMode "later" at minimum;
segments: 0 can stay red for a while by design, same as any other tracked
regression.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01SFeUKLbdHizrJjLBeK7ffz
Two independent signals feed one panel, per docs/05 §3:
- capture/media.ts (background): webRequest-based, recognizes .m3u8/.mpd
URLs and the HLS/DASH content types, deduped per (tab, url) so an
HLS live-refresh doesn't re-fire. media-bridge.ts relays a hit to the
tab's content script over runtime.sendMessage, and separately answers
the content script's media.listVariants/media.addVariant calls by
forwarding them to the background page's transport.
- content/media-observer.ts: watches the page's own <video> elements
(present now, added later, or with src changed) for the same
extension signal, independent of what the network sniffer saw.
Either firing opens content/video-panel.ts's "Download this video ▾"
panel (built with createElement, matching the popup's innerHTML-free
approach), which lists variants from media.listVariants and greys out
any variant.drm or a wholly drmProtected manifest with "Protected
content" rather than attempting it. The extension still never parses a
manifest itself — that stays in the daemon, one language, one place.
content/index.ts is the manifest-registered entry (content_scripts in
manifest.json, added in the previous commit); build.mjs builds it as an
IIFE rather than ESM, since a manifest content script has no "type":
"module" declaration and an emitted top-level export would be a syntax
error there. tsconfig.json adds DOM.Iterable for NodeList iteration.
docs/05-extension-spec.md gets a short addendum (§8) documenting the
popup/options bridge and this media-detection split, since neither was
in the original design write-up.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Ed8KEmAW48v4YHdxLtqsMB
Popup and Options are separate documents from the background page and
can't reach its live VeloxTransport directly, so background/bridge.ts
relays it over one browser.runtime.connect port per document
(call/subscribe/getStatus/reconnect/pair/unpair/setOverride in;
result/event/status/pairError out). shared/panel-client.ts is the
client side both surfaces use.
Popup (src/popup/): status dot + text, active-downloads list driven by
event.task.progress/added/state/removed (repaints ride the event's own
<=4 Hz cap rather than adding a second timer), pause/resume buttons,
"Start it" wired to a reconnect request. State lives in a DOM-free
store.ts for unit testing; rendering uses createElement, not innerHTML
(web-ext lint flags the latter).
Options (src/options/): transport override select, pairing (code entry
+ pair/unpair, backed by two new WebSocketTransport methods,
pairWithCode/unpair), and the daemon's capture policy mirrored via
capture.getRules. The capture-policy form is editable only when the
active transport is native messaging (uds) -- settings.set and
rules.upsert are privileged, uds-only methods per shared/protocol
METHODS, so WebSocket can't write them no matter what the page shows;
CLAUDE.md section 2 rules out working around that locally. The bridge's
status payload adds a kind field (which transport is live) for this to
key off. Default category is the one piece of state that's genuinely
the extension's own, not the daemon's, and lives in
browser.storage.local via options/prefs.ts. Pure decisions (statusLine,
pairingAvailable, captureRulesEditable) are split into view.ts for unit
testing without a DOM.
manifest.json registers the popup action and options_ui page (and, in
the same edit, the content_scripts entry the next commit's media
detection needs -- split by file, not by manifest line). build.mjs
gains popup/options as further esbuild entry points, plus copying their
static HTML/CSS into dist/.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Ed8KEmAW48v4YHdxLtqsMB
tests/transport/storage.test.ts covers the storage half of "the pairing
token survives a browser restart" (round-trip, unpair-clears, corrupted
override falls back to auto). websocket.test.ts adds the transport half:
a fresh WebSocketTransport instance over the same backing store reuses
the persisted token with no re-pairing, plus pairWithCode/unpair
coverage. "Wrong token rejected and rate-limited" was already covered
(websocket.test.ts's NotPaired/RateLimited cases).
docs/amo-permissions.md is the submission-ready permission justification
for AMO's Notes to Reviewer field, covering every permission in
manifest.json plus what was deliberately not requested and how cookie/
header data is handled.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Ed8KEmAW48v4YHdxLtqsMB
CLAUDE.md §3's rule was prose only for extension/ (GUI already has
gui_no_download_logic as a ctest). eslint.config.mjs adds a
no-restricted-syntax/no-restricted-globals rule banning fetch/XHR/Request,
ReadableStream.getReader, Range/Content-Range header construction, and
IndexedDB in src/**/*.ts. Verified red on a planted violation (fetch +
Range header + stream reader) and green on ordinary code; that check is
now a permanent regression test (tests/lint/no-download-logic.test.ts)
rather than a one-off manual run. Wired into the existing extension-lint
job in .github/workflows/ci.yml, ahead of web-ext lint.
Generated protocol code (src/shared/protocol/**) is excluded from lint
entirely — it must never be hand-edited, so flagging it as fixable would
be a lie.
Answers gui/docs/ext-requests-m1.md.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Ed8KEmAW48v4YHdxLtqsMB
Engine numbers never reached the store: download.get after a correctly-finished
download reported sizeBytes: null, downloadedBytes: 0, speedBps: 0, resumable:
false, segments: 0, segmentDetail: [] — segments: 0 breaks the frozen contract
(TaskSummary.segments is minimum:1, required).
- Scheduler::tick() now probes (EnginePort::probe) before every start(), persisting
sizeBytes/resumable/validators via Tasks::set_probe_result before a byte moves,
then starts the engine with that ProbeResult as probe_hint.
- Scheduler::persist_progress() (new) writes downloadedBytes/speedBps/segments/
segmentDetail from the engine's Progress. Called from progress_snapshot() (the
250ms tick) *and* once more from on_engine_state right before release()/unmap on
every terminal transition, so a task that finishes between two ticks — the common
case for anything small or fast — still leaves real numbers instead of the
pre-persistence defaults.
- TaskSummary.segments is sourced from segments.size() when the task has any
(matching what actually lands in segmentDetail, per the schema's "exactly
segments entries"), falling back to the engine's effective_segments (budget
slots *held*, not necessarily physical range count) only pre-segmentation.
- Tasks::set_final_bytes tops up on_finished's byte count as a last-resort
backstop.
- store/segments.{cpp,hpp}: read/write access to the segments table behind
TaskDetail.segmentDetail. Wired into daemon/CMakeLists.txt.
- migrations/0002: speed_bps on tasks and segments; fixes segments.state's CHECK
to include 'pending' (0001 omitted it, so a pre-connect snapshot could never be
written).
- store_migrations_test's forward-only loop faked "released version N" by setting
the user_version pragma alone, with no real schema underneath — never exercised
until 0002 existed. Fixed to actually build the db through migrations 1..N first.
Verified against real veloxd + tools/testserver (not just unit tests):
download.list/download.get correct immediately after completion and after a
daemon restart, with saveTo.allowedRoots pointed at an isolated dir.
Observed but not fixed (CORE, not this lane, noted in deferrals.md): Progress.
speed_bps reads back 0 for the whole lifetime of a live throttled download in the
same E2E check, despite downloadedBytes visibly advancing. DAEMON passes it
through unmodified; filed rather than worked around.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GRDjHGgpYmMoPE2UFbe7pP
Wires up the three dialogs from build order step 5 (docs/03-gui-spec.md
§§2-3) and the MainWindow slots that were declared but never implemented:
- AddUrlDialog: clipboard prefill is the explicit path docs/06 R2 calls
for (no passive monitoring, not advertised).
- FileInfoDialog: async download.probe never blocks the UI; ends by
calling download.add itself (Now / Later / Add to Queue). buildSpec()
is a pure static so the optional-field-omission logic is unit-testable
without touching a widget.
- ProgressDialog: non-modal, WA_DeleteOnClose, driven by the taskProgress/
taskStateChanged signals RpcClient already re-broadcasts; download.get
seeds state once for a dialog opened mid-transfer. Hosts SegmentBarsWidget
and SpeedGraphWidget.
MainWindow: openAddUrlDialog/openPropertiesForSelection/showTableContextMenu
now have bodies; category.list/queue.list responses are cached so File Info
can populate its category combo and queue menu without a second round trip.
The row context menu covers what already exists (Resume/Pause/Stop/Delete/
Properties) and deliberately leaves out Open/Open With/Move-Rename/
Redownload/Add to Queue — those need dialogs later build-order steps haven't
reached yet.
util/Format.hpp: pulled the bytes/rate/eta formatting out of MainWindow and
DownloadTableModel once the dialogs wanted the same strings a third time.
Verified end-to-end against a running mockd (category.list/queue.list,
download.probe, download.add, download.get, and live event.task.progress/
event.task.state) under ASan+UBSan: all three dialogs render correctly
against real fixture data and the flow runs clean with no leaks or
sanitizer reports.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01NSCdCWFXBTSBBK3MzWtJiC
Per-connection progress bars and the 60 s rolling speed graph the progress
dialog needs (docs/03-gui-spec.md §3). Both reuse row/widget state across
ticks instead of rebuilding, matching the discipline DownloadTableModel
already uses for progress patches.
SpeedGraphWidget keeps a fixed ring buffer and one reused QPainterPath —
no allocation in paintEvent or addSample. Fixed a real bug found while
writing tst_speedgraphwidget: the elapsed timer was started in the
constructor, so the very first sample after construction would silently
wait up to 1 s to be recorded instead of landing immediately.
Tested against mockd (both offline via QTest/offscreen, and manually
against a running mockd instance through ProgressDialog once that lands).
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01NSCdCWFXBTSBBK3MzWtJiC
Pointed the existing RPC client at a real veloxd instance (isolated
HOME/XDG_RUNTIME_DIR/XDG_DATA_HOME — never touched the real ~/Downloads),
added a real download.add against tools/testserver, and watched it render
live end to end with no GUI code changes:
- handshake, subscribe, category.list/queue.list all match what DAEMON
reported
- the full event.task.state sequence and batched event.task.progress
both applied correctly by DownloadTableModel
- the written file's SHA-256 matches the server's reference
Filed gui/docs/daemon-requests-m1.md: TaskSummary.sizeBytes is never
populated by this daemon build, even in download.get after the task
completes with the exact byte count already on disk. Not a GUI bug —
ProgressDelegate and the model already do the documented right thing
when size is unknown (fall back to plain text, no bar) — but it means
every task renders without a percentage against the real daemon today.
mockd always supplies sizeBytes so this doesn't block current GUI work;
flagging before the M1 GUI<->daemon integration pass.
mockd stays the primary harness for the unhappy paths
(--slow/--flaky/--drop-connection) that a real daemon won't misbehave on
command for.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_016Ne28kx4VreeBWZv82Nksd
The two items aimed at pointing GUI at a real veloxd instead of mockd.
rpc/event_hub — per-subscription fan-out shared by both transports.
subscribe() registers a connection with no interest; set_filter()
(session.subscribe, replaces not adds) turns on event kinds and an
optional per-task id filter; publish() delivers a pre-built
notification to every matching subscriber. session.subscribe on both
UdsServer and WsServer now does the real thing — registers/updates a
subscription, tears it down in close_conn.
sched/scheduler — the on_engine_state hook now actually publishes:
- transition() is the one place a task's row changes state; it reads
the store's own prior row for previousState (authoritative
regardless of engine/scheduler timing), writes the error columns,
and — when a hub is supplied — publishes event.task.state with
{taskId, state, previousState, summary, error}. Wired into every
transition: scheduler-driven (admission -> probing, resume ->
connecting, pause) and engine-reported (on_engine_state).
- progress_snapshot(): one row per task the engine is tracking
(EnginePort::progress(), a new interface method backed by
DownloadHandle::progress()), plus a store side-effect
(Tasks::update_progress) so download.list/get stay current between
state transitions. Returns rows; does NOT publish itself — batching
into one array message is the caller's job, per the schema's
x-maxRateHz: 4 and AGENT-DAEMON.md item 5 ("one message per task per
tick burns a core"). main.cpp's 250 ms timerfd is that caller: one
event.task.progress per tick, only when there's something to say.
dispatcher::on_download_add now publishes event.task.added (schema:
"summary is always present so a client can insert the row without a
follow-up download.get").
store/categories, store/queues — the two D3 handlers GUI's panels
call. category.list projects the six seeded built-ins; queue.list
derives taskIds from tasks.queue_id/queue_position (Queue's own schema
note: a queue's stored row never carries membership, download.update
/ queue.reorder do).
Verified live end to end against tools/testserver: a subscribed client
sees event.task.added on add, then the full event.task.state sequence
(queued -> probing -> connecting -> downloading -> assembling ->
verifying -> complete) with correct previousState at every step, and
real category.list / queue.list results.
Tests: event_hub (filter-by-kind, filter-by-task-id, replace-not-add,
unsubscribe), store_categories_queues, plus new sched_scheduler cases
for event.task.state publishing and progress_snapshot's store
side-effect. 38 daemon/cli tests green; sched_scheduler / event_hub /
ws_server / uds_roundtrip TSan-clean.
deferrals.md: D5 mostly closed (event.task.removed and the
still-unpublished events wait on their owning D3 handlers); D3 down to
the remaining download.* verbs, rules/settings/limiter/schedule,
queue mutation, category mutation, grabber, media.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
CORE stage 8 merged, so vdm::Engine is linkable. This closes D4a and
narrows D4b: `velox add <url>` now actually downloads.
- sched/engine_port_core.hpp — the real EnginePort: forwards to a live
vdm::Engine, keeps the DownloadHandle per task for pause/resume/
cancel/provide_auth/decide/refresh_url, drives set_task_order /
set_max_active_segments / set_host_segment_cap via
engine.segment_budget(). CORE confirmed the admission model: DAEMON
decides when to start(); the engine's own download_task calls
register_task/set_want internally — DAEMON never touches per-task
budget calls. EnginePort gains release(TaskId) so the port drops a
handle when the task goes terminal.
- rpc/event_loop — EventLoop::post(fn): thread-safe, runs fn on the
loop thread next iteration. The marshaller for engine-thread
callbacks.
- main.cpp — constructs vdm::Engine + EnginePortCore + Scheduler
(post_to_loop = loop.post). At startup: reconcile_after_restart()
(ADR 0013 §5), reload_config(), tick(). A 1 s timerfd on the loop
re-runs tick() (schedule windows, missed nudges); download.add nudges
via dispatcher.set_on_mutation.
End-to-end verified against tools/testserver: `velox add
http://127.0.0.1:.../file/512K` -> task queued -> scheduler admits ->
engine downloads 524288 bytes -> complete, file on disk. First
byte-path all the way through the project.
safepath-adversarial.md: re-verified per its own note — CORE landed
O_NOFOLLOW on the target open (core/src/io/sparse_file.cpp), so the
leaf-symlink TOCTOU is now closed; residual is down to one
intermediate-dir gap (documented post-M1 chase).
36 daemon/cli tests green; scheduler + uds_roundtrip TSan-clean.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
The Scheduler that D4 was waiting on. Built against CORE's engine
HEADERS (now in main); the real EnginePort and the veloxd wiring wait
for lane/core's stage-8 bodies to reach main (deferrals.md D4a/D4b) —
core/src/task/ is still .gitkeep there, so linking vdm::Engine now
would be an unresolved symbol.
- sched/engine_port — the abstract seam: start/pause/resume/cancel/
provide_auth/decide/refresh_url + the ADR 0011 admission config
(set_task_order / set_max_active_segments / set_host_segment_cap).
Keeps the Scheduler testable without a live engine and the daemon
unbound from the concrete vdm::Engine.
- sched/fake_engine_port — a recording impl for tests.
- sched/scheduler:
* owns the wire-UUID <-> vdm::TaskId map.
* tick(): snapshot queues (schedule window evaluated with an
injectable clock) + non-terminal tasks -> governor.evaluate ->
apply. to_start builds a vdm::task::DownloadSpec from the row and
calls EnginePort::start; to_resume -> resume(); to_pause ->
pause() + writes the pause_reason; priority_order -> set_task_order
over the mapped engine ids. `new` tasks are parked (startMode
manual) and skipped.
* on_engine_state(wire_id, state, err): projects an engine
transition onto the store row (state, pause_reason='auto' when an
error rides a paused transition per ADR 0013 §2, flattened error
columns) so the next tick sees ground truth. This is also the hook
event.task.state will fire from (D5).
* reconcile_after_restart(): CORE-owned states -> queued, paused
keeps its reason (ADR 0013 §5).
* reload_config(): reads connection.maxConcurrentDownloads /
maxActiveSegments + a daemon-local host-cap map, pushes caps to
the engine, updates the governor.
* Deps: injectable local-now clock and a post_to_loop marshaller
(engine callbacks arrive on engine threads; default runs inline
for tests).
Test veloxd.sched_scheduler (ASan+UBSan and TSan clean): admission +
ordering, a slot freeing on completion, queue-stop -> pause
(queue_stopped) then queue-restart -> resume (not a fresh start),
engine auto-pause -> pause_reason 'auto' + never auto-resumed,
reconcile_after_restart, reload_config caps push. 35 daemon/cli tests
green.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
Stage 8 of the CORE build order: the bodies behind the DownloadSpec /
callback API reviewed in core/docs/engine-api-m1.md. Wires probe -> segment
workers -> WriteBuffer -> SparseFile -> .veloxpart.meta -> retry/backoff ->
SegmentBudget -> RateLimiter -> callbacks into one event-driven machine.
- Engine (src/engine.cpp): owns HttpClient, Prober, SegmentBudget,
RateLimiter and one timer jthread (min-heap of scheduled fns). start()
builds a task and returns a DownloadHandle; ~Impl quiesces every task
before joining the timer so no callback fires during teardown.
- DownloadTaskState (src/task/download_task.cpp): one `mu` task lock; a
shared_mutex over the worker map for the curl write path; callbacks
collected under `mu` and fired after release via a separate deferred
queue; weak_from_this() in every async hop. State machine over the
CORE-owned EngineState subset, auto-pause on 401/407 and on a 200 where
206 was expected, validated resume via If-Range.
- digest (src/task/digest.cpp): OpenSSL EVP hash_file() for the optional
post-download checksum; links OpenSSL::Crypto PRIVATE.
- Segmenter::release_segment(): hand a paused segment back to the pool
unassigned so resume's assign_slot() picks it up instead of splitting a
still-"assigned" range and orphaning its front half.
- DownloadHandle now names the real control block (vdm::task::
DownloadTaskState, defined only in the engine TU) via a namespace-scope
fwd decl and a public-but-effectively-engine-only ctor, replacing the
nested State/friend pair. Every public signature is unchanged; DAEMON
(vdm-79) confirmed sched/ names only the public API.
Fixes found while building the end-to-end suite (tests/task/engine_test.cpp,
9 cases against tools/testserver, green under ASan/UBSan and TSan):
- a dropped connection lost its unflushed WriteBuffer tail while advance()
had already counted those bytes as done -> a retry resumed past an
unwritten hole. Flush on the failure path.
- when the byte counters hit total while other workers were still live,
teardown dropped their buffered tails. Now: cancel them and let each
worker's own seg_finished drain it (the `assembling` state), last one
starts verification -- no cross-thread buffer access.
- seg_head() let a 401 with credentials present abort before libcurl's
resend; now it proceeds once and acts on the final status.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HPPSGhiArbvQgwC2DNiURS
run_all() takes an optional substring list; vtest_main forwards argv and a
comma-separated $VT_ONLY. No filter => run everything, as before. Makes
iterating on one slow end-to-end case (the engine suite) practical
without a framework swap.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HPPSGhiArbvQgwC2DNiURS
libcurl with CURLAUTH_ANY answers a 401/407 by resending the request with
an Authorization header. Two spots in net/ cut that short:
- http_client's header callback delivered the response head exactly once
and latched `head_delivered`, so after an auth challenge the caller only
ever saw the 401 — never the 2xx of the authenticated resend. Reset the
latch when a fresh status line follows a delivered 401/407 (redirects
never reach that path — their head is suppressed).
- the prober's head callbacks return DataAction::abort to skip the body,
which also aborts the transfer mid-handshake. Return `proceed` for a
401/407 when credentials were supplied, so curl's resend can run; the
real status lands on the next header block.
Also give ProbeRequest an `auth` field (default scheme == none) and pass
it through base_request(), so a re-probe after a 401 can present the
credentials the user just entered. No behaviour change when no auth is
configured.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HPPSGhiArbvQgwC2DNiURS
VeloxDispatcher now takes a store::Db& and three handlers are real:
- download.list -> store::Tasks::list (filter / sort / paging in SQL) ->
to_summary per row. No more empty-table stub.
- download.add -> resolve saveDir (spec, else saveTo.defaultDir; ~ expanded)
and the leaf (spec.filename, else the URL's last segment percent-decoded,
else download.bin) -> fs::resolve_target against canonicalize_root'd
saveTo.allowedRoots. Any path-destination failure is -32011 with the
*original* saveDir in data.path. On success a TaskRow is inserted in
state `queued` (or `new` for startMode "manual") and {taskId, state}
returned. The scheduler that would then admit it is D4.
- download.get -> store::Tasks::get; a real -32010 + data.taskId for an
unknown id, else a TaskDetail (segmentDetail empty until the engine
segments the task, which the schema permits).
util/time.hpp: now_iso() factored out of ws_server.cpp.
main.cpp constructs the dispatcher with the opened db. The three
integration tests build an in-memory migrated db for it; velox.client
now drives the full slice through the CLI — add outside roots -> -32011
with data.path, add into an allowed root -> a task that download.list
shows and download.get details, unknown id -> -32010. Verified with the
real binaries: velox add persists, velox ls shows it, it survives a
daemon restart, /etc is refused.
ASan+UBSan and TSan clean; 34 daemon/cli tests green. deferrals.md:
D2 down to just download.probe; D3 down to categories/queues/rules/
settings/limiter/schedule.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
DAEMON's safepath-adversarial.md accepts a TOCTOU residual between its
canonicalise-and-check and the download starting, on the stated grounds
that CORE's O_NOFOLLOW open of the final file closes it. That flag was
never actually set: SparseFile::open used O_WRONLY|O_CREAT|O_CLOEXEC, so
a symlink swapped in as the final path component after DAEMON's check
would be followed and redirect our pwrites outside the allowed roots.
Add O_NOFOLLOW. A symlinked leaf now fails the open with ELOOP, which
errno_to_error already maps to Error::path_rejected. Regular files and
the O_CREAT of a fresh part file are unaffected; resume (existing regular
part file) is unaffected. Test that a symlinked destination is rejected
rather than silently followed, and that the link target is never touched.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HPPSGhiArbvQgwC2DNiURS
The residual section claimed the leaf/component TOCTOU is "closed in
practice by CORE's O_NOFOLLOW open of the final file". Verified: it is
not — core/src/io/sparse_file.cpp:77 opens O_WRONLY|O_CREAT|O_CLOEXEC,
no O_NOFOLLOW, no O_EXCL. Requested the flags from CORE via PKG/QA.
Doc now states the residual is currently OPEN, names the file:line and
flags checked and the date, says what actually limits exposure today
(0700 parent dirs), and flags this as the boundary where a reader
stops checking. Step 5 reworded the same way. Re-verify the flags when
the CORE change lands.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
The read/write surface the dispatcher handlers need, so download.add
persists and download.list / download.get project real rows when the
engine lands.
- store/tasks — TaskRow (1:1 with the schema), insert / get / remove /
set_state / count, and list(filter, sort, offset, limit) that does
all the WHERE / ORDER BY / LIMIT in SQL (M1 DoD: a 1000-row list
never materialised client-side). Filter covers states / category /
queue / case-insensitive filename+url substring / date range; sort
is a whitelisted column + direction with NULLs last, default
newest-first; the enum spellings in a state IN (...) come from
proto::to_string, never from user text. to_summary() projects a row
onto proto::TaskSummary including the flattened error block when the
task failed / retry_wait / auto-paused.
- store/settings — key -> JSON-text with a built-in default table
mirroring Settings.schema.json / ADR 0012; get_raw / set_raw /
overrides plus typed get_int / get_string / get_string_array for the
governor config and saveTo.allowedRoots. Full settings.get/set wire
projection lands with those handlers.
- veloxd_store now links velox::proto + nlohmann_json for the
projection.
Test veloxd.store_tasks: insert/get round trip, PK duplicate rejected,
the error-block projection, list total+paging+sort+every filter,
set_state pause_reason clear-on-unpause, remove, and settings default
vs override. ASan+UBSan and TSan clean; 34 daemon/cli tests green.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
veloxd is the one process that turns an untrusted string into a
filesystem destination, and via capture.offer that string can come
from a web page. CLAUDE.md §4 and the M1 DoD both name this.
daemon/docs/safepath-adversarial.md is the spec, written before the
code the way EXT did for shouldCapture: 21 rows — .. traversal
(A1/A2), absolute-outside-roots (A3), prefix-match confusion (A4),
symlink-out (A7), TOCTOU on a created tail (A8), NUL/control bytes in
the leaf that CORE's fuzzer hit through Content-Disposition (A9/A10),
degenerate and overlong leaves (A11/A13), overlong dir component
(A14), symlinked root (A16), destination-is-a-file (A17), and the
legitimate cases that must still pass — non-ASCII (A18), redundant "."
(A19), trailing space/dot trimming (A20).
fs/safepath.cpp:
- sanitize_leaf: strip <0x20 and 0x7F, trim ws, strip trailing dots,
reject ""/"."/".."/contains-'/', cap 255 UTF-8 bytes on a codepoint
boundary. Mirrors core/src/net/content_disposition.cpp.
- canonicalize_root: expand ~ and realpath each allowedRoots entry
once, so a symlinked root resolves to its target.
- resolve_target: reject relative saveDir and any ".." component
lexically; if the dir exists, realpath + component-wise containment
(a symlink that escapes is caught, one that stays inside passes); if
a tail is missing, realpath+check the deepest existing ancestor then
create the tail via an openat/mkdirat O_NOFOLLOW walk and re-derive
the final path from the fd. Every failure is -32011 with data.path =
the *original* saveDir (never the resolved path). Residual TOCTOU on
a pre-existing intermediate dir is documented and closed by CORE's
O_NOFOLLOW open of the file.
veloxd_fs static lib; veloxd_rpc links it for the download.add wiring
next. Test veloxd.safepath is the adversarial table, on a real temp
tree. ASan+UBSan and TSan clean; 33 daemon/cli tests green.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
The scheduling brain, built against CORE's headers (vdm/engine.hpp,
vdm/segment/budget.hpp) — signatures only; the Engine bodies land in
CORE stage 8 and the Scheduler that wires governor <-> store <-> engine
<-> timer comes after that (daemon/docs/deferrals.md D4).
- sched/governor — a pure decision function. In: a snapshot of every
task's coarse RunState and every queue's state (schedule windows
pre-resolved). Out: {to_start, to_resume, to_pause, pause_reasons,
priority_order}. Enforces, all in TASK units per ADR 0011 §1:
connection.maxConcurrentDownloads; the min(that, maxActiveSegments)
clamp (§2); Queue.maxConcurrent; the per-host task cap (§4); and a
stopped queue / closed window runs nothing. Never touches a task
paused for `user` or CORE's `auto` (ADR 0013 §3) — only Schedule /
QueueStopped / AdmissionReconcile are auto-resumable. Deterministic:
main-list before queued-in-queue, then queue order, then FIFO, then
task_id.
- sched/schedule_window — window_open(Schedule, local tm): disabled =>
always open; `once` => date + time match; `periodic` => weekday in
daysOfWeek (empty = every day) + time in [start, stop); null start =>
midnight, null stop => end of day, stop < start => overnight window.
Pure; re-evaluated every tick, no cached instants.
- veloxd_sched static lib; veloxd links it (nothing calls it yet).
Tests (ASan+UBSan and TSan clean): veloxd.sched_window (10 window
cases incl. overnight, once, null bounds), veloxd.sched_governor
(global/clamp/per-queue/per-host caps, stop vs window pause reasons,
resume-only-governor-reasons, auth-pause untouched, admission
reconcile, determinism under shuffled input). 32 daemon/cli tests
green; full tree green.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
CORE resolved all four (lane/core@3da4cd6): vdm::TaskId is hashable and
DAEMON never constructs one; DAEMON mkdir -p's save_path's parent
(engine -> Error::path_rejected if missing); sha512 added as the 4th
Checksum::Algo so no -32602 at the RPC edge; on_state(cancelled) then
on_finished(Err{Error::canceled}) -- note the one-L spelling in the
error taxonomy. Also notes rate/token_bucket + Engine::rate_limiter()
for the limiter.set wiring.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
rate/token_bucket.hpp — a lazily-refilled TokenBucket (starts full: burst
then throttle, IDM behaviour; rate 0 == unlimited; burst caps idle
accumulation) and RateLimiter, the global -> per-queue -> per-task
hierarchy (docs/04 §6). acquire(task, n) peeks every applicable level and
commits on all-or-none so a blocked attempt never leaks tokens at a level
that had them; held under one mutex so a concurrent detach can't dangle
the bucket it's using. vdm/ids.hpp gains QueueId.
Tests: burst/refill/cap/unlimited for the bucket; tightest-level-binds,
no-partial-consumption, detach-safety, and an 8-thread aggregate-rate
check for the hierarchy. Green under ASan/UBSan and TSan.
Engine-API review (DAEMON signed off, no sched/ or dispatch rewrite):
- Engine::rate_limiter() accessor added (limiter.set -> set_global_limit).
- Checksum::Algo gains sha512 to match the wire Checksum set.
- DownloadSpec: DAEMON creates save_path's parent dir before start();
missing dir -> Error::path_rejected (made explicit).
- cancel(): documented to fire on_state(_, cancelled, nullopt) then
on_finished(Err{canceled}), in that order; download.cancel ==
cancel(false), download.remove == cancel(true).
- engine-api-m1.md: the five open questions resolved with DAEMON's
answers (probe_hint optional, single cancel flag, {restart,
keep_partial, abort} is the whole set, per-task 4 Hz is fine,
refresh_url restarts all segments after a validating re-probe).
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HPPSGhiArbvQgwC2DNiURS
Review of core/docs/engine-api-m1.md (lane/core@d6cf1fe) in
daemon/docs/engine-api-review.md. Sign off: nothing forces a sched/ or
RPC-dispatch rewrite; the split matches ADR 0011 and ADR 0013.
Answers: (1) keep probe_hint optional — DAEMON has a ProbeResult only
on the File Info path; (2) one cancel(discard_partial), download.remove
is cancel(true) + DAEMON-side row/file cleanup; (3) {restart,
keep_partial, abort} suffices if the engine owns the mechanical 416
re-probe/re-split; (4) per-task 4 Hz progress is fine — DAEMON
re-batches across tasks for event.task.progress anyway; (5) refresh_url
restarts all segments on the new URL (the signed-URL case), mirror
rotation is spec.mirrors not refresh_url.
Four things to confirm, none blocking: vdm::TaskId copy/hash semantics
and that DAEMON never constructs one; who mkdir -p's save_path's
parent; sha512 (in the wire Checksum, not the engine enum) rejected at
the RPC edge; on_finished(Err{cancelled}) code + ordering vs
on_state(_, cancelled, _).
Integration timing: wire after sched/ lands. sched/ builds against
these signatures in parallel with CORE stage 8.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
The extension's fallback transport (docs/05 §4). veloxd now also listens
on 127.0.0.1, first free port in 52000-52016, and writes it to
<runtime>/ws.port (0600).
- rpc/ws_frame — RFC 6455 frame codec. Incremental; reassembles
continuation frames; enforces "client frames MUST be masked" (§5.1);
caps a reassembled message at 8 MiB. This is the attacker-adjacent
parser, so it has its own test table.
- rpc/ws_handshake — HTTP upgrade parse, Sec-WebSocket-Accept
(SHA-1 + base64 via libcrypto), and the two non-negotiable checks:
an Origin header must be present and must be moz-extension:// (a page
cannot pair). Version must be 13.
- rpc/ws_server — per-connection Handshake -> Open state machine on the
shared EventLoop. Token gate: session.pair mints a token behind the
approver + rate limiter; session.hello must present a valid one;
every other method is -32002 until authed. Privileged methods are
refused -32003 by the generated dispatch(). Ping -> Pong; Close
echoed. session.hello major-version mismatch -> -32001.
- rpc/pairing — PairingApprover interface + EnvAutoApprover dev stub
(approves iff VELOX_PAIR_AUTO=1); PairingRateLimiter (5 failures / 60 s
per origin, then 60 s lockout -> -32014, survives reconnect); a
four-digit code generator.
- store/pairings — the pairings table: create() returns the plaintext
token once and stores only its SHA-256; find_active_by_token,
touch, revoke, list_active.
- util/crypto — sha1 / sha256_hex / base64 / random_token over libcrypto.
- store/sqlite — pin the DB file (and -wal/-shm) to 0600.
- runtime_dir — resolve_data_dir() for $XDG_DATA_HOME/velox (velox.db).
- main.cpp — opens + migrates velox.db, starts both transports; a WS
bind failure is logged, not fatal (capture must fail open, the Unix
socket still serves the GUI/CLI).
Real gap, flagged not hidden: the pairing prompt is EnvAutoApprover for
now — a GUI dialog / desktop notification is build step 7. Pairing
needs VELOX_PAIR_AUTO=1 until then.
Tests (ASan+UBSan and TSan clean): veloxd.ws_frame (codec + handshake
vectors incl. the RFC 6455 §1.3 accept sample), veloxd.pairings (token
create/find/revoke, hash-not-token, rate-limit window + lockout +
per-origin isolation + success reset), veloxd.ws_server (full flow: 101
handshake, -32002 gate, deny-then-approve pairing, hello-with-token,
-32003 privileged refusal, real download.list). 27 daemon/cli tests
green; full tree green.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
docs/05 §3: "Download with Velox" on a link (linkUrl) or a media element
(srcUrl) hands that one URL to download.add with the page as referrer; a
configurable command (Ctrl+Shift+U) does the same for the active tab. These
are explicit user requests, so they skip shouldCapture and the fail-open
path — a failure is surfaced to the user via notifications instead.
The page/selection "Download all links…" item waits on the content-script
link harvester (build-order step 7) and will be added there.
manifest: commands.velox-grab-current-tab. 9 tests. 110 total, lint clean.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_012Y9RU58hD1BuwP82DySUHk
- CategoryPanel: the left tree — All Downloads / Unfinished / Finished,
then Categories and Queues populated from category.list / queue.list,
with per-node task counts. Selecting a node emits a TaskSelection.
- DownloadFilterProxy: QSortFilterProxyModel keyed off that selection.
Client-side for M1 (the whole list fits); asTaskFilter() exposes the
equivalent TaskFilter for a server-side download.list once paging lands.
- MainWindow: menu bar (Tasks / Downloads / View / Help) sharing QAction
objects with the toolbar; QSplitter [panel | table]; Delete with a
confirm; Resume/Pause All; View menu toggles the panel. Actions
disabled while offline.
- Counts are computed off a throttled 400 ms timer, not the 4 Hz progress
path. Fixed a debounce-vs-throttle bug found in the first screenshot:
restarting the timer on every progress tick meant it never fired and
the status bar sat at "0 of 0 downloads".
- First-run column widths that fit the content.
- tst_downloadfilterproxy: nodes filter to their own rows, the
Finished/Unfinished split is correct, and the filter is dynamic (a row
that finishes leaves the Unfinished node with no re-list). Verified
against mockd --tasks 400 (screenshot: tree counts 75/84/89/83/69 sum
to 400, status bar "400 of 400, 21 active").
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_016Ne28kx4VreeBWZv82Nksd
Downloads that never reach the header hook (form POST results, service-worker
responses, clicks Firefox routes straight to its downloader) surface here.
If one looks like the daemon's, offer it FIRST and only cancel + erase
Firefox's copy on {action:"take"} — a failed or slow offer can never leave
the user with nothing. blob:/data: downloads are left to Firefox (the daemon
can't fetch a blob URL).
- offered-urls.ts: short-lived, bounded TTL set of URLs the header hook has
already offered; the safety net checks it (via wasOffered) so nothing is
double-handled. Hook gains an onOffered hook to populate it.
- background/index.ts: both paths share one offer(), getCookies, rules
mirror, and OfferedUrls instance.
13 new tests incl. fail-open (offer rejects -> 'error', ignore ->
'offer_declined', cancel() throwing after take still returns 'taken').
101 tests green; web-ext lint clean.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_012Y9RU58hD1BuwP82DySUHk
The download entry point the AGENT-CORE brief asked for on day one and
that slipped. DAEMON has an RPC surface and a store and, until this is
agreed, nothing in velox::core to call.
vdm/task/download.hpp — DownloadSpec (the resolved subset DAEMON hands in:
absolute save_path, verbatim browser headers, requested segments/buffer,
optional probe_hint / checksum / auth, allow_resume), EngineState (the
CORE-owned subset of the wire TaskState), Progress / SegmentProgress,
DownloadCallbacks (on_progress <=4 Hz, on_state for every transition
incl. auto-pauses, on_auth_required, on_decision_needed, on_finished
last), DownloadHandle (pause/resume/cancel — idempotent per the ADR 0013
signature — plus provide_auth / decide / refresh_url, and synchronous
state()/progress() snapshots).
vdm/engine.hpp — Engine: start(spec, callbacks) -> handle, segment_budget()
(DAEMON's sched/ admission surface, ADR 0011), live connection.* setters,
a standalone probe() on the pool outside the segment budget.
core/docs/engine-api-m1.md — the review doc: field semantics, the state
machine, threading/lifetime rules (which thread callbacks arrive on, what
is legal from inside one, handle/engine lifetime), the shared-`paused`
idempotency contract as a signature, and five open questions for DAEMON.
Value types compile and are covered by api_compiles_test; Engine /
DownloadHandle bodies land in stage 8, built against whatever DAEMON
signs off here.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HPPSGhiArbvQgwC2DNiURS
For a response shouldCapture() likes: gather cookies, offer to the daemon
under a hard 750 ms budget, and return {cancel:true} ONLY on an explicit
{action:"take"}. Every other outcome — shouldCapture says no, daemon down /
slow / erroring, cookies fail, anything throws — resolves to {} and Firefox
downloads normally.
Fail-open tests written first (tests/capture/hook.test.ts): offer() rejects,
offer() never settles (resolves within budget), timeout-shaped rejection,
getRules() throws, malformed details. Plus the happy paths and a check that
the capture.offer payload is well-formed from details + stash + headers.
background/index.ts: wire the stash + hook onto browser.webRequest, mirror
capture rules via capture.getRules on connect and on a capture.* settings
change; DEFAULT_CAPTURE_RULES (enabled:false) until the first mirror lands.
84 tests green; web-ext lint clean.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_012Y9RU58hD1BuwP82DySUHk
Six lanes pick ADR numbers with no allocator. This lane dodged one collision
by taking 0012 while DAEMON drafted 0011, and just hit a real one — two 0014s
in the same integration round. Codify what already happened in practice: take
the next free number in main's docs/adr/, and on collision whoever merges
second renumbers and fixes cross-refs rather than round-tripping.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_012fgjnqFCS5h5L7gZTZo3rV
PKG landed docs/adr/0014-conformance-runs-through-ctest.md (dad88fc) and this
lane's 0014-generated-binding-changes-and-versioning.md landed in the same
integration round, both as 0014. Renumbered this one to 0015 — second merger
renumbers. Updated the two cross-references (contracts/README.md rule 4,
proto-answers-daemon-m1.md P1) and the in-file header.
Also while in contracts/README.md: the file-tree comment and the "Method
surface" heading still said v1.2.0; both now v1.4.0 to match VERSION.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_012fgjnqFCS5h5L7gZTZo3rV
Rebased onto main at 1.4.0. The regenerated Dispatcher returns
HandlerResult<T> = expected<T, HandlerError{code, message, data}>
(ADR 0014); the covariant-return break on all 39 overrides is the swap
predicted in daemon/docs/proto-requests-m1.md P1.
- dispatcher.hpp/.cpp: Result<T> -> HandlerResult<T> on every override;
not_implemented() now returns HandlerError{InternalError, ...} rather
than a ParseError forwarded as -32603.
- download.get: returns -32010 TaskNotFound with data.taskId. Not a
placeholder — with no store, every id is genuinely not-found, which
is the real answer for contracts/ fixture download.get.not-found. It
becomes a store lookup when store/ is wired in.
- uds_roundtrip: the -32603-collapse guard is now a -32010 + data.taskId
assertion, the regression guard the P1 note promised.
download.add (-32011) and download.probe (-32013) stay InternalError
until they have real bodies (canonicalization / probe); they get their
fixture codes when that logic lands.
All 24 tests green; uds_roundtrip TSan-clean.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
The daemon's persistent state. SQLite in WAL mode, foreign keys on,
5 s busy timeout so a writer waits rather than SQLITE_BUSY under the
RPC loop.
- store/sqlite — RAII Db/Stmt over the C API; errors returned as
DbResult<T> (std::expected), never thrown — the RPC loop must not
unwind. transaction() helper: BEGIN / fn / COMMIT, ROLLBACK on error.
- store/migrations/0001_initial.sql — the eight tables from the brief:
settings, categories, queues, tasks, segments, rules, history,
pairings. Notable choices:
* tasks columns project onto proto TaskSummary with no computation;
requested vs effective segments/buffer split per ADR 0010/0012;
pause_reason column per ADR 0013.
* segments end_byte is NOT constrained >= 0 so a whole-file
zero-length download is one row with end_byte = -1 (ADR 0010 B3a).
* pairings stores only token_sha256 — the plaintext token is
returned once from session.pair and never persisted (CLAUDE.md §4).
* indices on tasks(state), (category_id), (queue_id, queue_position),
(created_at), (completed_at) for the "1000 tasks, download.list
under 50 ms" DoD.
* six built-in categories + a Main queue seeded.
- store/migrations — runs every embedded migration past PRAGMA
user_version, each in its own transaction, forward-only. SQL files
are embedded at build time by cmake/embed_migrations.cmake.
Test veloxd.store_migrations (ASan+UBSan and TSan clean): fresh DB ->
head, all tables present, seed rows, FK cascade (segment orphan
rejected, task delete cascades), the end_byte=-1 zero-length case,
idempotent re-run, and forward-only from every released user_version.
Also: daemon/docs/proto-requests-m1.md — P1 marked landed on lane/proto
as 1.4.0 (HandlerError/HandlerResult), to be adopted in rpc/ once that
merges to main; P2 resolved.
Not linked into the running daemon yet — the store is wired to the
dispatcher when download.add/list/get get real bodies, next.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
The scriptable client, built now rather than last: it is how the daemon
gets exercised before the GUI is pointed at it (AGENT-DAEMON.md).
- src/client — synchronous blocking RPC over the Unix socket: resolve
$XDG_RUNTIME_DIR/velox/velox.sock, connect, session.hello, one framed
request/reply per call. Distinguishes transport failure (exit 3) from
a daemon-returned error (exit 1).
- src/main — subcommands add/ls/pause/resume/rm; --json prints the raw
JSON-RPC result or error; --dir/--out/--segments on add;
--delete-file on rm. Usage errors exit 2.
ls works end to end against veloxd today (empty table). add and the
bulk verbs reach the daemon and surface its "not implemented" (-32603)
cleanly until the store lands — the plumbing is done, the commands
light up as handlers do.
Test velox.client: the real Client against an in-process UdsServer —
no-daemon path, session.hello, download.list, and a not-implemented
method surfacing as an RPC error rather than a transport error.
ASan+UBSan clean; full tree (21 tests, incl. conformance) green.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
First real code in daemon/. veloxd now listens on
$XDG_RUNTIME_DIR/velox/velox.sock (0600, SO_PEERCRED same-UID check),
frames NDJSON, and routes every method through the generated
velox::proto::dispatch(). The CLI and GUI have a server to talk to.
Modules:
- rpc/ndjson.hpp — newline-delimited framing, 8 MiB frame cap, CRLF-
tolerant, partial-tail buffering. Header-only, tested.
- rpc/event_loop — single-threaded poll(2) reactor; never blocks the
loop. stop()/wake() are async-signal-safe (eventfd).
- rpc/runtime_dir — $XDG_RUNTIME_DIR/velox resolution, 0700, owner-checked;
refuses an insecure fallback rather than using /tmp.
- rpc/uds_server — listener + non-blocking per-conn read/write with
backpressure; handles session.hello (protocol-major
check -> -32001, sessionId, transport=uds) and
session.subscribe in the server layer; routes the
rest through dispatch().
- rpc/dispatcher — VeloxDispatcher : proto::Dispatcher, all 39 methods.
download.list answers an empty table; the rest return
"not implemented" (-> -32603) until the store lands.
- main.cpp — abstract-namespace single-instance lock, signal ->
clean shutdown, socket unlinked on exit.
Tests (ASan+UBSan and TSan clean):
- veloxd.ndjson — framing edge cases
- veloxd.uds_roundtrip — real socket: hello ok / version mismatch / empty
list / -32601 / -32700 / pipelined requests, and a
guard on the -32603 collapse documented in P1.
Known gap, filed not worked around: daemon/docs/proto-requests-m1.md P1 —
the generated Dispatcher has no error channel below -32603, so handlers
cannot yet return -32010/-32011/-32013 with their data payloads. The
server layer handles -32001/-32002/-32003 around dispatch(); genuine
in-handler errors collapse to -32603 until PROTO gives handlers a real
error return. Three error fixtures are non-conformant until then.
Not in this drop: rpc/ws_server (next; needs the store for hashed pairing
tokens), store/, sched/, cli/. WS reuses this event loop.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
Records the rule 1.4.0 applied, so the next generated-binding retype cites it
instead of relitigating README rule 4's "retype → major + ADR" from scratch.
The rule: VERSION tracks the wire protocol, not any binding's API or ABI. A
change that leaves the wire byte-identical but breaks a generated binding's
source API (a C++ virtual's return type, a struct name) is a minor bump plus
a migration note — major would make session.hello refuse a client whose wire
behaviour is unchanged, which is worse than the problem. An ADR is still
required when the change encodes a design decision; "only one lane consumes
it" is not a reason to skip that, since GUI already links velox::proto and
the next such change starts with more than one consumer.
Rule 4 in contracts/README.md now points here so its "major + ADR" line
isn't read in isolation. proto-answers-daemon-m1.md's P1 writeup references
it as the durable home for the reasoning that was otherwise only in a commit
message.
Also names the nlohmann brace-init hazard the 1.4.0 work hit: json{nullptr}
is the array [null], not JSON null, so HandlerError::data is `= nullptr`. The
kind of thing a regeneration reintroduces; caught here only by an end-to-end
assertion on dispatch() output, which is called out to keep.
Docs only — no schema, VERSION, or generated-code change.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_012fgjnqFCS5h5L7gZTZo3rV
vdm/ids.hpp — TaskId, an opaque engine handle (DAEMON keeps the wire
UUID <-> TaskId map; the engine never sees the UUID).
segment/segmenter — per-download range management (docs/04 §3). Initial
lazy split; assign_slot() splits the largest remaining range when the
budget grants a slot; on_complete(may_steal) either *steals* the second
half of the largest remaining range for the same worker (slot-neutral) or
returns nullopt so the caller *yields* the slot (ADR 0011 A1); on_failed()
returns requeue only on the 3rd consecutive connection error with a mirror
present — the remaining range is orphaned and re-split. Non-resumable or
unknown-size => exactly 1 segment; never split below min_segment_bytes
(1 MiB). Resume ctor rebuilds from a persisted table (falls back to a
fresh layout if it doesn't tile [0,total)). One mutex == "the task lock";
segment fields are std::atomic and the store is a std::deque so a steal's
append never moves a worker's record.
segment/budget — the global allocator (ADR 0011). Owns exactly one
ceiling (maxActiveSegments) and min-1-before-seconds fairness: a two-pass
allocation (guarantee pass gives every wanting task 1 slot in DAEMON's
priority order, then a growth pass round-robins the rest up to each
task's effective cap = min(per_task_cap, host cap, 1 if non-resumable)),
recomputed from scratch on every edge so a live set_max_active_segments
cut naturally yields the excess lowest-priority-first, never a
mid-segment kill. DAEMON-facing surface exactly as promised in
daemon/docs/core-requests-m1.md / ADR 0011: budget(), segments_active(),
starved_tasks(), starved_since(), set_max_active_segments (drain),
set_host_segment_cap, set_task_order, on_budget_changed (a jthread
coalesces at <=4 Hz; the tasks_starved 0<->nonzero edge fires
immediately). Callbacks are copied out and run after the lock is
dropped.
Tests: segmenter split/steal/requeue/resume math + a concurrent
steal-and-advance run; budget min-1 under a tight budget, round-robin
growth, host-cap and non-resumable clamps, live-lower shedding
lowest-priority-first, starvation below the task count, starved-edge
notification, and a concurrent set_want hammer. Green under ASan/UBSan;
the steal path and the budget green under TSan.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HPPSGhiArbvQgwC2DNiURS
DAEMON's daemon/docs/proto-requests-m1.md P1: velox::proto::Dispatcher's
on_* methods returned Result<T> = expected<T, ParseError>, and dispatch()
mapped every handler error to -32603 InternalError. A handler had no way to
return -32010 (download.get not-found), -32011 (download.add invalid-path)
or -32013 (probe-failed) with their data payloads -- three error fixtures a
conformant server must satisfy were unreachable, blocking DAEMON's
"conformance as a server" M1 DoD.
Two error channels now, kept separate on purpose:
- parse: Result<T> / ParseError -- dispatch() failing to turn the wire into
typed params. Always -32602, always structural.
- handler: HandlerResult<T> / HandlerError -- a handler deciding the request
can't be fulfilled. Carries any ErrorCode + message + free-form data.
struct HandlerError {
ErrorCode code{ErrorCode::InternalError}; // bare {} is a valid -32603
std::string message;
nlohmann::json data = nullptr; // straight into the error's data
};
template <class T> using HandlerResult = std::expected<T, HandlerError>;
dispatch()'s handler branch is now
make_error(id, r.error().code, r.error().message, r.error().data)
instead of a hard-coded InternalError. -32001/-32002/-32003 stay the server
layer's to raise around dispatch(), as DAEMON already does.
Verified end to end against the real dispatch() path: a handler returning
TaskNotFound/InvalidPath/ProbeFailed produces -32010/-32011/-32013 with the
data object intact, and a bare HandlerError{} still yields a clean -32603
with no data field. The `= nullptr` on the member (not `{nullptr}`) matters:
brace-init of nlohmann::json from nullptr is the array [null], not JSON null.
FixtureDispatcher regenerated to HandlerResult; conformance_main.cpp only
inspects dispatch()'s JSON and needed no change. TS side is untouched beyond
the version string -- no server Dispatcher is generated there.
P2 also handled: session.hello.version-mismatch's data.expected was a stale
"1.0.0"; now $any, with a note that the error-fixture compare is on `code`
only so a server echoing kProtocolVersion there is fine.
Version: minor, 1.3.0 -> 1.4.0. Wire is byte-identical (no schema, fixture,
or OpenRPC change) but every Dispatcher implementer must swap Result ->
HandlerResult on regen, and the bump is how lanes are told to. Not an ADR:
one lane consumes this binding, it's the one that asked, and the shape is
the one they proposed. Answered in contracts/proto-answers-daemon-m1.md.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_012fgjnqFCS5h5L7gZTZo3rV
R3: the three GUI DoD gates (10k rows at 60fps, flat RSS over 10 min,
--slow/--flaky/--drop-connection recovery) have nowhere to run. GUI owns the
harness, PKG/QA owns the job. tests/integration/README.md records the wiring
contract — driver invocation, exit-code and --json semantics — and carries
the pre-drafted per-PR and nightly job stanzas with TODO(GUI) markers for the
harness path. Wire for real when GUI files the follow-up.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0143aKiohmDiyefJBwHDJJqw
PROTO wired the suite into ctest (label "conformance": the end-to-end
`conformance` test that shells to run.sh, plus the native `conformance_cpp`);
the CI job called run.sh directly. Two entry points, and the required M0 gate
exercised only one of them, so the ctest registration could rot.
The conformance job now configures, builds velox_conformance_cpp, and runs
`ctest --preset dev -L conformance`. The dev test preset's
noTestsAction: error is the rot guard: an empty label match exits non-zero
instead of the old silent pass. build/sanitizers exclude the heavy e2e test
with -E '^conformance$' (the dedicated job owns that run; conformance_cpp
still runs under every sanitizer). ADR 0014 records the decision.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0143aKiohmDiyefJBwHDJJqw
libFuzzer writes crash-* / oom-* / leak-* / timeout-* into CWD on a find and
each holds the crashing input verbatim. None were ignored; CORE caught two by
hand before committing. Prevention, not cleanup.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0143aKiohmDiyefJBwHDJJqw
--check verified outcomes — binaries on PATH, pkg-config modules — on a box
that already installed everything. It never looked at the APT_* names, which
is the only part that breaks on a clean machine of the wrong release, as R1
just showed. Add a loop over the assembled PKGS array that fails on any name
with no installable candidate (apt-cache policy; no root, no network).
All-missing is treated as stale lists (warn), not 36 bad names.
The bootstrap-script job runs on a 24.04 runner, where the bad name still
resolves, so name validation there checks the wrong archive. Add
bootstrap-script-2604: a real --with-clang install in an ubuntu:26.04
container — the release the project ships on, and the first time the
fuzz-toolchain half of bootstrap is exercised anywhere (CORE had been running
clang++-21 directly). Also pass --with-clang/--packaging to the 24.04 --check
so the optional and M6 names can't rot unnoticed. BRANCH_PROTECTION.md gains
the 2604 row and the stale "when X merges" rows are corrected to "now".
gui/docs/pkg-qa-requests-m1.md R2 + the --with-clang note.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0143aKiohmDiyefJBwHDJJqw
`apt-cache policy libqt6svg6-dev` is "Candidate: (none)" on 26.04; the package
that carries the Svg headers and Qt6SvgConfig.cmake is qt6-svg-dev. Since
gui/CMakeLists.txt landed on main the root build's
find_package(Qt6 ... Svg REQUIRED) is live, so a clean 26.04 box could not
configure the project at all. Same name in README.md and AGENT-PKG-QA.md.
Reported in gui/docs/pkg-qa-requests-m1.md R1.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0143aKiohmDiyefJBwHDJJqw
core/docs/proto-requests-conformance-cmake.md: the runner's header comment
said it links libveloxproto, but it actually compiled
core/generated/velox_proto.cpp straight into the executable and found
nlohmann_json itself — the only option while ADR 0009's target didn't exist.
It exists now on main as velox::proto (core/CMakeLists.txt, PUBLIC generated
include dir, PUBLIC nlohmann_json).
if(TARGET velox::proto): link it. else: fall back to compiling the generated
.cpp directly, for a configure with no core/ in the tree. Both paths verified
— full tree links libveloxproto.a (compiled once, by veloxproto's own
target); with core/CMakeLists.txt hidden the fallback compiles the .cpp and
finds nlohmann itself. conformance_cpp passes either way.
Beyond tidiness: once GUI links velox::proto too, the conformance runner
linking the same target is what guarantees the suite and the clients exercise
byte-identical generated code, rather than two compiles of one .cpp under two
warning configs — the exact skew a conformance suite exists to catch.
run.sh's own direct g++ compile is unaffected and stays independent by
design; this only changes the ctest-driven path CI and lanes use.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_012fgjnqFCS5h5L7gZTZo3rV
samiandClaude Sonnet 5