139 Commits
Author SHA1 Message Date
sami 3e553f08c0 merge: lane/pkg-qa — GUI DoD gates wired, conformance required
CI / clang-format (push) Waiting to run
CI / testserver (push) Waiting to run
CI / bootstrap-script (push) Waiting to run
CI / bootstrap-script-2604 (push) Waiting to run
CI / extension-lint (push) Waiting to run
CI / build (clang) (push) Waiting to run
CI / build (gcc) (push) Waiting to run
CI / sanitizers (dev) (push) Waiting to run
CI / sanitizers (tsan) (push) Waiting to run
CI / clang-tidy (push) Waiting to run
CI / conformance (push) Waiting to run
CI / nightly-integration (push) Waiting to run
CI / gui-dod (push) Waiting to run
CI / gui-dod-nightly (push) Waiting to run
2026-09-12 22:05:06 +04:00
samiandClaude Sonnet 5 1d359af5a3 pkg: wire GUI's DoD harness into CI, mark live-veloxd conformance required
gui-dod (per-PR: scroll-60fps + unhappy-path) and gui-dod-nightly (rss-flat,
schedule/workflow_dispatch) are live in ci.yml, driving GUI's newly-landed
gui/tests/dod/run.sh + gui-dod-harness. No Xvfb step: run.sh already runs
QT_QPA_PLATFORM=offscreen itself.

Each gate forced red once before being trusted (tests/integration/README.md
has the transcripts): VELOX_DOD_FRAME_BUDGET_MS=0.01 for scroll-60fps,
VELOX_DOD_RSS_SLACK_KIB=-999999999 for rss-flat, and — since run.sh always
starts a working mockd — a direct gui-dod-harness invocation against an
unreachable socket for unhappy-path, which hit the harness's own 75s
watchdog exactly as documented.

Recorded GUI's live finding (gui/docs/proto-requests-m1.md) that mockd
--drop-connection is a no-op over the UDS transport, so unhappy-path's
drop-connection phase can't yet exercise a real drop — coordinating with
PROTO on the fix rather than working around it locally. gui-dod stays
required regardless: its other two phases and the crash/hang/watchdog paths
still catch real regressions.

Added gui-dod to BRANCH_PROTECTION.md's required-checks table.

ADR 0019: the live-veloxd conformance runner (run.sh step 3b, already
unconditional inside the already-required conformance job) stays required
as PROTO's xfail list shrinks (18 entries now, down from 34; 57/57 fixtures
passing on main). No CI change needed — it was already inside a required
check; this records the decision not to carve out an exception for it.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01RBPR7iM3YPyxrjWsVtZDPJ
2026-09-12 22:04:09 +04:00
sami 75536304bb merge: lane/proto — conformance stops downloading real files 2026-09-12 21:44:55 +04:00
sami 27865888be merge: lane/gui 2026-09-12 21:40:31 +04:00
sami 939a19b7ea merge: lane/core 2026-09-12 21:40:31 +04:00
samiandClaude Sonnet 5 79c29b47e8 core: finish the hostile-mode matrix -- 8 remaining end-to-end cases
Was 7/16 of tools/testserver/README.md's mode table covered by
engine_test.cpp. Adds the rest:

- engine_expiring_signed_url_recovers_via_refresh_url: an expired signed
  URL 403s, the engine asks (paused, decision_calls >= 1) rather than
  failing terminally, and DownloadHandle::refresh_url() with a freshly
  signed URL completes it -- exercises both do_refresh_url() fixes and
  the probe-level referrer retry's second-403 path from the previous
  commit.
- engine_403_without_referer_retries_with_origin: no spec.referrer set,
  the automatic single retry (previous commit) recovers with zero
  decisions asked.
- engine_redirect_chain_follows_to_completion: 5 hops of a plain 302.
  No core-side change needed -- documents that CURLOPT_FOLLOWLOCATION/
  MAXREDIRS (already on, RequestOptions::follow_redirects) cover both
  the probe's and every worker's own request, not just one of the two.
- engine_slow_loris_stall_timeout_fires: proves curl's stall detector
  (CURLOPT_LOW_SPEED_LIMIT/_TIME, download_task.cpp's hardcoded 1024 B/s
  for 30s) actually fires rather than hanging. Needed a real fix, not
  just a test: every other test in this file relies on TestServer's
  short 1s loris dribble to keep runtime down, but 1s of trickle
  followed by full-speed streaming never accumulates curl's required 30
  CONSECUTIVE seconds under the floor, so it would never actually abort
  -- a test built on the default dribble would pass by the download
  merely finishing a bit late, not by observing the stall timeout fire.
  testserver_fixture.hpp's TestServer gained an explicit-loris-seconds
  constructor (default ctor unchanged, still 1s) so this one test can
  ask for a dribble (40s) that genuinely outlasts the threshold.
- engine_401_digest_then_provide_auth_completes: same shape as the
  existing 401-basic test: http_client.cpp already asks libcurl for
  CURLAUTH_ANY regardless of net::AuthScheme, so this needed no core
  change -- it passed on the first run and is here to prove that's true
  end-to-end, not just at the http_client unit level.
- engine_chunked_no_length_completes_single_segment: Transfer-Encoding:
  chunked, no Content-Length anywhere (including HEAD). No core change
  needed -- takes the same size-agnostic "unknown size, one plain-GET
  segment" path as the existing no-range test.
- utf8/legacy-content-disposition: already covered end-to-end by
  probe_reads_utf8_content_disposition and
  probe_reads_legacy_content_disposition in probe_test.cpp (probe-level,
  as these modes only affect the initial request) -- verified passing,
  no new test needed.

All 20 engine_test.cpp cases and all 10 probe_test.cpp cases pass. Every
testserver.py spawned while writing and running this was reaped by
TestServer's destructor; verified no stragglers with `ps aux` after each
run.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Q3QrF7rCt21bkAjt9BCDFQ
2026-09-12 21:34:24 +04:00
samiandClaude Sonnet 5 8e7d14ba7e core: retry once with the original referrer on a 403, at probe and worker
docs/04-engine-design.md §7's failure policy table has said "403 after
redirect: retry once with the original referrer; many CDNs require it"
since it was written, and Error::forbidden's own enum comment says the
same -- but grepping download_task.cpp and http_client.cpp for 403 turned
up nothing. It was never built.

Implemented at both points a 403 can surface:

- The probe (net::Prober, a separate request path from segment workers):
  on_probe_result() now retries once via restart_probe(false), with
  effective_referrer set to the download URL's own origin (origin_of(),
  via net::split_url()), when the failure is Error::forbidden and this is
  the first retry. A second 403 asks rather than fails outright --
  auto_pause_locked(..., false, true), the same "ask, don't just fail"
  path 416/etag-mismatch already use -- specifically so DownloadHandle::
  refresh_url() stays usable afterward (its own contract requires a
  non-terminal task); this is what makes the expiring-signed-url mode's
  README-documented refresh_url() recovery actually reachable.

- Each segment worker (SegWorker::forbidden, set in seg_head() on a 403
  HEAD): the same one-shot referrer retry via retry_worker(), landing on
  auto_pause_locked() on a second 403 for the same reason.

Both paths route the retry's Referer through a new effective_referrer
field rather than spec.referrer directly, since the origin-retry must not
overwrite what the caller actually asked for -- start_worker_locked() and
restart_probe() were switched to send effective_referrer instead.

do_refresh_url() had two latent bugs surfaced by actually exercising the
expiring-signed-url recovery path end-to-end:

1. It unconditionally proceeded to resume even when the refresh probe
   itself failed -- a bad refresh URL would silently un-pause a task with
   nothing behind it. Now returns (stays paused) on !r.has_value().
2. It only handled "already probed once, just refreshing a few fields" --
   for a task whose first-ever probe never succeeded (every hostile mode
   this commit adds a test for that pauses at the initial probe, not
   mid-download), s->registered was never true, so the existing
   `if (s->registered) set_want()` never fired and nothing happened. Now
   detects !s->have_probe and calls finish_probe_locked() directly, the
   actual first-time registration/segmenter-construction path.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Q3QrF7rCt21bkAjt9BCDFQ
2026-09-12 21:34:24 +04:00
samiandClaude Sonnet 5 c2eef96175 gui: floating drop target, clipboard global shortcut, theming, UI watchdog
Continues the build order past Options/Scheduler/Speed Limiter/Batch/
Grabber/tray.

- DropTargetWidget: frameless always-on-top drop target (docs/03-gui-spec.md
  §5), position persisted, accepts a dropped http(s) URL or link text and
  opens FileInfoDialog directly (skipping Add URL, since the URL is already
  known). Shown/hidden from general.showDropTarget, live via
  event.settings.changed, same pattern MainWindow already used for
  general.minimizeToTray.
- Clipboard, explicit path #2 (docs/06-risks-and-spikes.md R2):
  GlobalShortcut wraps org.freedesktop.portal.GlobalShortcuts
  (CreateSession -> BindShortcuts -> Activated), triggering the same Add URL
  flow. Guarded end-to-end on `if(TARGET Qt6::DBus)` / VELOX_GUI_HAVE_DBUS
  so a build without the component degrades to "feature skipped," not
  broken (gui/docs/pkg-qa-requests-m1.md R4). Best-effort by design per the
  risk doc: fails silent, never advertised.

  Verified live against the real portal (a real Wayland session, not just
  offscreen): `CreateSession` refuses every caller with "An app id is
  required" — reproduced identically via a bare `busctl` call with no Qt
  involved at all, so this is the portal requiring a sandboxed caller
  identity, not something fixable from an unconfined process. Recorded as
  a partial Spike S2 answer in docs/06-risks-and-spikes.md: this explicit
  path likely doesn't work for Velox as a traditionally-packaged app on
  stock GNOME, only if/when it ships confined. Also fixed a real leak this
  verification caught: QDBusInterface's introspection cache reads as a
  LeakSanitizer leak the first time anything touches D-Bus (tst_rtl went
  red under ASan) — switched to QDBusMessage::createMethodCall, which
  needs no introspection.
- Theming (docs/03-gui-spec.md §7): gui/resources/qss/{idm-like,dark}.qss,
  each with a documented palette block up top (QSS itself has no variable
  syntax), applied by ThemeManager and kept live via
  QStyleHints::colorSchemeChanged. util/Theme.hpp gives the handful of
  inline C++ styles (status dot, offline banner, the eleven identical
  error-label styles across dialogs) named constants instead of a twelfth
  copy of the same hex.
- UiThreadWatchdog: the M1 DoD's 200 ms debug-build watchdog. A background
  std::thread pings the UI thread every 50 ms via a queued invokeMethod and
  warns once (not per-poll) if a ping goes unanswered past 200 ms; no
  QThread, no Qt event loop of its own, so the watchdog itself can never be
  what blocks the thread it watches. No-op in a release build. Proven both
  ways in tst_uithreadwatchdog: fires on a genuinely blocked UI thread
  (synchronous sleep, no processEvents) and stays silent on a responsive
  one.

Full non-conformance suite (55 tests across every lane, `ctest -LE
conformance`) passes clean at this point, including the whole gui label
under ASan+UBSan.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01NSCdCWFXBTSBBK3MzWtJiC
2026-09-12 21:30:13 +04:00
samiandClaude Sonnet 5 1fd2e0a0db gui: Options — Capture tab, saveTo.allowedRoots, lock coverage to the schema
Options was missing 8 of the 43 real settings.* keys: capture.enabled,
monitoredExtensions, monitoredMimeTypes, minSizeBytes, excludedHosts,
bypassModifier, autoStartTypes, and saveTo.allowedRoots. The capture.* keys
were dropped in an earlier pass on the mistaken read that the spec's "File
Types" tab meant per-category extension lists (which do live on Category,
not settings.*) — they're real settings.* keys for a real daemon feature
(the extension's auto-capture policy), so the tab exists now, named
"Capture" to match what it actually configures rather than the spec's
label.

New tst_optionsdialog case (allKeysMatchesTheSchemaExactly) loads
Settings.schema.json itself at test time and diffs its property set against
OptionsDialog::allKeys() — this drifted silently once already, so the
regression is now a build-time gate an unused import or a future key
addition would trip, not something that needs re-discovering by hand again.

Verified against a real veloxd (not just mockd): settings.get across all
43 keys, a settings.set/get round trip on a scalar (connection.timeoutSec)
and on array-valued keys in the shapes OptionsDialog::currentValues()
actually produces (capture.monitoredExtensions, proxy.bypassHosts,
saveTo.allowedRoots), and event.settings.changed fanning out to a second
subscribed client — all round-tripped and restored to their original
values afterward. The real OptionsDialog widget also loads and renders
correctly against that same daemon's live defaults with no crash under
ASan+UBSan.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01NSCdCWFXBTSBBK3MzWtJiC
2026-09-12 21:29:19 +04:00
samiandClaude Sonnet 5 755d85964e gui: M1 DoD harness — scroll-60fps, rss-flat, unhappy-path
gui/docs/pkg-qa-requests-m1.md R3, filed by the previous session: three GUI
M1 DoD items (10k rows at 60 fps, flat RSS over 10 minutes,
--slow/--flaky/--drop-connection recovery) had nowhere to run in CI. This is
the harness — `gui/tests/dod/run.sh <gate> [--json <path>]`, exactly the
path/invocation contract tests/integration/README.md already specified —
plus `gui/tests/dod/dod_harness.cpp`, the Qt/RpcClient-driven binary that
actually runs each gate against a real mockd run.sh starts and tears down
itself.

- scroll-60fps: an eased scripted scroll over the whole loaded table,
  timing each step's synchronous repaint; p99 against a 16.6 ms budget
  (auto-scaled 4x under a sanitized build — ASan/UBSan overhead, not a
  loosened bar, see the harness's isSanitizedBuild()).
- rss-flat: samples this process's own VmRSS at 1 Hz across the run,
  discards a warm-up window, checks post-warm-up growth against a stated
  20 MiB slack.
- unhappy-path: three phases (slow/flaky/drop-connection), each its own
  mockd instance; passes when the client reaches and holds Connected with
  no crash or hang. A watchdog (the harness's own QTimer, backstopped by
  run.sh's external `timeout`) turns a genuine hang into a bounded non-zero
  exit rather than needing the CI caller to timeout(1) around it.

Every gate honours the exit-code and --json contract PKG/QA's pre-drafted
CI job expects unchanged (one addition needed: the build step must also
build the `gui-dod-harness` target, noted in the R3 update). No leaked mockd
processes on any exit path (`trap cleanup EXIT INT TERM`); no writes outside
a tempdir except the caller's own --json path.

Verified live end-to-end (not just unit-level): all three gates run against
a real mockd under the exact `ASAN_OPTIONS=detect_leaks=1:halt_on_error=1`
`.github/workflows/ci.yml`'s sanitizers job already sets, all pass, and
scroll-60fps was forced red once on purpose
(VELOX_DOD_FRAME_BUDGET_MS=1) to prove the fail path and exit code actually
work. Building this is also what surfaced the two RpcClient bugs fixed in
the previous commit, and one real gap in mockd itself — --drop-connection
never worked over the Unix socket transport (only WebSocket) — filed as
gui/docs/proto-requests-m1.md since tools/mockd is PROTO's file.

gui/docs/pkg-qa-requests-m1.md R3 and R4 (an unrelated, non-blocking Qt6::DBus
CMake hygiene note filed while wiring the clipboard global-shortcut path)
are updated with the concrete findings above.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01NSCdCWFXBTSBBK3MzWtJiC
2026-09-12 21:28:36 +04:00
samiandClaude Sonnet 5 c6f864ea30 gui: fix RpcClient double-free on stop() and the 1000-row list cap
Both found live while building gui/tests/dod's DoD harness, not from reading
the code — see that commit for how.

RpcClient::stop() left conn_ dangling after joining the worker thread: the
thread's own finish() flushes the DeferredDelete stop()'s
connect(&thread_, &QThread::finished, conn_, &QObject::deleteLater) already
posted, so conn_ is gone by the time stop() returns, but nothing cleared the
pointer. Any caller that calls stop() and later lets the client destruct
(the harness's own client.stop() at shutdown; also plain, correct API usage)
hit a double-free in the destructor's leftover `delete conn_`. Caught by
ASan on the very first run that actually exercised the stop-then-destroy
path.

requestInitialList() also called download.list with a hardcoded
`{"limit": 1000}`, silently capping the table at 1000 rows no matter how
many the daemon actually has — download.list.schema.json's own description
says "the GUI pages", not "the GUI takes it all in one call". The
scroll-60fps DoD gate refused to run against mockd --tasks 10000 rather
than "pass" against a 1000-row table, which is what surfaced it.
requestInitialList() now pages (5000 per call, the schema's own max) until
`total` is satisfied, then resets the model once with everything.

Separately: RpcConnection's session.subscribe list never included
event.settings.changed or event.grabber.progress, even though RpcClient has
carried signals for both since the Options/Grabber work — session.subscribe
"replaces the previous selection" and "nothing is delivered until this is
called", so both events were being silently dropped by any real daemon that
enforces the subscription (mockd does; verified live with a second
subscribed client actually receiving event.settings.changed after this
fix, round-tripped through a real veloxd's settings.set). GrabberWizard's
5 s poll fallback is exactly why this went unnoticed until now — it covered
for the missing push the whole time.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01NSCdCWFXBTSBBK3MzWtJiC
2026-09-12 21:27:28 +04:00
sami c1c5c82f8b merge: lane/ext — real-veloxd verification, session.subscribe fix 2026-09-12 17:20:19 +04:00
samiandClaude Sonnet 5 4e177ec809 proto: stop conformance from downloading real files, ADR the null-clearing gap
1. download.add.json had startMode "now" against a real, large (~6 GB)
   Ubuntu ISO with saveDir hardcoded to /home/sami/Downloads/Programs.
   Against a real veloxd (tests/conformance/run.sh) that's a real
   download into the real user's real home, every single run — it had
   already happened twice. startMode -> "later" (exercises the add path,
   hands nothing to the engine) and saveDir is dropped entirely (resolves
   to saveTo.defaultDir instead, checked against allowedRoots the same
   way). Documented the rule this fixture was breaking in
   contracts/fixtures/README.md so it doesn't happen a third time.

   Auditing the rest for the same shape (now real: capture.offer, D7)
   found a second, subtler instance: capture.offer.take.json's "take"
   admits a real, immediately-started task the same way download.add
   does, and the "Programs" category's saveDir is a migration-seeded
   builtin (~/Downloads/Programs) that no isolated test setup can
   redirect -- so even after pointing the URL at example.org (RFC 2606),
   a real ~6 GB sparse .veloxpart still landed in the real home on the
   declared Content-Length alone. Shrunk to a plausible-but-small 5 MiB.
   Also scoped to "transport": "uds" -- a real "take" persists an active
   task, so replaying the same fixture again on the second live transport
   against the same shared daemon was hitting capture.offer's own
   dedupe-by-URL and failing on a missing taskId, not a bug.
   download.add's other real-URL siblings (errors/*.invalid-path,
   *.invalid-params, *.disk-full) all fail before admission or are
   requires-gated; left alone.

2. ADR 0018: DAEMON can set a nullable field through download.update /
   settings.set but never clear it back to null, because the generated
   C++ parser collapses "absent" and "explicit null" to the same
   std::nullopt for every optional field (contracts/codegen/gen_cpp.py,
   on purpose, and correct for create-style params -- just wrong for
   patch-style ones, which is the only place the schema documents
   "explicit null clears"). Decision: an opt-in x-clearable schema
   annotation makes just those fields std::optional<std::optional<T>> in
   C++ (TS already round-trips this natively); not a blanket rule
   (would retype response fields like TaskSummary.effectiveUrl that have
   no clear-vs-absent distinction to make), not an explicit clear-list
   field (would redesign a wire contract DAEMON already built against
   just to route around a generator gap). Recorded, not implemented here
   -- that's its own PROTO PR (schema annotations + gen_cpp.py + gen_ts.py
   + regeneration + a minor VERSION bump per ADR 0015), not bundled into
   a fixture-safety pass. Left a pointer to the ADR at the generator
   comment it concerns.

3. Re-verified every xfail entry against current deferrals.md rather
   than trust the reasons already on file: D7/D8 (capture.offer/
   getRules), D3d/e/f/g/h/i (rules, queue.reorder, schedule, limiter,
   download.update/refreshUrl) and D9 (settings) have all closed since
   the list was last pruned, so most of it was stale. Removed everything
   that now cleanly passes; kept and re-reasoned everything that doesn't:
   - errors/download.provideAuth.not-found.json stays, as asked: real
     bug, on_download_provideAuth never checks the task exists.
   - category.list.json (mimeTypes -- documented D3a gap), schedule.set.json
     (nextRunAt -- documented D3f gap): unchanged in substance, reason
     text was already accurate.
   - download.probe/get/list/update.json, session.hello.json,
     queue.start/reorder.json, category.remove.json: not bugs -- each
     golden depicts a richer lifecycle/config state (a probed download,
     real queue or category membership, media/grabber capabilities) than
     this harness's fresh, never-started bound tasks and empty isolated
     DB can produce.
   - limiter.get.json: real fixture bug, not a daemon one -- applyToRunning
     is a write-only instruction on limiter.set, on_limiter_get never
     returns it; the golden shouldn't have had it either. Fixed the
     fixture and tools/mockd's own limiter.get, which had the same field
     hardcoded into its in-memory state independent of the fixture file.
   - grabber.*/media.*: still genuinely stub (M4 territory).
   Only remaining unexpected-pass surfaced while re-verifying
   (errors/capture.offer.ignore.json, always "take" instead of "ignore")
   traced to capture.minSizeBytes defaulting to 0 on a fresh daemon,
   making its below-minimum-size scenario unreachable -- not a bug, so
   raised the setting in run.sh's isolated seeding instead of xfailing it.

ctest -L conformance: green, 100% (2/2), ~87s.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01SFeUKLbdHizrJjLBeK7ffz
2026-09-12 16:56:35 +04:00
samiandClaude Sonnet 5 b5c6e1f47d ext: verify against a real veloxd, not just FakeDaemon
main.cpp's single-instance lock is now keyed to the runtime directory
instead of the euid, so an isolated XDG_RUNTIME_DIR/XDG_DATA_HOME/
XDG_CONFIG_HOME/HOME gets its own veloxd alongside anyone else's.
tests/live/real-veloxd.test.ts spawns one (VELOX_PAIR_AUTO=1 standing
in for the GUI's Allow click) plus tools/testserver/testserver.py, and
exercises the real WebSocketTransport end to end: session.hello,
pairing, the token surviving a reconnect, a wrong token rejected and
then rate-limiting the next pairing attempt, download.add reaching a
real running task, the real capture.offer path (rules table + category
folder + dedupe) taking a monitored download and ignoring its own
duplicate, and fail-open proven by SIGKILLing the daemon mid-offer —
the hook still resolves to {} inside its 750ms budget. A last case
proves fail-open at the transport layer too: a call against a closed
socket rejects instead of hanging.

Guarded behind  so it skips itself (with a clear message)
when no daemon binary is around — npm test and CI are unaffected;
run it with VELOXD_BIN=/path/to/veloxd npx vitest run tests/live.

Real-daemon testing found one actual bug, fixed here: background/
index.ts never called session.subscribe, so event.task.progress and
friends never reached this connection at all — FakeDaemon's tests
never caught it because FakeDaemon broadcasts regardless of
subscription state. Now subscribed to the full event set on every
connect (first connect and every reconnect), which is what the popup's
live-progress path actually depends on against a real daemon.

Per instructions: ctest -L conformance was not run (pending PROTO's
fixture fix).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Ed8KEmAW48v4YHdxLtqsMB
2026-09-12 16:43:55 +04:00
sami 0468b0176a merge: lane/daemon 2026-09-12 16:27:29 +04:00
sami d8b7c128be merge: lane/proto 2026-09-12 16:27:29 +04:00
samiandClaude Sonnet 5 4f6c0cc9d2 daemon: D3's remainder — rules.*, queue.reorder, schedule.*, limiter.*, download.update/refreshUrl
rules.list/rules.upsert: new store/rules.{hpp,cpp} (list in priority order; apply()
upserts+removes atomically, generating an id when absent, per the schema's own "never
leaves the table in a half-valid state"). Found and fixed along the way: migration
0001's rules table had no column for Rule.name at all — every rules.list/.upsert call
failed outright ("no such column: name"), unit tests included, since :memory: migrates
through the same path. Migration 0004 adds it.

queue.reorder: new store::Queues::reorder — taskIds must be an exact permutation of
the queue's current membership (compared as sorted sets) or nothing is written and
-32602 names the queue; a valid permutation rewrites every member's queue_position in
one transaction.

schedule.get/schedule.set: a thin wrapper over the queues.schedule column (already
read since D3b, never independently settable). nextRunAt is deliberately left unset —
computing it needs the same local-time, DST-aware window logic
sched/schedule_window.hpp's window_open() only has half of; called out rather than
approximated, and the field is optional.

limiter.get/limiter.set: backed by the same downloads.speedLimitEnabled/
downloads.speedLimitBps settings keys D9 already wired — one bag of truth, not two.
The new part is reaching the engine: EnginePort/TaskActionPort gain
set_global_speed_limit(bps) (0 = unlimited, TokenBucket's own convention), wired to
Engine::rate_limiter().set_global_limit(). Pushed live on every limiter.set *and* on
Scheduler::reload_config() so a limit from a previous run isn't silently unlimited
again after a restart. applyToRunning is accepted but has no lever to pull
differently — a single shared global bucket has no "next task only" variant. Also
found, not chased further: the schema's "globalBps:0 with enabled:true means 'stop
everything'" is the opposite of what TokenBucket does with rate_bps==0 (unlimited) —
a real discrepancy, but the schema says the GUI must not offer that combination.

download.update: "moving saveDir or filename moves the file on disk in the same
operation" — resolved and root-checked like download.add's destination, then the
.veloxpart/.veloxpart.meta pair (or the finished file, if complete) is moved via
rename, falling back to copy+remove across filesystems, only when the resolved
location actually differs. categoryId/queueId(appended to the new queue's run
order)/description/segments/bufferBytes/checksum apply through new
store::Tasks::apply_update.

download.refreshUrl: same async server-layer special-case as download.probe (a real
network round trip, same 30s deadline). Re-probes, flags contentChanged only when
size or validator are both known and actually differ, persists the new URL and probe
result, and swaps the URL on a live engine handle via a newly-widened
EnginePort::refresh_url (now takes headers too, matching DownloadHandle's real
signature — the seam had silently dropped them).

Found and documented, not fixed: the generated parser collapses "field absent" and
"field explicitly null" to the same nullopt for every optional<T> patch field
(DownloadUpdateParamsPatch, Settings) — both schemas document "an explicit null
clears the field" but neither handler can act on it because the wire distinction is
already gone by the time either sees the parsed struct. A generator-level gap
(PROTO's), not something to hand-route around locally.

Verified against real veloxd + tools/testserver: rules create/list, limiter.set
takes effect and reads back, schedule.set/get round-trips, queue.reorder against real
membership (and rejects a non-permutation), download.update renames+recategorizes a
task, download.refreshUrl swaps a paused task's URL and reports contentChanged
correctly. Full ctest: 55/55 (excluding the pre-existing, unrelated conformance
failure noted two commits back).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GRDjHGgpYmMoPE2UFbe7pP
2026-09-12 14:26:18 +04:00
samiandClaude Sonnet 5 e30d994d74 proto: fix conformance run.sh flakiness, prune the veloxd xfail list
Two run.sh fixes plus the xfail prune, all requested together:

1. VELOX_PAIR_AUTO=1 for the isolated veloxd. Pairing is the D1 dev stub
   (EnvAutoApprover) and denies without it, so session.pair never issued
   a token and the WS half of the veloxd step could never even connect.

2. WS_PORT was hardcoded to 52080 with no free-port search, so one leaked
   mockd made every future run fail EADDRINUSE. free_port() binds :0 and
   asks the kernel instead. The EXIT trap's stop() used `pkill -P "$pid"`,
   which only reaps direct children — tsx's actual listener is often a
   grandchild, which that missed and left holding the port. Every server
   (mockd, slow mockd, veloxd) now launches under `setsid`, making it the
   leader of its own process group, so stop() does `kill -TERM -"$pid"`
   (a process-group kill) and reaches everything it spawned in one shot.

3. Pruned the xfail list now that D2, D4b and most of D3 have landed.

Pruning surfaced two more bugs than expected, both in the test harness
itself, not veloxd — worth recording since they were indistinguishable
from real daemon hangs until isolated:

- errors/session.hello.version-mismatch.json documents that the *server*
  closes the connection after replying (correct, intended behavior). The
  harness replays every fixture on one shared connection per transport,
  so once this fixture ran, every later UDS fixture sent into the dead
  socket and just sat there until its own timeout — including ones still
  on the xfail list, which applyXfail waved through as "expected -32603"
  regardless of the real reason. Fixed with a `closesConnection` fixture
  flag: replay() reconnects (fresh session.hello) right after such a
  fixture instead of leaving the rest of the run to time out one by one.
  This is what was actually behind queue.*/session.*/download.remove
  appearing to hang — none of them do; verified individually and via a
  raw probe script before finding the real cause.
- category.remove.json (deletes the "firmware" category) sorted before
  category.upsert.json (creates it) alphabetically, so it was failing
  -32602 "no such category" against a fresh DB — never a daemon bug.
  Added it to DESTRUCTIVE so it now replays after every other fixture.

Also fixed while verifying "confirm each really passes": download.addBatch.json's
`defaults.categoryId` was "compressed", a category nothing ever creates —
real veloxd correctly enforces the FK on tasks.category_id, so all three
batch items failed instead of the two expected. Changed to "programs" (a
migration-seeded builtin).

Of the 15 fixtures named for pruning, 10 turned out to cleanly pass and
are gone from the list entirely: download.pause/resume/start/cancel,
download.remove, download.addBatch, queue.upsert/stop, download.probe's
success path (D2, including errors/download.probe.probe-failed.json),
and category.upsert. Two do NOT cleanly pass and are kept, with reasons
rewritten to match what's actually happening now instead of the stale D3
text: download.probe.json (see below) and errors/download.provideAuth.not-found.json,
a real bug — on_download_provideAuth never checks the task exists, so an
unknown taskId gets a normal `{ok:false}` result instead of -32010.

Five more fixtures newly needed xfail entries to reach green, none of
them stubs:
- category.list.json — documented gap (deferrals.md's D3a note): the
  categories table has no mimeTypes/sortOrder columns.
- download.probe.json, download.get.json, download.list.json,
  session.hello.json — not bugs. Each golden depicts a richer lifecycle
  state (a probed/in-progress download, a daemon with media/grabber/
  Secret Service implemented) than this harness's bound tasks, which are
  always fresh and never started, can produce. Optional/omit-if-absent
  fields (effectiveUrl, requiresAuth, capabilities) are correctly absent;
  the mismatch is against the golden's illustrative values, not the
  contract.
- queue.start.json, category.remove.json — same class: startedTaskIds /
  reassignedTaskIds are correctly empty because this run's queue/category
  have no real membership.

`ctest -L conformance` is green: 100% (2/2), 81.7s (down from ~240s now
that pairing and the port/reconnect fixes remove the retries and the
5-10s timeouts the connection-death bug was producing).

One thing NOT fixed here, flagged for a follow-up decision rather than
touched mid-task: download.add.json's fixture is `startMode: "now"`
against a real, large (~6GB) Ubuntu ISO on the real internet, with
saveDir hardcoded to /home/sami/Downloads/Programs. Every run against a
real veloxd writes a real multi-GB file into that path — confirmed by
running this repeatedly during verification. Isolating the daemon's XDG
dirs doesn't isolate this. Worth its own change (startMode: "later"
would still exercise the add path without the transfer) but out of scope
for a fixture I wasn't asked to touch beyond what blocked this task.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01SFeUKLbdHizrJjLBeK7ffz
2026-09-12 14:04:18 +04:00
samiandClaude Sonnet 5 6632b75099 daemon: capture.offer for real (D7/D8) — rules, category resolution, dedupe, deadline
capture.offer applies capture.enabled/excludedHosts/monitoredExtensions/
monitoredMimeTypes/minSizeBytes from settings, then the rules table (new
store::Rules + CORE's vdm::rules::match_rules/glob_match — DAEMON only converts its
own stored proto::Rule JSON into CORE's plain vocabulary, per that header's own
layering note), resolves the category folder (a rule's explicit categoryId/saveDir,
else store::Categories::guess_by_extension — the same extension guess
download.probe's suggestedCategoryId already used, now shared instead of
duplicated), dedupes against active (non-terminal) tasks by exact URL, and on `take`
calls add_one() — the same path download.add itself uses, so a captured download is
a real, admitted, persisted task, not a special case.

The 750ms deadline (CLAUDE.md §4 / AGENT-DAEMON.md build step 6) is checked
cooperatively between every step via a new rpc::CaptureDataSource seam: the real
implementation wraps store::Settings/Categories/Rules/Tasks; a test fake can jump its
own injected clock forward to simulate "the store was slow just now" with zero real
sleep. This catches the realistic failure mode (several slow steps adding up) though
it cannot preempt a single pathologically stuck call mid-flight — a true preemptive
guarantee would need the same async/background-thread treatment as download.probe,
which isn't safe to do against the same sqlite3 connection (opened SQLITE_OPEN_NOMUTEX,
explicitly not for concurrent use) without a second connection; left as a known,
documented limit of this pass rather than adding that plumbing speculatively.

capture.getRules returns the same settings-backed fields capture.offer itself reads,
so the two can never drift. rulesVersion is a placeholder constant (1) — no persisted
revision counter exists yet, and the extension already re-fetches on
event.settings.changed regardless.

New store/rules.{hpp,cpp}: rules table CRUD (list, and an atomic upsert+remove for
rules.upsert later). store::Categories::guess_by_extension replaces a duplicate copy
that used to live in sched/scheduler.cpp. store::Tasks::has_active_duplicate for the
dedupe check.

Verified against real veloxd + tools/testserver: a monitored-type offer answers in
~5ms and actually creates + downloads the task, correctly categorized; an
unmonitored type, an excluded host, a rule-vetoed host, and a second offer for a
still-active URL all answer ignore with the right reason; a bad category save dir
surfaces its real -32011. New capture_offer_test covers all of the above plus the
deadline itself (two cases, one per "slow" checkpoint), asserting real wall-clock
time barely moves even though the fake clock jumped 2 simulated seconds. Full ctest:
54/54 (excluding the pre-existing, unrelated conformance failure noted in the
previous commit).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GRDjHGgpYmMoPE2UFbe7pP
2026-09-12 13:58:45 +04:00
samiandClaude Sonnet 5 e0edf7a084 daemon: file capture.offer/getRules in deferrals (D7/D8); close settings.get/set (D9)
Rebased lane/daemon onto main first — fast-forward, no conflicts (lane/daemon was
already fully merged; main has since taken lane/gui, lane/pkg-qa, lane/proto).

1. capture.offer and capture.getRules were stubs, buried inside D3's generic stub list
   with no flag for how load-bearing capture.offer is: it's a CLAUDE.md §4
   non-negotiable (750ms deadline, fails open) and AGENT-DAEMON.md build step 6, and
   against the real daemon it 500s to -32603 every time — the extension captures
   nothing, not "falls through to Firefox on a slow path." Split into their own D7/D8
   rows so they stop being invisible.

2. settings.get/settings.set (D9, closed). Four pointer-to-member tables in
   dispatcher.cpp (bool / ranged int / plain string / string array), five enum-typed
   keys handled individually since parse_XXX already validates those — covers all 43
   SettingKeys without ~40 repetitive hand blocks. store::Settings::kDefaults grew from
   14 entries to 43 (the schema itself carries no "default" keyword anywhere, so these
   are hand-chosen — conservative for the ones ADR 0012 doesn't speak to;
   capture.monitoredExtensions defaults to the union of every builtin category's
   extensions rather than an arbitrary list of its own).

   settings.set validates every field before writing any of them: numeric min/max
   (hand-checked — the generated parser only checks JSON type, not schema
   constraints) and saveTo.* paths via fs::resolve_target/canonicalize_root (-32011) —
   allowedRoots entries checked as roots in their own right, defaultDir/tempDir checked
   as paths resolving inside the (possibly just-updated, same call) root list. Reports
   exactly the keys whose effective value actually changed, publishes
   event.settings.changed with that list, and calls the scheduler's reload_config()
   when a connection.* key took effect — live, not on next restart.

   Verified against real veloxd: all 43 keys round-trip with defaults, a keys subset
   filters, an out-of-range value rejects the whole call, a bad saveTo.defaultDir is
   -32011, allowedRoots+defaultDir set together cross-validate against the new roots,
   event.settings.changed fires over a live subscription. New dispatcher_settings_test
   covers the same ground without a socket. Full ctest: 53/53 (excluding a pre-existing,
   unrelated conformance failure — see below).

Also found, not fixed (not this lane): tests/conformance/run.sh's live-veloxd leg fails
"pairing failed: no token issued" reproducibly, isolated, on main before any of this
session's changes — it launches the isolated veloxd without VELOX_PAIR_AUTO=1, so
EnvAutoApprover denies every session.pair and the WS leg's session.hello never gets a
token. tests/conformance/ is PROTO/QA-owned; flagging rather than editing across the
lane boundary.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GRDjHGgpYmMoPE2UFbe7pP
2026-09-12 13:42:45 +04:00
sami f4a6cebb3e merge: lane/proto 2026-09-12 11:05:07 +04:00
sami 963a76b6be merge: lane/pkg-qa 2026-09-12 11:05:07 +04:00
sami 765c1701d4 merge: lane/gui 2026-09-12 11:05:07 +04:00
sami b578e6de1a merge: lane/daemon 2026-09-12 11:05:07 +04:00
sami ff721f4065 merge: lane/core 2026-09-12 11:05:07 +04:00
samiandClaude Sonnet 5 f2ee45f818 core: add a state dump to tools/bench load on task timeout
Per-task engine state, downloaded/effective_segments, every segment's own
state, and the engine's SegmentBudget snapshot -- printed once, when a task
times out, instead of needing to re-run the bench under a debugger or add
throwaway instrumentation to find out why. This is what actually diagnosed
the SegmentBudget over-admission bug fixed in the previous commit: the dump
showed budget.active pinned at max_active_segments while multiple tasks sat
starved, which is what pointed straight at confirm_slot()'s missing
engine-wide check rather than a per-task target bug.

Also updates the vdm_bench_load20 ctest registration's comment: the
straggler under --preset tsan that motivated running it at reduced
concurrency (docs/adr/0016's postscript) is now suspected to have been the
same SegmentBudget bug (docs/adr/0017), not the TSan-timing artifact first
guessed -- not reverified at the DoD's full shape under TSan in this change,
so the reduced-concurrency registration stays for now.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Q3QrF7rCt21bkAjt9BCDFQ
2026-09-12 10:58:32 +04:00
samiandClaude Sonnet 5 322a20efa5 core: fix SegmentBudget over-admission and add a wait-list wakeup
Root cause of the M7 RSS gap (core/docs/m7-baseline.md: ~70 MB measured
against a 60 MB target): SegmentBudget::confirm_slot() only checked a task's
own held count against its own target -- never the engine-wide active_ sum.
reallocate_locked()'s two-pass fairness allocation does bound
sum(target) <= max_active_ at the moment it computes a plan, but that bound
says nothing about sum(held): a task can be legitimately holding more than
its own just-lowered target for a while (yield is deferred to a segment
boundary, never mid-segment -- ADR 0011 A1), and another task's target can
correctly rise to claim that capacity before the first task has physically
released it. Both confirm_slot() calls could then succeed against their own,
individually-correct targets while sum(held) exceeded max_active_ --
tools/bench heap-profile caught this directly: budget.active reading 56-86
against a total of 32.

confirm_slot() now also checks active_ < max_active_, unconditionally, as a
backstop that doesn't depend on any task's target bookkeeping being in sync
with what every other task holds. That creates a liveness question the
original design never answered: a task denied only by this new check has a
target that's already correct, so it never changes again and
reallocate_locked()'s plain "fire a callback when a task's target changes"
mechanism never revisits it. Task gained a waiting_for_slot flag, set on
exactly this denial; release_slot()/deregister_task() (the only two places
that free real capacity) now hand a freed slot directly to the
highest-priority waiting task via wake_one_waiter_locked(), if
reallocate_locked()'s own plan didn't already produce a callback for anyone.

download_task.cpp's fill_slots_locked() needed a matching fix: a woken
task's stalled segments (SegState::stalled -- backed off mid-retry, its own
release_slot() already called) have no live worker and never surface
through Segmenter::assign_slot(), which only hands out unassigned or fresh
ranges. fill_slots_locked() now restarts any stalled segment with no live
worker directly (bounded by slot_target, same as its assign_slot() loop)
before looking for new work; a segment it doesn't get to keeps its own
scheduled retry_worker() timer as a second chance.

Also fixes a real TSan-caught data race this work surfaced: SegWorker::
speed_bps was written only by its own segment's curl callback and, before
Progress.speed_bps's polled-path fix, only ever read from that same thread
-- safe without synchronization. snapshot_progress() reading it from
whatever thread calls DownloadHandle::progress() broke that invariant
(workers_mu's shared_lock protects the workers map's structure, not an
individual SegWorker's fields). Now std::atomic<double> with relaxed
ordering -- an informational EMA, nothing synchronizes real state on it --
rather than adding a lock to the write side.

core/tests/segment/budget_test.cpp adds two tests reproducing the actual
gap (budget_active_never_exceeds_max_active_segments_under_concurrent_load,
budget_wait_list_wakes_a_task_whose_target_never_changed) plus a sanity
baseline (budget_release_wakes_a_denied_waiter), and introduces AsyncFakeTask
+ TestTimer for the one existing test that drives the budget from multiple
concurrent threads -- mirroring production's real dispatch (register_task()'s
on_target lambda posts through host.schedule(), download_task.cpp, never a
synchronous call) rather than adding reentrancy-guarding machinery to
SegmentBudget itself to compensate for a synchronous test double being
unlike production. See docs/adr/0017 for the full writeup, including what an
earlier version of this fix got wrong chasing a same-thread reentrancy
hazard that doesn't actually exist in production.

core/docs/m7-baseline.md updated: the RSS number now clears the DoD line
(45.41 MiB via heap-profile), root-caused rather than just re-measured.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Q3QrF7rCt21bkAjt9BCDFQ
2026-09-12 10:58:14 +04:00
samiandClaude Sonnet 5 7ea04fa79c core: add tools/bench heap-profile (no massif/heaptrack in this environment)
For the M7 RSS gap (core/docs/m7-baseline.md: ~70 MB measured vs ADR 0012's
~45-50 MB estimate) -- "find where before tuning anything". No root here to
install massif or heaptrack, so this is a small in-process equivalent:
operator new/delete (already overridden for alloc-check's counter) now also
track, per call site (one return address via __builtin_return_address(0),
cheap enough to run for a whole scenario), live (not-yet-freed) bytes. Runs
the same 20-task concurrent scenario as `load`, waits for peak RSS to stop
growing, then prints the top sites by live bytes -- resolved via one batched
addr2line invocation against /proc/self/exe (dladdr alone only resolves
dynamic-symbol-table entries, which misses most of this codebase's
internal-linkage call sites), with dladdr as a per-site fallback.

Also adds the nothrow operator new/delete overloads alongside the existing
plain ones: without them, anything that allocates via the nothrow form (e.g.
std::stable_sort's std::get_temporary_buffer, hit once while investigating
the RSS gap) falls through to ASan's own default nothrow new while still
being freed through this file's plain delete override -- an
alloc-dealloc-mismatch ASan correctly flags as a real ABI-level bug in an
allocator override that claims to intercept "everything".

Using this tool, sum(effective_segments) across all 20 tasks vs.
Engine::segment_budget().budget() (now cross-checked and printed together)
showed the budget hands out well more than max_active_segments -- the actual
root cause, in core/src/segment/budget.cpp, not covered by this commit.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Q3QrF7rCt21bkAjt9BCDFQ
2026-09-11 22:35:09 +04:00
samiandClaude Sonnet 5 ef58796d22 core: fix Progress.speed_bps reading 0 on the polled path
DAEMON reported Progress.speed_bps reading 0 for the whole life of a live
throttled download while downloaded bytes visibly advanced. DAEMON reads
progress by polling DownloadHandle::progress() (engine_port_core.hpp), not
the on_progress push callback.

DownloadTaskState::snapshot_progress() -- the body behind progress() -- never
set speed_bps, per-segment speed_bps, or eta_seconds at all; only the
event-driven emit_progress_if_due() (which drives the on_progress callback)
computed them, from the same live SegWorker::speed_bps EMA seg_data()
maintains. snapshot_progress() now reads that same per-worker speed while
building its segment list, so a segment with no live worker (idle, paused,
complete, failed) correctly reports 0 and a segment with an active transfer
reports its real EMA, matching emit_progress_if_due()'s math including the
eta_seconds derivation.

engine_polled_progress_reports_nonzero_speed reproduces the bug (fails
without the fix, confirmed) by polling .progress() -- the same path DAEMON
uses -- during a throttled download and asserting speed_bps > 0 once real
progress has accumulated.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Q3QrF7rCt21bkAjt9BCDFQ
2026-09-11 22:34:47 +04:00
samiandClaude Sonnet 5 cf9e226e61 daemon: D1 — checked, not attempted; document why
libdbus-1-dev / libsystemd-dev have no headers installed in this build environment
(only the runtime .so's — apt-cache policy confirms libdbus-1-dev is available but not
installed). A real org.freedesktop.Notifications-backed PairingApprover needs one of
those linked into veloxd, which is a new build dependency for daemon/CMakeLists.txt
and, since packaging manifests would need to know about it too, a decision to surface
rather than reach for silently mid-session.

PairingApprover::approve() is also still synchronous by shape — its own doc comment
already says the real approver "will run async and is not this shape." The async
pattern this session built for download.probe (rpc::TaskActionPort + the server-layer
deferred-reply special-case in uds_server.cpp/ws_server.cpp) is the right shape to
reuse once there's a real implementation to justify reshaping the interface; doing
that with nothing behind it yet would just be churn.

Left EnvAutoApprover in place rather than hand-roll a D-Bus wire client to route
around the missing headers — a broken pairing approver is a worse outcome than an
honest, already-documented stub. Findings recorded in deferrals.md for whoever picks
this up once the dependency is available and approved.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GRDjHGgpYmMoPE2UFbe7pP
2026-09-11 17:46:43 +04:00
samiandClaude Sonnet 5 c89158ea09 daemon: D3 (partial) — category.upsert/remove, queue.upsert, download.remove/addBatch/provideAuth
Closes a bounded, high-value slice of the remaining D3 stubs. settings.*/rules.*/
limiter.*/schedule.*/grabber.*/media.*/capture.*/queue.reorder/download.refreshUrl/
download.update stay deferred — reasons noted individually in deferrals.md (settings.*
specifically: a real, large field<->key<->JSON-type mapping table across ~43 fields /
~50 SettingKeys, not started rather than rushed).

category.upsert/remove: store/categories.hpp gains get/upsert/remove. upsert generates
an id when absent and always ignores the payload's `builtin` (preserved from the
existing row on replace, false on create); saveDir goes through the same
fs::resolve_target canonicalize-and-root-check as download.add. remove refuses a
builtin at both layers (dispatcher's -32602 pre-check; the store's own
"DELETE ... AND builtin = 0" as defense in depth) and reassigns member tasks to
reassignTo (default "general") inside one transaction before deleting the row.

queue.upsert: store/queues.hpp gains get/upsert (set_state already existed from D4b).
Same create-generates-id pattern; a create always starts 'stopped', a replace keeps
the queue's current run state (upsert edits config, not run state — that's
queue.start/stop). Also fixed in passing: on_complete has been a real column since
migration 0001 but Queues::list/get never projected it onto Queue.onComplete.

download.remove: cancels with discard_partial=true (always drops the .veloxpart pair —
unlike download.cancel, which keeps them, the row is gone either way), deletes the
finished file only when deleteFile is true and the task was complete (best-effort),
deletes the row (segments cascade via the FK), and publishes event.task.removed
(closing the last open note under D5).

download.addBatch: on_download_add's body is now a shared add_one(), called once per
item after merging each item's unset fields against params.defaults.

download.provideAuth: forwards to EnginePort::provide_auth via a new
TaskActionPort::provide_auth. `remember`/persisting to the Secret Service is accepted
but not acted on — nothing in this build talks to libsecret yet.

Verified against real veloxd + tools/testserver: category create/replace/
remove-with-reassignment, queue create/replace-keeps-run-state, a batch add sharing
defaults.saveDir, and download.remove with deleteFile actually deleting the file and
the task then 404ing download.get with -32010. Full ctest: 39/39.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GRDjHGgpYmMoPE2UFbe7pP
2026-09-11 17:43:43 +04:00
samiandClaude Sonnet 5 4f6fb1029d daemon: D2 — download.probe, genuinely async on both transports
download.probe was a stub (-32603). It needs an HTTP round trip on the engine's probe
pool (up to the schema's 30s x-deadlineMs), which cannot fit VeloxDispatcher's
synchronous on_download_probe -> HandlerResult<T> return without blocking the RPC
loop for the duration — a hard no per CLAUDE.md ("never block the RPC loop") and
AGENT-DAEMON.md build step 1.

uds_server.cpp and ws_server.cpp special-case "download.probe" before the generic
dispatch(), exactly the way they already special-case session.hello/session.subscribe:
parse the params, call the port, and queue the reply whenever the callback fires
(dropped silently if the connection is gone by then).

rpc::TaskActionPort gains probe_now(DownloadProbeParams, callback) — kept in proto/std
terms, no vdm::net::* in the signature, so veloxd_rpc never needs core/include's vdm
headers just to declare this. sched::Scheduler::probe_now is the implementation:
builds a vdm::net::ProbeRequest, runs it on the engine's probe pool, marshals the
engine-thread callback back onto the loop (deps_.post_to_loop, same as every other
engine callback here), maps a probe failure to -32013 ProbeFailed (data.httpStatus set
when there was an HTTP response), and fills suggestedCategoryId/suggestedSaveDir with a
plain extension match against the categories table — not the real rules engine, which
is still D3; noted in a comment.

Verified against real veloxd + tools/testserver, not just unit tests: a real probe
answers in ~5ms with size/resumable/etag/redirect chain; a bad host maps to -32013;
and — the actual point of the async design — a connection running a 10s slow-loris
probe does not block a second connection's download.list, which answers in ~1ms while
the probe is still outstanding.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GRDjHGgpYmMoPE2UFbe7pP
2026-09-11 17:30:41 +04:00
samiandClaude Sonnet 5 55f0c6099d daemon: D4b — download.pause/resume/start/cancel and queue.start/stop drive the scheduler
download.pause/resume/start/cancel and queue.start/stop were stubs; now they call into
the scheduler and take effect immediately, not on the next 1s tick — pausing, resuming
or cancelling a live transfer can't wait, and per ADR 0013 §3 the governor never
touches a user-owned pause on its own.

New rpc::TaskActionPort interface (owned by rpc/, implemented by sched::Scheduler) is
what dispatcher.hpp depends on instead of sched/scheduler.hpp directly. Needed because
veloxd_sched already links veloxd_rpc (for EventHub); dispatcher.hpp pulling in
sched/scheduler.hpp directly would make it a real circular library dependency, breaking
anything that links veloxd_rpc alone (cli's tests, as it turned out — hit and fixed
during this change).

Scheduler::user_pause/user_resume/user_start/user_cancel + pause_queue follow tick()'s
existing to_pause pattern: call the engine (async, no synchronous effect) and transition
the store eagerly so download.get/list are correct the instant the RPC call returns.

Fixed a real bug surfaced while building this: transition() always overwrote
pause_reason to NULL when the engine's own delayed pause-ack callback (on_state to
paused, no error) arrived after whoever actually initiated the pause had already
written the real reason — now it preserves the stored reason when the callback supplies
none, instead of clobbering it. Covered by a regression check in sched_scheduler_test.

store/queues gets get() and set_state() (was list()-only) for queue.start/stop.

Verified against real veloxd + tools/testserver, not just unit tests: pausing a live
single-segment throttled transfer freezes downloadedBytes, resume continues it from
that point, cancel stops it; a bad taskId comes back in BulkTaskResult.failed with
-32010, not a crash; queue.stop(pauseRunning:true) pauses the queue's running task
immediately and queue.start resumes admission.

Known gap: download.start's contract "a task in 'queued' jumps its queue" (priority
bump) is not implemented — admission is still plain FIFO by created_at. Noted in
deferrals.md.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GRDjHGgpYmMoPE2UFbe7pP
2026-09-11 17:20:58 +04:00
samiandClaude Sonnet 5 de83ee3cce gui: Options, Scheduler, Speed Limiter, Batch, Grabber, and the tray icon
Continues the build order past the Add-URL/File-Info/Progress dialog flow.

- OptionsDialog: General/Save To/Connection/Downloads/Proxy/Sounds tabs, every
  control bound to a real settings.* key (contracts/schema/types/Settings.
  schema.json). The spec's File Types and Site Logins tabs have no settings.*
  backing (categories go through category.upsert, credentials through the
  Secret Service) so they don't exist here — a tab either binds to a real key
  or isn't shipped. diffChanged() sends only what actually changed, matching
  settings.set's "changed[] names exactly what took effect" contract.
- SchedulerDialog: per-queue schedule (schedule.get/set) plus maxConcurrent/
  onComplete (queue.upsert), Start Now/Stop. Queue.schema.json already carries
  the schedule so queue.list alone seeds the window.
- SpeedLimiterDialog: the live global limiter (limiter.get/set) — a different
  thing from Options' downloads.speedLimit* default. buildParams() enforces
  the schema's "0 with enabled true must not be offered".
- BatchDialog: clipboard-blob and {start..end}-wildcard tabs sharing one
  category/queue/start-mode footer into download.addBatch.
- GrabberWizard: 4-step QWizard (project label -> start URL/depth/filters ->
  file-type filter -> review), grabber.start feeding a poll+event.grabber.
  progress-driven review page, Finish = grabber.harvest for the checked files.
- TrayIcon: active-count tooltip, Show/Add URL/Pause All/Resume All/Speed
  Limiter submenu/Quit. Quit only closes the GUI — there is no RPC to stop
  veloxd itself, filed as a new gap in daemon-requests-m1.md. MainWindow now
  also hides to tray instead of closing when general.minimizeToTray is set.

Every dialog's non-widget logic (diffChanged, buildSchedule, buildParams,
parseUrlBlob/expandWildcard/buildAddBatchParams, buildFileTypes/
buildStartParams/buildHarvestParams) is a static pure function with its own
test, same shape as FileInfoDialog::buildSpec from the previous round.

Verified end-to-end against a running mockd under ASan+UBSan: all five
surfaces render real data (settings.get values, queue.list's two seeded
queues, limiter.get, a live grabber.start/status crawl returning 3 files) with
no sanitizer reports. gui-check (non-ASan) and dev (ASan+UBSan) presets both
build the whole repo clean; all gui-labeled ctest targets pass.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01NSCdCWFXBTSBBK3MzWtJiC
2026-09-11 17:17:43 +04:00
samiandClaude Sonnet 5 39c69f3871 gui: rpc plumbing for settings.changed and grabber.progress
Protocol.hpp gains method-name constants for every RPC the next batch of dialogs
needs (settings.*, limiter.*, schedule.*, queue.*, download.addBatch, grabber.*)
and RpcClient re-broadcasts the two events nothing consumed yet:
event.settings.changed and event.grabber.progress. No behaviour change on its
own — OptionsDialog, SchedulerDialog and GrabberWizard are what actually call
these, landing next.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01NSCdCWFXBTSBBK3MzWtJiC
2026-09-11 17:17:23 +04:00
samiandClaude Sonnet 5 efe76c319d pkg: nightly integration run — real veloxd + testserver, 50 concurrent
Adds tests/integration/nightly_run.py, wired as the nightly-integration job in
ci.yml (schedule + workflow_dispatch). Real veloxd + tools/testserver, 50
concurrent downloads mixing flaky-reset/throttled/no-range/plain, asserting:
every completed file's SHA-256 against testserver's own /sha256/ route (never
trusting veloxd's own success claim), every task reaching a terminal state
inside the timeout, and veloxd's own open-FD count settling back to baseline.

veloxd runs isolated (XDG_RUNTIME_DIR/XDG_DATA_HOME/XDG_CONFIG_HOME under a
fresh mkdtemp — not the session scratch dir, whose path overflows AF_UNIX's
sun_path). saveTo.allowedRoots is seeded directly into velox.db after a
migrations-only warm-up start, since settings.set returns -32603 'not
implemented in this build' on the veloxd this job builds (verified live).

Every task gets its own filename override on download.add: tasks sharing
(mode, size) share a URL, and without distinct filenames they raced each
other's rename on the first real run (48/50 'passed' with io_errors and
checksum mismatches on the collided tasks) before this fix.

Every assertion was forced red once on purpose and the transcript recorded in
tests/integration/README.md, per this repo's history of green checks that
didn't look where the bug was.

Every spawned child (veloxd, testserver.py) gets PR_SET_PDEATHSIG plus its own
process group, so a hard-killed harness can't strand a daemon the way a prior
run did (4h40m under systemd --user, because SIGKILL never reaches a
finally: block). Proven by kill -9'ing a running harness mid-download and
confirming both children exit with it.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01RBPR7iM3YPyxrjWsVtZDPJ
2026-09-11 17:07:42 +04:00
samiandClaude Sonnet 5 de748cc2fc daemon: fix startMode 'later' -32603 and isolate the single-instance lock by runtime dir
Two bugs blocking PROTO's live-veloxd conformance check.

1. tasks.start_mode's CHECK was ('auto','now','queue','manual') — not the
   contract's StartMode enum (['now','later','queue']) at all. 'later', a real
   documented value (the File Info dialog's Download Later button), hit the CHECK
   on every insert and surfaced as an unhandled -32603; 'auto'/'manual' were
   never contract values to begin with.

   Migration 0003 rebuilds tasks (SQLite can't ALTER a CHECK) with the contract's
   values, remapping existing rows by what they actually meant: 'auto' -> 'now'
   (eligible for the scheduler immediately), 'manual' -> 'later' (parked, matching
   StartMode's own "lands the task in paused" description). store_migrations_test
   covers the remap and that 'later' inserts clean while the retired spellings
   are rejected.

   dispatcher.cpp's on_download_add matched: default (absent startMode) is now
   'now' instead of the invented 'auto'; 'later' actually lands the task in
   `paused` (pause_reason 'user') instead of a dead 'manual' -> `new` branch that
   spec.startMode (typed as the 3-value enum) could never even reach.
   TaskRow::start_mode's in-memory default followed suit ('now').

2. main.cpp's single-instance guard bound an abstract socket named
   "velox-daemon-<euid>" — one name per user, system-wide. XDG_RUNTIME_DIR
   isolation never reached it: a leaked test veloxd held the lock for 4h40m and
   locked out every other isolated instance with the same euid (PROTO, EXT, the
   orchestrator), real daemon included.

   Extracted rpc/single_instance.{hpp,cpp} (was a static in main.cpp, untestable)
   and derived the abstract-socket name from a hash of the resolved runtime dir
   path instead of euid alone. The real per-user daemon is still unique (its
   runtime dir is unique to it); isolated instances pointed at their own runtime
   dirs now coexist. main() resolves the runtime dir before acquiring the lock
   (was the other way around). New single_instance_test covers same-dir refusal,
   different-dir coexistence, and release-on-close.

Verified against real veloxd binaries, not just unit tests: startMode: "later"
via a live download.add lands in `paused`; two veloxd with different runtime
dirs run concurrently, two with the same one and the second refuses with the
runtime dir named in the error. Full ctest: 39/39.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GRDjHGgpYmMoPE2UFbe7pP
2026-09-11 17:04:02 +04:00
sami 67aefbce4b merge: lane/ext 2026-09-11 16:50:41 +04:00
sami 4df4d7ad08 merge: lane/gui 2026-09-11 16:50:41 +04:00
sami f73a57a9ad merge: lane/daemon 2026-09-11 16:50:41 +04:00
sami c710481018 merge: lane/core 2026-09-11 16:50:41 +04:00
samiandClaude Sonnet 5 d4ad48d494 core: stage 9 — rules/ (filename sanitization, collision policy, rule matching)
Pure functions only, per AGENT-CORE.md's build order: no I/O, no JSON, no SQL,
no notion of the wire Rule type — DAEMON decodes its own stored/wire
representation into these plain structs and calls in.

- rules/filename.hpp: sanitize_filename() turns a raw candidate (from
  net::parse_content_disposition or net::url_filename — neither is
  filesystem-safe by design; both headers say so and point here) into one
  safe to create on ext4/APFS/NTFS: strips separators and control bytes,
  folds NTFS-illegal characters, neutralizes reserved Windows device names,
  clamps length on a UTF-8 boundary. Total on hostile input; never empty.
  Not the path-traversal security boundary — that's daemon/fs/safepath,
  downstream of this and the one that actually matters adversarially.
- rules/collision.hpp: resolve_collision() finds the next free name
  Explorer/Finder-style ("name (1).ext", ...) given an existence predicate,
  or returns the desired name unchanged under an overwrite policy. Never
  fabricates a guaranteed-unique name past its attempt bound — hands back
  the last candidate tried rather than hiding a persistent collision.
- rules/match.hpp: match_rules() is the evaluation half of
  contracts/schema/types/Rule.schema.json — priority order, first rule
  whose present match clauses (extensions/mimeTypes/host & url glob/size
  bounds) all hold, wins; a size clause never matches speculatively before
  the probe fills in size_bytes. glob_match() is the iterative (not
  recursive — bounded work on an all-'*' pattern) matcher both host_pattern
  and url_pattern use.

Every header compiles standalone; tests (39 cases) pass under ASan+UBSan and
TSan. core/include/vdm/README.md documents the new public surface.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Q3QrF7rCt21bkAjt9BCDFQ
2026-09-11 13:25:15 +04:00
samiandClaude Sonnet 5 b60d4e6f5b core: add tools/bench (throughput/load/alloc-check) and record the M7 baseline
Three subcommands in one binary, driving vdm::Engine directly (docs/04 §8):

- throughput: a single download against a fast local origin
  (support/local_server.hpp, busybox httpd), reporting Mbps/CPU%/RSS. Gates
  on --require-mbps/--max-cpu-pct only when passed, so the ctest smoke
  registration stays a correctness check, not a hardware-dependent
  perf gate -- the real 1-Gbit-link sign-off is a manual/CI job (see the
  file's header comment).
- load: N concurrent tasks against tools/testserver's `throttled` mode
  (support/testserver_client.hpp), reporting peak RSS via getrusage(). Paced
  externally rather than through the engine's own rate::RateLimiter or
  busybox: the limiter's pause/resume path allocates on every throttle event
  (would contaminate alloc-check's measurement) and under heavy segment
  contention was found to starve individual tasks indefinitely (see
  docs/adr/0016, added here); busybox couldn't sustain the DoD's ~160
  concurrent connections (20 tasks * default_segments=8) reliably. The
  ctest registration runs at reduced concurrency under sanitizer presets --
  see the CMakeLists.txt comment and the ADR's postscript.
- alloc-check: operator new/delete overridden process-wide, sampling the
  allocation count across a steady mid-transfer window against a paced
  tools/testserver origin. Caught a real bug in the same change (see the
  http_client.cpp commit) and, by dropping its Engine mid-download to end
  cleanly, also surfaced the quiesce() use-after-free (see that commit).

core/docs/m7-baseline.md records actual measured numbers against the M1/M7
DoD lines, including where they don't clear yet (RSS ~70 MB vs a 60 MB
target; throughput/CPU only measured on loopback, no 1 Gbit link available
here) rather than rounding them away.

docs/adr/0016 documents a rate::RateLimiter fairness gap found building the
load subcommand: a single shared TokenBucket under heavy segment contention
has no fairness ordering across its peek/commit race and can starve a
waiter well past what its configured rate implies. Filed as a follow-up
(it's a core/src/rate design question, not a tools/bench one) rather than
fixed here, along with a related TSan-only load-test straggler that could
not be root-caused in this environment.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Q3QrF7rCt21bkAjt9BCDFQ
2026-09-11 13:12:57 +04:00
samiandClaude Sonnet 5 6163898c14 core: stop allocating an empty deque on every worker-loop iteration
HttpClient::Impl::drain_commands() default-constructed a std::deque<Command>
every call, on every iteration of the worker thread's event loop (once per
curl_multi_poll wake -- i.e. once per socket-readiness event on the transfer
hot path), then swapped the (usually empty) command queue into it. In
libstdc++, an empty std::deque still allocates its map array on construction,
so this was a real allocation on the hot path regardless of whether any
command (add/pause/resume/cancel) was actually pending -- which is the
common case, since those are rare next to data arriving.

Found via tools/bench's alloc-check, which is built in this change and
exists specifically to catch this class of bug (AGENT-CORE.md: "no
allocation in the curl write callback... checked in review and by a bench
assertion"): before this fix it reported thousands of allocations/sec under
a sustained transfer; after, single digits.

Fixed by checking `w.queue.empty()` under the lock before touching `local`
at all, so the deque is only constructed when there's actually something to
swap into it.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Q3QrF7rCt21bkAjt9BCDFQ
2026-09-11 13:12:57 +04:00
samiandClaude Sonnet 5 ef816c21fb core: fix quiesce() racing an in-flight write callback (ASan use-after-free)
DownloadTaskState::quiesce() (Engine shutdown / ~Engine, via quiesce_task())
cancelled every live worker's transfer, then immediately cleared `workers` on
the calling thread. transfer.cancel() only *requests* the HttpClient worker
thread stop the transfer -- it does not wait for that to happen. If that
thread was mid write-callback (seg_data -> WriteBuffer::append ->
SparseFile::write_at), clearing the map destroyed the SegWorker (and its
ring buffer) it was still writing through: a heap-use-after-free, caught by
ASan via tools/bench alloc-check, which by design drops its Engine while a
download is still active mid-sample.

Every other exit path (verify/fail/auto_pause/demote, via begin_drain_locked)
already gets this right: cancel, then let each worker remove and flush
itself through seg_finished once HttpClient actually confirms the transfer
stopped, on the correct thread. quiesce() now does the same instead of
tearing the map down itself -- wait on a condition variable, notified from
seg_finished right after it erases, until `workers` is empty.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Q3QrF7rCt21bkAjt9BCDFQ
2026-09-11 13:12:57 +04:00
samiandClaude Sonnet 5 c99d1d9701 core: drain-aware hostile-mode handling (etag/416/lying-ranges/mismatch)
Resumes work left mid-session on the M1 hostile-mode matrix. download_task.cpp:

- Replace the old cancel-then-clear teardown (cancel_all_transfers_locked /
  start_assembly_locked) with a single begin_drain_locked()/PendingAction
  mechanism: cancel every live worker, remember what to do (verify / fail /
  auto_pause / demote), and let whichever worker's seg_finished finds the
  worker map empty carry it out. Every sibling still flushes its buffer on
  the way out, so no buffered-but-unflushed tail is lost when a download
  finishes or fails while other segments are still mid-transfer.
- A 200 where 206 was expected (wrong_status) now checks the response's
  ETag/Last-Modified against the probe's: a real mismatch asks the user
  (server_file_changed, "ask, never silently corrupt" -- docs/04 §5); a match
  means the server just stopped honouring Range for this connection, so
  demote to one segment and keep going without a round trip (docs/04 §7).
- 416 mid-download (stale range metadata) now surfaces as a decision instead
  of retrying the same now-invalid range to exhaustion.
- do_decide's abort path surfaces the actual reason a decision was asked
  for (last_error) instead of hardcoding server_file_changed, which was
  mislabeling a 416 abort.

engine_test.cpp adds the four hostile modes where a bug means silent
corruption rather than a visible failure: etag-changes, 416-always,
lies-about-accept-ranges, content-length-mismatch.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Q3QrF7rCt21bkAjt9BCDFQ
2026-09-11 13:12:57 +04:00
samiandClaude Sonnet 5 5b03eff926 proto: wire veloxd into the conformance suite's canonical entry point
run.sh's TS runner only ever started mockd; veloxd existed but nothing in
the ctest -L conformance path touched it, so mockd's always-valid fixtures
were the only thing ts/replay.ts ever saw. That let a real bug through:
veloxd's download.get can return segments: 0, which TaskSummary.segments
forbids (minimum 1, required) — nothing caught it.

Add a 3b step to run.sh (still the one canonical entry, per ADR 0014):
builds veloxd, starts it isolated (its own XDG_RUNTIME_DIR/XDG_DATA_HOME/
XDG_CONFIG_HOME), seeds saveTo.allowedRoots/defaultDir directly into the
isolated velox.db (settings.set is itself a stub, and the default
~/Downloads root doesn't isolate download.add's writes), then replays
every fixture against it over both transports.

Most handlers are still stubs (daemon/docs/deferrals.md D1-D4b). Fixtures
that hit them get an expected-failure entry in the new veloxd-xfail.json,
loaded by replay.ts's new --xfail flag. This is a maintained allowlist,
not a snapshot: a listed fixture that unexpectedly *passes* is flipped
back to a failure (applyXfail), so the list can only shrink as DAEMON
lands handlers, never rot into a list nobody rechecks. download.get's
segments: 0 is deliberately *not* on it — that's the regression this
step exists to catch.

Also hardened setupBindings: a server that can't even complete fixture
binding used to take the whole runner down with an uncaught exception
before a single fixture was checked. It's now a reported Outcome instead,
so the run still produces a coherent report. That robustness fix earned
its keep immediately: veloxd's download.add crashes on startMode
"later" (a valid, documented StartMode — "the File Info dialog's
Download Later button") with a SQLite CHECK constraint violation, because
migrations/0001_initial.sql's start_mode CHECK never had 'later' added to
it (and includes 'manual'/'auto', neither a contract value). That's a
second, more severe bug this wiring found, unrelated to segments: 0 and
currently blocking most of the veloxd run — filed for DAEMON in
tests/conformance/README.md, not fixed here (out of lane). capture.offer
and capture.getRules are also stubs but missing from deferrals.md's
D-list; xfailed with a note asking DAEMON to add the row.

Verified live once against a real, isolated veloxd before this session's
sandbox became persistently contended for veloxd's single-instance lock
(UID-scoped, not namespaced by XDG_RUNTIME_DIR — daemon/src/main.cpp;
documented as a caveat in the README): it built, started isolated, seeded
settings, connected over both transports, and surfaced the startMode bug
above as a real, non-xfailed failure — confirming the whole pipeline
including --xfail end to end. segments: 0 is confirmed by direct reading
of daemon/src/store/tasks.{hpp,cpp} (TaskRow::eff_segments defaults to 0,
copied verbatim into TaskSummary.segments) rather than by a second live
run reaching that specific fixture, since setup itself fails first on the
startMode bug above. mockd path re-verified green after these changes
(200/200, up from 196/196 — the new setup/$taskId outcomes are visible
and passing).

Recommendation for PKG: don't flip this required yet. The existing
`conformance` ctest entry is already a required check, and right now the
startMode bug fails most of the veloxd run, not just the one expected
segments: 0 case — merging as-is would block every lane's PRs on two
DAEMON bugs at once, one of them unrelated to what this task set out to
catch. Required once DAEMON lands a fix for startMode "later" at minimum;
segments: 0 can stay red for a while by design, same as any other tracked
regression.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01SFeUKLbdHizrJjLBeK7ffz
2026-09-11 12:52:13 +04:00
samiandClaude Sonnet 5 03b6253b5a ext: content/ media detection (build order step 7)
Two independent signals feed one panel, per docs/05 §3:
- capture/media.ts (background): webRequest-based, recognizes .m3u8/.mpd
  URLs and the HLS/DASH content types, deduped per (tab, url) so an
  HLS live-refresh doesn't re-fire. media-bridge.ts relays a hit to the
  tab's content script over runtime.sendMessage, and separately answers
  the content script's media.listVariants/media.addVariant calls by
  forwarding them to the background page's transport.
- content/media-observer.ts: watches the page's own <video> elements
  (present now, added later, or with src changed) for the same
  extension signal, independent of what the network sniffer saw.

Either firing opens content/video-panel.ts's "Download this video ▾"
panel (built with createElement, matching the popup's innerHTML-free
approach), which lists variants from media.listVariants and greys out
any variant.drm or a wholly drmProtected manifest with "Protected
content" rather than attempting it. The extension still never parses a
manifest itself — that stays in the daemon, one language, one place.

content/index.ts is the manifest-registered entry (content_scripts in
manifest.json, added in the previous commit); build.mjs builds it as an
IIFE rather than ESM, since a manifest content script has no "type":
"module" declaration and an emitted top-level export would be a syntax
error there. tsconfig.json adds DOM.Iterable for NodeList iteration.

docs/05-extension-spec.md gets a short addendum (§8) documenting the
popup/options bridge and this media-detection split, since neither was
in the original design write-up.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Ed8KEmAW48v4YHdxLtqsMB
2026-09-11 12:26:48 +04:00
samiandClaude Sonnet 5 55932a2e11 ext: popup, options, and the panel bridge (build order step 6)
Popup and Options are separate documents from the background page and
can't reach its live VeloxTransport directly, so background/bridge.ts
relays it over one browser.runtime.connect port per document
(call/subscribe/getStatus/reconnect/pair/unpair/setOverride in;
result/event/status/pairError out). shared/panel-client.ts is the
client side both surfaces use.

Popup (src/popup/): status dot + text, active-downloads list driven by
event.task.progress/added/state/removed (repaints ride the event's own
<=4 Hz cap rather than adding a second timer), pause/resume buttons,
"Start it" wired to a reconnect request. State lives in a DOM-free
store.ts for unit testing; rendering uses createElement, not innerHTML
(web-ext lint flags the latter).

Options (src/options/): transport override select, pairing (code entry
+ pair/unpair, backed by two new WebSocketTransport methods,
pairWithCode/unpair), and the daemon's capture policy mirrored via
capture.getRules. The capture-policy form is editable only when the
active transport is native messaging (uds) -- settings.set and
rules.upsert are privileged, uds-only methods per shared/protocol
METHODS, so WebSocket can't write them no matter what the page shows;
CLAUDE.md section 2 rules out working around that locally. The bridge's
status payload adds a kind field (which transport is live) for this to
key off. Default category is the one piece of state that's genuinely
the extension's own, not the daemon's, and lives in
browser.storage.local via options/prefs.ts. Pure decisions (statusLine,
pairingAvailable, captureRulesEditable) are split into view.ts for unit
testing without a DOM.

manifest.json registers the popup action and options_ui page (and, in
the same edit, the content_scripts entry the next commit's media
detection needs -- split by file, not by manifest line). build.mjs
gains popup/options as further esbuild entry points, plus copying their
static HTML/CSS into dist/.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Ed8KEmAW48v4YHdxLtqsMB
2026-09-11 12:26:25 +04:00
samiandClaude Sonnet 5 25c171f742 ext: pairing-restart test + AMO permission justification
tests/transport/storage.test.ts covers the storage half of "the pairing
token survives a browser restart" (round-trip, unpair-clears, corrupted
override falls back to auto). websocket.test.ts adds the transport half:
a fresh WebSocketTransport instance over the same backing store reuses
the persisted token with no re-pairing, plus pairWithCode/unpair
coverage. "Wrong token rejected and rate-limited" was already covered
(websocket.test.ts's NotPaired/RateLimited cases).

docs/amo-permissions.md is the submission-ready permission justification
for AMO's Notes to Reviewer field, covering every permission in
manifest.json plus what was deliberately not requested and how cookie/
header data is handled.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Ed8KEmAW48v4YHdxLtqsMB
2026-09-11 12:25:42 +04:00
samiandClaude Sonnet 5 2cb1959bff ext: no-download-logic ESLint gate in the extension-lint CI job
CLAUDE.md §3's rule was prose only for extension/ (GUI already has
gui_no_download_logic as a ctest). eslint.config.mjs adds a
no-restricted-syntax/no-restricted-globals rule banning fetch/XHR/Request,
ReadableStream.getReader, Range/Content-Range header construction, and
IndexedDB in src/**/*.ts. Verified red on a planted violation (fetch +
Range header + stream reader) and green on ordinary code; that check is
now a permanent regression test (tests/lint/no-download-logic.test.ts)
rather than a one-off manual run. Wired into the existing extension-lint
job in .github/workflows/ci.yml, ahead of web-ext lint.

Generated protocol code (src/shared/protocol/**) is excluded from lint
entirely — it must never be hand-edited, so flagging it as fixable would
be a lie.

Answers gui/docs/ext-requests-m1.md.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Ed8KEmAW48v4YHdxLtqsMB
2026-09-11 12:25:19 +04:00
samiandClaude Sonnet 5 a967eca669 daemon: persist engine progress/probe to the store — segments:0 no longer leaks
Engine numbers never reached the store: download.get after a correctly-finished
download reported sizeBytes: null, downloadedBytes: 0, speedBps: 0, resumable:
false, segments: 0, segmentDetail: [] — segments: 0 breaks the frozen contract
(TaskSummary.segments is minimum:1, required).

- Scheduler::tick() now probes (EnginePort::probe) before every start(), persisting
  sizeBytes/resumable/validators via Tasks::set_probe_result before a byte moves,
  then starts the engine with that ProbeResult as probe_hint.
- Scheduler::persist_progress() (new) writes downloadedBytes/speedBps/segments/
  segmentDetail from the engine's Progress. Called from progress_snapshot() (the
  250ms tick) *and* once more from on_engine_state right before release()/unmap on
  every terminal transition, so a task that finishes between two ticks — the common
  case for anything small or fast — still leaves real numbers instead of the
  pre-persistence defaults.
- TaskSummary.segments is sourced from segments.size() when the task has any
  (matching what actually lands in segmentDetail, per the schema's "exactly
  segments entries"), falling back to the engine's effective_segments (budget
  slots *held*, not necessarily physical range count) only pre-segmentation.
- Tasks::set_final_bytes tops up on_finished's byte count as a last-resort
  backstop.
- store/segments.{cpp,hpp}: read/write access to the segments table behind
  TaskDetail.segmentDetail. Wired into daemon/CMakeLists.txt.
- migrations/0002: speed_bps on tasks and segments; fixes segments.state's CHECK
  to include 'pending' (0001 omitted it, so a pre-connect snapshot could never be
  written).
- store_migrations_test's forward-only loop faked "released version N" by setting
  the user_version pragma alone, with no real schema underneath — never exercised
  until 0002 existed. Fixed to actually build the db through migrations 1..N first.

Verified against real veloxd + tools/testserver (not just unit tests):
download.list/download.get correct immediately after completion and after a
daemon restart, with saveTo.allowedRoots pointed at an isolated dir.

Observed but not fixed (CORE, not this lane, noted in deferrals.md): Progress.
speed_bps reads back 0 for the whole lifetime of a live throttled download in the
same E2E check, despite downloadedBytes visibly advancing. DAEMON passes it
through unmodified; filed rather than worked around.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GRDjHGgpYmMoPE2UFbe7pP
2026-09-11 12:11:19 +04:00
samiandClaude Sonnet 5 a71d904a1f gui: finish Add URL -> File Info -> Progress dialog flow
Wires up the three dialogs from build order step 5 (docs/03-gui-spec.md
§§2-3) and the MainWindow slots that were declared but never implemented:

- AddUrlDialog: clipboard prefill is the explicit path docs/06 R2 calls
  for (no passive monitoring, not advertised).
- FileInfoDialog: async download.probe never blocks the UI; ends by
  calling download.add itself (Now / Later / Add to Queue). buildSpec()
  is a pure static so the optional-field-omission logic is unit-testable
  without touching a widget.
- ProgressDialog: non-modal, WA_DeleteOnClose, driven by the taskProgress/
  taskStateChanged signals RpcClient already re-broadcasts; download.get
  seeds state once for a dialog opened mid-transfer. Hosts SegmentBarsWidget
  and SpeedGraphWidget.

MainWindow: openAddUrlDialog/openPropertiesForSelection/showTableContextMenu
now have bodies; category.list/queue.list responses are cached so File Info
can populate its category combo and queue menu without a second round trip.
The row context menu covers what already exists (Resume/Pause/Stop/Delete/
Properties) and deliberately leaves out Open/Open With/Move-Rename/
Redownload/Add to Queue — those need dialogs later build-order steps haven't
reached yet.

util/Format.hpp: pulled the bytes/rate/eta formatting out of MainWindow and
DownloadTableModel once the dialogs wanted the same strings a third time.

Verified end-to-end against a running mockd (category.list/queue.list,
download.probe, download.add, download.get, and live event.task.progress/
event.task.state) under ASan+UBSan: all three dialogs render correctly
against real fixture data and the flow runs clean with no leaks or
sanitizer reports.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01NSCdCWFXBTSBBK3MzWtJiC
2026-09-11 12:11:13 +04:00
samiandClaude Sonnet 5 41bce91770 gui: add SegmentBarsWidget and SpeedGraphWidget
Per-connection progress bars and the 60 s rolling speed graph the progress
dialog needs (docs/03-gui-spec.md §3). Both reuse row/widget state across
ticks instead of rebuilding, matching the discipline DownloadTableModel
already uses for progress patches.

SpeedGraphWidget keeps a fixed ring buffer and one reused QPainterPath —
no allocation in paintEvent or addSample. Fixed a real bug found while
writing tst_speedgraphwidget: the elapsed timer was started in the
constructor, so the very first sample after construction would silently
wait up to 1 s to be recorded instead of landing immediately.

Tested against mockd (both offline via QTest/offscreen, and manually
against a running mockd instance through ProgressDialog once that lands).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01NSCdCWFXBTSBBK3MzWtJiC
2026-09-11 12:10:06 +04:00
sami 0a38867579 merge: GUI daemon-requests, verified against veloxd 2026-09-11 07:55:50 +04:00
samiandClaude Sonnet 5 0af4a5c4fc gui: verify against the real veloxd; file one daemon gap
Pointed the existing RPC client at a real veloxd instance (isolated
HOME/XDG_RUNTIME_DIR/XDG_DATA_HOME — never touched the real ~/Downloads),
added a real download.add against tools/testserver, and watched it render
live end to end with no GUI code changes:

  - handshake, subscribe, category.list/queue.list all match what DAEMON
    reported
  - the full event.task.state sequence and batched event.task.progress
    both applied correctly by DownloadTableModel
  - the written file's SHA-256 matches the server's reference

Filed gui/docs/daemon-requests-m1.md: TaskSummary.sizeBytes is never
populated by this daemon build, even in download.get after the task
completes with the exact byte count already on disk. Not a GUI bug —
ProgressDelegate and the model already do the documented right thing
when size is unknown (fall back to plain text, no bar) — but it means
every task renders without a percentage against the real daemon today.
mockd always supplies sizeBytes so this doesn't block current GUI work;
flagging before the M1 GUI<->daemon integration pass.

mockd stays the primary harness for the unhappy paths
(--slow/--flaky/--drop-connection) that a real daemon won't misbehave on
command for.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_016Ne28kx4VreeBWZv82Nksd
2026-09-11 07:55:06 +04:00
sami ba29fcb5bd merge: event fan-out, category.list, queue.list 2026-09-11 07:27:09 +04:00
samiandClaude Sonnet 5 824fa481bb daemon: event.* fan-out (D5) + category.list/queue.list (D3) — GUI-ready
The two items aimed at pointing GUI at a real veloxd instead of mockd.

rpc/event_hub — per-subscription fan-out shared by both transports.
subscribe() registers a connection with no interest; set_filter()
(session.subscribe, replaces not adds) turns on event kinds and an
optional per-task id filter; publish() delivers a pre-built
notification to every matching subscriber. session.subscribe on both
UdsServer and WsServer now does the real thing — registers/updates a
subscription, tears it down in close_conn.

sched/scheduler — the on_engine_state hook now actually publishes:
- transition() is the one place a task's row changes state; it reads
  the store's own prior row for previousState (authoritative
  regardless of engine/scheduler timing), writes the error columns,
  and — when a hub is supplied — publishes event.task.state with
  {taskId, state, previousState, summary, error}. Wired into every
  transition: scheduler-driven (admission -> probing, resume ->
  connecting, pause) and engine-reported (on_engine_state).
- progress_snapshot(): one row per task the engine is tracking
  (EnginePort::progress(), a new interface method backed by
  DownloadHandle::progress()), plus a store side-effect
  (Tasks::update_progress) so download.list/get stay current between
  state transitions. Returns rows; does NOT publish itself — batching
  into one array message is the caller's job, per the schema's
  x-maxRateHz: 4 and AGENT-DAEMON.md item 5 ("one message per task per
  tick burns a core"). main.cpp's 250 ms timerfd is that caller: one
  event.task.progress per tick, only when there's something to say.

dispatcher::on_download_add now publishes event.task.added (schema:
"summary is always present so a client can insert the row without a
follow-up download.get").

store/categories, store/queues — the two D3 handlers GUI's panels
call. category.list projects the six seeded built-ins; queue.list
derives taskIds from tasks.queue_id/queue_position (Queue's own schema
note: a queue's stored row never carries membership, download.update
/ queue.reorder do).

Verified live end to end against tools/testserver: a subscribed client
sees event.task.added on add, then the full event.task.state sequence
(queued -> probing -> connecting -> downloading -> assembling ->
verifying -> complete) with correct previousState at every step, and
real category.list / queue.list results.

Tests: event_hub (filter-by-kind, filter-by-task-id, replace-not-add,
unsubscribe), store_categories_queues, plus new sched_scheduler cases
for event.task.state publishing and progress_snapshot's store
side-effect. 38 daemon/cli tests green; sched_scheduler / event_hub /
ws_server / uds_roundtrip TSan-clean.

deferrals.md: D5 mostly closed (event.task.removed and the
still-unpublished events wait on their owning D3 handlers); D3 down to
the remaining download.* verbs, rules/settings/limiter/schedule,
queue mutation, category mutation, grabber, media.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
2026-09-11 07:19:11 +04:00
sami 72344134cc merge: engine wired into veloxd — vertical slice closes 2026-09-10 20:54:01 +04:00
samiandClaude Sonnet 5 08d7ee9263 daemon: wire the engine into veloxd — the vertical slice runs end to end
CORE stage 8 merged, so vdm::Engine is linkable. This closes D4a and
narrows D4b: `velox add <url>` now actually downloads.

- sched/engine_port_core.hpp — the real EnginePort: forwards to a live
  vdm::Engine, keeps the DownloadHandle per task for pause/resume/
  cancel/provide_auth/decide/refresh_url, drives set_task_order /
  set_max_active_segments / set_host_segment_cap via
  engine.segment_budget(). CORE confirmed the admission model: DAEMON
  decides when to start(); the engine's own download_task calls
  register_task/set_want internally — DAEMON never touches per-task
  budget calls. EnginePort gains release(TaskId) so the port drops a
  handle when the task goes terminal.
- rpc/event_loop — EventLoop::post(fn): thread-safe, runs fn on the
  loop thread next iteration. The marshaller for engine-thread
  callbacks.
- main.cpp — constructs vdm::Engine + EnginePortCore + Scheduler
  (post_to_loop = loop.post). At startup: reconcile_after_restart()
  (ADR 0013 §5), reload_config(), tick(). A 1 s timerfd on the loop
  re-runs tick() (schedule windows, missed nudges); download.add nudges
  via dispatcher.set_on_mutation.

End-to-end verified against tools/testserver: `velox add
http://127.0.0.1:.../file/512K` -> task queued -> scheduler admits ->
engine downloads 524288 bytes -> complete, file on disk. First
byte-path all the way through the project.

safepath-adversarial.md: re-verified per its own note — CORE landed
O_NOFOLLOW on the target open (core/src/io/sparse_file.cpp), so the
leaf-symlink TOCTOU is now closed; residual is down to one
intermediate-dir gap (documented post-M1 chase).

36 daemon/cli tests green; scheduler + uds_roundtrip TSan-clean.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
2026-09-10 20:36:11 +04:00
samiandClaude Sonnet 5 d93c8e10a0 daemon: sched/scheduler — governor <-> store <-> engine, against an EnginePort seam
The Scheduler that D4 was waiting on. Built against CORE's engine
HEADERS (now in main); the real EnginePort and the veloxd wiring wait
for lane/core's stage-8 bodies to reach main (deferrals.md D4a/D4b) —
core/src/task/ is still .gitkeep there, so linking vdm::Engine now
would be an unresolved symbol.

- sched/engine_port — the abstract seam: start/pause/resume/cancel/
  provide_auth/decide/refresh_url + the ADR 0011 admission config
  (set_task_order / set_max_active_segments / set_host_segment_cap).
  Keeps the Scheduler testable without a live engine and the daemon
  unbound from the concrete vdm::Engine.
- sched/fake_engine_port — a recording impl for tests.
- sched/scheduler:
  * owns the wire-UUID <-> vdm::TaskId map.
  * tick(): snapshot queues (schedule window evaluated with an
    injectable clock) + non-terminal tasks -> governor.evaluate ->
    apply. to_start builds a vdm::task::DownloadSpec from the row and
    calls EnginePort::start; to_resume -> resume(); to_pause ->
    pause() + writes the pause_reason; priority_order -> set_task_order
    over the mapped engine ids. `new` tasks are parked (startMode
    manual) and skipped.
  * on_engine_state(wire_id, state, err): projects an engine
    transition onto the store row (state, pause_reason='auto' when an
    error rides a paused transition per ADR 0013 §2, flattened error
    columns) so the next tick sees ground truth. This is also the hook
    event.task.state will fire from (D5).
  * reconcile_after_restart(): CORE-owned states -> queued, paused
    keeps its reason (ADR 0013 §5).
  * reload_config(): reads connection.maxConcurrentDownloads /
    maxActiveSegments + a daemon-local host-cap map, pushes caps to
    the engine, updates the governor.
  * Deps: injectable local-now clock and a post_to_loop marshaller
    (engine callbacks arrive on engine threads; default runs inline
    for tests).

Test veloxd.sched_scheduler (ASan+UBSan and TSan clean): admission +
ordering, a slot freeing on completion, queue-stop -> pause
(queue_stopped) then queue-restart -> resume (not a fresh start),
engine auto-pause -> pause_reason 'auto' + never auto-resumed,
reconcile_after_restart, reload_config caps push. 35 daemon/cli tests
green.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
2026-09-10 20:29:37 +04:00
sami 14128c1935 merge: stage 8 engine, O_NOFOLLOW fix, auth handshake 2026-09-10 20:22:01 +04:00
samiandClaude Sonnet 5 91636f8a4d core: add the download engine — task machine, DownloadHandle, Engine
Stage 8 of the CORE build order: the bodies behind the DownloadSpec /
callback API reviewed in core/docs/engine-api-m1.md. Wires probe -> segment
workers -> WriteBuffer -> SparseFile -> .veloxpart.meta -> retry/backoff ->
SegmentBudget -> RateLimiter -> callbacks into one event-driven machine.

- Engine (src/engine.cpp): owns HttpClient, Prober, SegmentBudget,
  RateLimiter and one timer jthread (min-heap of scheduled fns). start()
  builds a task and returns a DownloadHandle; ~Impl quiesces every task
  before joining the timer so no callback fires during teardown.

- DownloadTaskState (src/task/download_task.cpp): one `mu` task lock; a
  shared_mutex over the worker map for the curl write path; callbacks
  collected under `mu` and fired after release via a separate deferred
  queue; weak_from_this() in every async hop. State machine over the
  CORE-owned EngineState subset, auto-pause on 401/407 and on a 200 where
  206 was expected, validated resume via If-Range.

- digest (src/task/digest.cpp): OpenSSL EVP hash_file() for the optional
  post-download checksum; links OpenSSL::Crypto PRIVATE.

- Segmenter::release_segment(): hand a paused segment back to the pool
  unassigned so resume's assign_slot() picks it up instead of splitting a
  still-"assigned" range and orphaning its front half.

- DownloadHandle now names the real control block (vdm::task::
  DownloadTaskState, defined only in the engine TU) via a namespace-scope
  fwd decl and a public-but-effectively-engine-only ctor, replacing the
  nested State/friend pair. Every public signature is unchanged; DAEMON
  (vdm-79) confirmed sched/ names only the public API.

Fixes found while building the end-to-end suite (tests/task/engine_test.cpp,
9 cases against tools/testserver, green under ASan/UBSan and TSan):
- a dropped connection lost its unflushed WriteBuffer tail while advance()
  had already counted those bytes as done -> a retry resumed past an
  unwritten hole. Flush on the failure path.
- when the byte counters hit total while other workers were still live,
  teardown dropped their buffered tails. Now: cancel them and let each
  worker's own seg_finished drain it (the `assembling` state), last one
  starts verification -- no cross-thread buffer access.
- seg_head() let a 401 with credentials present abort before libcurl's
  resend; now it proceeds once and acts on the final status.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HPPSGhiArbvQgwC2DNiURS
2026-09-10 20:19:31 +04:00
samiandClaude Sonnet 5 efbf366c18 core: add test-name filters to the vtest harness
run_all() takes an optional substring list; vtest_main forwards argv and a
comma-separated $VT_ONLY. No filter => run everything, as before. Makes
iterating on one slow end-to-end case (the engine suite) practical
without a framework swap.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HPPSGhiArbvQgwC2DNiURS
2026-09-10 20:19:11 +04:00
samiandClaude Sonnet 5 afaded85f8 core: carry credentials through the probe and its auth handshake
libcurl with CURLAUTH_ANY answers a 401/407 by resending the request with
an Authorization header. Two spots in net/ cut that short:

- http_client's header callback delivered the response head exactly once
  and latched `head_delivered`, so after an auth challenge the caller only
  ever saw the 401 — never the 2xx of the authenticated resend. Reset the
  latch when a fresh status line follows a delivered 401/407 (redirects
  never reach that path — their head is suppressed).

- the prober's head callbacks return DataAction::abort to skip the body,
  which also aborts the transfer mid-handshake. Return `proceed` for a
  401/407 when credentials were supplied, so curl's resend can run; the
  real status lands on the next header block.

Also give ProbeRequest an `auth` field (default scheme == none) and pass
it through base_request(), so a re-probe after a 401 can present the
credentials the user just entered. No behaviour change when no auth is
configured.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HPPSGhiArbvQgwC2DNiURS
2026-09-10 20:19:05 +04:00
sami b02bdb0466 merge: store-backed download.add/list/get 2026-09-10 20:04:44 +04:00
samiandClaude Sonnet 5 3db0d01f9d daemon: download.add / download.list / download.get behind the store
VeloxDispatcher now takes a store::Db& and three handlers are real:

- download.list -> store::Tasks::list (filter / sort / paging in SQL) ->
  to_summary per row. No more empty-table stub.
- download.add -> resolve saveDir (spec, else saveTo.defaultDir; ~ expanded)
  and the leaf (spec.filename, else the URL's last segment percent-decoded,
  else download.bin) -> fs::resolve_target against canonicalize_root'd
  saveTo.allowedRoots. Any path-destination failure is -32011 with the
  *original* saveDir in data.path. On success a TaskRow is inserted in
  state `queued` (or `new` for startMode "manual") and {taskId, state}
  returned. The scheduler that would then admit it is D4.
- download.get -> store::Tasks::get; a real -32010 + data.taskId for an
  unknown id, else a TaskDetail (segmentDetail empty until the engine
  segments the task, which the schema permits).

util/time.hpp: now_iso() factored out of ws_server.cpp.

main.cpp constructs the dispatcher with the opened db. The three
integration tests build an in-memory migrated db for it; velox.client
now drives the full slice through the CLI — add outside roots -> -32011
with data.path, add into an allowed root -> a task that download.list
shows and download.get details, unknown id -> -32010. Verified with the
real binaries: velox add persists, velox ls shows it, it survives a
daemon restart, /etc is refused.

ASan+UBSan and TSan clean; 34 daemon/cli tests green. deferrals.md:
D2 down to just download.probe; D3 down to categories/queues/rules/
settings/limiter/schedule.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
2026-09-10 19:58:45 +04:00
samiandClaude Sonnet 5 479f882324 core: O_NOFOLLOW the download target open
DAEMON's safepath-adversarial.md accepts a TOCTOU residual between its
canonicalise-and-check and the download starting, on the stated grounds
that CORE's O_NOFOLLOW open of the final file closes it. That flag was
never actually set: SparseFile::open used O_WRONLY|O_CREAT|O_CLOEXEC, so
a symlink swapped in as the final path component after DAEMON's check
would be followed and redirect our pwrites outside the allowed roots.

Add O_NOFOLLOW. A symlinked leaf now fails the open with ELOOP, which
errno_to_error already maps to Error::path_rejected. Regular files and
the O_CREAT of a fresh part file are unaffected; resume (existing regular
part file) is unaffected. Test that a symlinked destination is rejected
rather than silently followed, and that the link target is never touched.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HPPSGhiArbvQgwC2DNiURS
2026-09-10 19:50:28 +04:00
samiandClaude Sonnet 5 7514f5a4c4 daemon: correct an unverified cross-lane claim in safepath-adversarial.md
The residual section claimed the leaf/component TOCTOU is "closed in
practice by CORE's O_NOFOLLOW open of the final file". Verified: it is
not — core/src/io/sparse_file.cpp:77 opens O_WRONLY|O_CREAT|O_CLOEXEC,
no O_NOFOLLOW, no O_EXCL. Requested the flags from CORE via PKG/QA.

Doc now states the residual is currently OPEN, names the file:line and
flags checked and the date, says what actually limits exposure today
(0700 parent dirs), and flags this as the boundary where a reader
stops checking. Step 5 reworded the same way. Re-verify the flags when
the CORE change lands.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
2026-09-10 19:49:51 +04:00
sami 477a98a31b merge: fs/safepath, store/tasks, store/settings 2026-09-10 19:44:52 +04:00
samiandClaude Sonnet 5 6787784936 daemon: store/ query layer — tasks + settings (prep for the download.add vertical slice)
The read/write surface the dispatcher handlers need, so download.add
persists and download.list / download.get project real rows when the
engine lands.

- store/tasks — TaskRow (1:1 with the schema), insert / get / remove /
  set_state / count, and list(filter, sort, offset, limit) that does
  all the WHERE / ORDER BY / LIMIT in SQL (M1 DoD: a 1000-row list
  never materialised client-side). Filter covers states / category /
  queue / case-insensitive filename+url substring / date range; sort
  is a whitelisted column + direction with NULLs last, default
  newest-first; the enum spellings in a state IN (...) come from
  proto::to_string, never from user text. to_summary() projects a row
  onto proto::TaskSummary including the flattened error block when the
  task failed / retry_wait / auto-paused.
- store/settings — key -> JSON-text with a built-in default table
  mirroring Settings.schema.json / ADR 0012; get_raw / set_raw /
  overrides plus typed get_int / get_string / get_string_array for the
  governor config and saveTo.allowedRoots. Full settings.get/set wire
  projection lands with those handlers.
- veloxd_store now links velox::proto + nlohmann_json for the
  projection.

Test veloxd.store_tasks: insert/get round trip, PK duplicate rejected,
the error-block projection, list total+paging+sort+every filter,
set_state pause_reason clear-on-unpause, remove, and settings default
vs override. ASan+UBSan and TSan clean; 34 daemon/cli tests green.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
2026-09-10 19:43:13 +04:00
samiandClaude Sonnet 5 ab479e7885 daemon: fs/safepath — the saveDir/filename path-traversal boundary (security)
veloxd is the one process that turns an untrusted string into a
filesystem destination, and via capture.offer that string can come
from a web page. CLAUDE.md §4 and the M1 DoD both name this.

daemon/docs/safepath-adversarial.md is the spec, written before the
code the way EXT did for shouldCapture: 21 rows — .. traversal
(A1/A2), absolute-outside-roots (A3), prefix-match confusion (A4),
symlink-out (A7), TOCTOU on a created tail (A8), NUL/control bytes in
the leaf that CORE's fuzzer hit through Content-Disposition (A9/A10),
degenerate and overlong leaves (A11/A13), overlong dir component
(A14), symlinked root (A16), destination-is-a-file (A17), and the
legitimate cases that must still pass — non-ASCII (A18), redundant "."
(A19), trailing space/dot trimming (A20).

fs/safepath.cpp:
- sanitize_leaf: strip <0x20 and 0x7F, trim ws, strip trailing dots,
  reject ""/"."/".."/contains-'/', cap 255 UTF-8 bytes on a codepoint
  boundary. Mirrors core/src/net/content_disposition.cpp.
- canonicalize_root: expand ~ and realpath each allowedRoots entry
  once, so a symlinked root resolves to its target.
- resolve_target: reject relative saveDir and any ".." component
  lexically; if the dir exists, realpath + component-wise containment
  (a symlink that escapes is caught, one that stays inside passes); if
  a tail is missing, realpath+check the deepest existing ancestor then
  create the tail via an openat/mkdirat O_NOFOLLOW walk and re-derive
  the final path from the fd. Every failure is -32011 with data.path =
  the *original* saveDir (never the resolved path). Residual TOCTOU on
  a pre-existing intermediate dir is documented and closed by CORE's
  O_NOFOLLOW open of the file.

veloxd_fs static lib; veloxd_rpc links it for the download.add wiring
next. Test veloxd.safepath is the adversarial table, on a real temp
tree. ASan+UBSan and TSan clean; 33 daemon/cli tests green.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
2026-09-10 19:37:01 +04:00
sami 1914eed7db merge: sched/ governor and schedule windows 2026-09-10 19:26:22 +04:00
samiandClaude Sonnet 5 b010139421 daemon: sched/ — the concurrency governor + schedule-window evaluation (build step 4)
The scheduling brain, built against CORE's headers (vdm/engine.hpp,
vdm/segment/budget.hpp) — signatures only; the Engine bodies land in
CORE stage 8 and the Scheduler that wires governor <-> store <-> engine
<-> timer comes after that (daemon/docs/deferrals.md D4).

- sched/governor — a pure decision function. In: a snapshot of every
  task's coarse RunState and every queue's state (schedule windows
  pre-resolved). Out: {to_start, to_resume, to_pause, pause_reasons,
  priority_order}. Enforces, all in TASK units per ADR 0011 §1:
  connection.maxConcurrentDownloads; the min(that, maxActiveSegments)
  clamp (§2); Queue.maxConcurrent; the per-host task cap (§4); and a
  stopped queue / closed window runs nothing. Never touches a task
  paused for `user` or CORE's `auto` (ADR 0013 §3) — only Schedule /
  QueueStopped / AdmissionReconcile are auto-resumable. Deterministic:
  main-list before queued-in-queue, then queue order, then FIFO, then
  task_id.
- sched/schedule_window — window_open(Schedule, local tm): disabled =>
  always open; `once` => date + time match; `periodic` => weekday in
  daysOfWeek (empty = every day) + time in [start, stop); null start =>
  midnight, null stop => end of day, stop < start => overnight window.
  Pure; re-evaluated every tick, no cached instants.
- veloxd_sched static lib; veloxd links it (nothing calls it yet).

Tests (ASan+UBSan and TSan clean): veloxd.sched_window (10 window
cases incl. overnight, once, null bounds), veloxd.sched_governor
(global/clamp/per-queue/per-host caps, stop vs window pause reasons,
resume-only-governor-reasons, auth-pause untouched, admission
reconcile, determinism under shuffled input). 32 daemon/cli tests
green; full tree green.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
2026-09-10 19:06:19 +04:00
sami ddb5db01f5 merge: lane/gui 2026-09-10 18:55:19 +04:00
sami dbdbd6049a merge: lane/ext 2026-09-10 18:55:19 +04:00
sami 1d7c7db076 merge: lane/daemon 2026-09-10 18:55:19 +04:00
sami b97ed5a5c6 merge: lane/core 2026-09-10 18:55:19 +04:00
sami 7289943375 merge: lane/proto 2026-09-10 18:55:19 +04:00
samiandClaude Sonnet 5 ea8fd7ca8f daemon: fold CORE's confirms into the engine-API review
CORE resolved all four (lane/core@3da4cd6): vdm::TaskId is hashable and
DAEMON never constructs one; DAEMON mkdir -p's save_path's parent
(engine -> Error::path_rejected if missing); sha512 added as the 4th
Checksum::Algo so no -32602 at the RPC edge; on_state(cancelled) then
on_finished(Err{Error::canceled}) -- note the one-L spelling in the
error taxonomy. Also notes rate/token_bucket + Engine::rate_limiter()
for the limiter.set wiring.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
2026-09-10 15:51:16 +04:00
samiandClaude Sonnet 5 3da4cd6e91 core: rate/token_bucket + fold in DAEMON's engine-API review (stage 7)
rate/token_bucket.hpp — a lazily-refilled TokenBucket (starts full: burst
then throttle, IDM behaviour; rate 0 == unlimited; burst caps idle
accumulation) and RateLimiter, the global -> per-queue -> per-task
hierarchy (docs/04 §6). acquire(task, n) peeks every applicable level and
commits on all-or-none so a blocked attempt never leaks tokens at a level
that had them; held under one mutex so a concurrent detach can't dangle
the bucket it's using. vdm/ids.hpp gains QueueId.

Tests: burst/refill/cap/unlimited for the bucket; tightest-level-binds,
no-partial-consumption, detach-safety, and an 8-thread aggregate-rate
check for the hierarchy. Green under ASan/UBSan and TSan.

Engine-API review (DAEMON signed off, no sched/ or dispatch rewrite):
 - Engine::rate_limiter() accessor added (limiter.set -> set_global_limit).
 - Checksum::Algo gains sha512 to match the wire Checksum set.
 - DownloadSpec: DAEMON creates save_path's parent dir before start();
   missing dir -> Error::path_rejected (made explicit).
 - cancel(): documented to fire on_state(_, cancelled, nullopt) then
   on_finished(Err{canceled}), in that order; download.cancel ==
   cancel(false), download.remove == cancel(true).
 - engine-api-m1.md: the five open questions resolved with DAEMON's
   answers (probe_hint optional, single cancel flag, {restart,
   keep_partial, abort} is the whole set, per-task 4 Hz is fine,
   refresh_url restarts all segments after a validating re-probe).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HPPSGhiArbvQgwC2DNiURS
2026-09-10 15:50:18 +04:00
samiandClaude Sonnet 5 207acb0c00 daemon: review CORE's engine API — sign off, answer the five questions
Review of core/docs/engine-api-m1.md (lane/core@d6cf1fe) in
daemon/docs/engine-api-review.md. Sign off: nothing forces a sched/ or
RPC-dispatch rewrite; the split matches ADR 0011 and ADR 0013.

Answers: (1) keep probe_hint optional — DAEMON has a ProbeResult only
on the File Info path; (2) one cancel(discard_partial), download.remove
is cancel(true) + DAEMON-side row/file cleanup; (3) {restart,
keep_partial, abort} suffices if the engine owns the mechanical 416
re-probe/re-split; (4) per-task 4 Hz progress is fine — DAEMON
re-batches across tasks for event.task.progress anyway; (5) refresh_url
restarts all segments on the new URL (the signed-URL case), mirror
rotation is spec.mirrors not refresh_url.

Four things to confirm, none blocking: vdm::TaskId copy/hash semantics
and that DAEMON never constructs one; who mkdir -p's save_path's
parent; sha512 (in the wire Checksum, not the engine enum) rejected at
the RPC edge; on_finished(Err{cancelled}) code + ordering vs
on_state(_, cancelled, _).

Integration timing: wire after sched/ lands. sched/ builds against
these signatures in parallel with CORE stage 8.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
2026-09-10 15:46:16 +04:00
samiandClaude Sonnet 5 dab071c41a daemon: rpc/ws_server — loopback WebSocket transport + pairing (build step 1, second half)
The extension's fallback transport (docs/05 §4). veloxd now also listens
on 127.0.0.1, first free port in 52000-52016, and writes it to
<runtime>/ws.port (0600).

- rpc/ws_frame — RFC 6455 frame codec. Incremental; reassembles
  continuation frames; enforces "client frames MUST be masked" (§5.1);
  caps a reassembled message at 8 MiB. This is the attacker-adjacent
  parser, so it has its own test table.
- rpc/ws_handshake — HTTP upgrade parse, Sec-WebSocket-Accept
  (SHA-1 + base64 via libcrypto), and the two non-negotiable checks:
  an Origin header must be present and must be moz-extension:// (a page
  cannot pair). Version must be 13.
- rpc/ws_server — per-connection Handshake -> Open state machine on the
  shared EventLoop. Token gate: session.pair mints a token behind the
  approver + rate limiter; session.hello must present a valid one;
  every other method is -32002 until authed. Privileged methods are
  refused -32003 by the generated dispatch(). Ping -> Pong; Close
  echoed. session.hello major-version mismatch -> -32001.
- rpc/pairing — PairingApprover interface + EnvAutoApprover dev stub
  (approves iff VELOX_PAIR_AUTO=1); PairingRateLimiter (5 failures / 60 s
  per origin, then 60 s lockout -> -32014, survives reconnect); a
  four-digit code generator.
- store/pairings — the pairings table: create() returns the plaintext
  token once and stores only its SHA-256; find_active_by_token,
  touch, revoke, list_active.
- util/crypto — sha1 / sha256_hex / base64 / random_token over libcrypto.
- store/sqlite — pin the DB file (and -wal/-shm) to 0600.
- runtime_dir — resolve_data_dir() for $XDG_DATA_HOME/velox (velox.db).
- main.cpp — opens + migrates velox.db, starts both transports; a WS
  bind failure is logged, not fatal (capture must fail open, the Unix
  socket still serves the GUI/CLI).

Real gap, flagged not hidden: the pairing prompt is EnvAutoApprover for
now — a GUI dialog / desktop notification is build step 7. Pairing
needs VELOX_PAIR_AUTO=1 until then.

Tests (ASan+UBSan and TSan clean): veloxd.ws_frame (codec + handshake
vectors incl. the RFC 6455 §1.3 accept sample), veloxd.pairings (token
create/find/revoke, hash-not-token, rate-limit window + lockout +
per-origin isolation + success reset), veloxd.ws_server (full flow: 101
handshake, -32002 gate, deny-then-approve pairing, hello-with-token,
-32003 privileged refusal, real download.list). 27 daemon/cli tests
green; full tree green.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
2026-09-10 15:44:28 +04:00
samiandClaude Sonnet 5 ec288db5ea ext: context-menus.ts — link/media menu items + grab-tab command
docs/05 §3: "Download with Velox" on a link (linkUrl) or a media element
(srcUrl) hands that one URL to download.add with the page as referrer; a
configurable command (Ctrl+Shift+U) does the same for the active tab. These
are explicit user requests, so they skip shouldCapture and the fail-open
path — a failure is surfaced to the user via notifications instead.

The page/selection "Download all links…" item waits on the content-script
link harvester (build-order step 7) and will be added there.

manifest: commands.velox-grab-current-tab. 9 tests. 110 total, lint clean.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_012Y9RU58hD1BuwP82DySUHk
2026-09-10 15:41:54 +04:00
samiandClaude Sonnet 5 51fa1201bd gui: category tree, menus, toolbar, splitter — build-order step 3
- CategoryPanel: the left tree — All Downloads / Unfinished / Finished,
  then Categories and Queues populated from category.list / queue.list,
  with per-node task counts. Selecting a node emits a TaskSelection.
- DownloadFilterProxy: QSortFilterProxyModel keyed off that selection.
  Client-side for M1 (the whole list fits); asTaskFilter() exposes the
  equivalent TaskFilter for a server-side download.list once paging lands.
- MainWindow: menu bar (Tasks / Downloads / View / Help) sharing QAction
  objects with the toolbar; QSplitter [panel | table]; Delete with a
  confirm; Resume/Pause All; View menu toggles the panel. Actions
  disabled while offline.
- Counts are computed off a throttled 400 ms timer, not the 4 Hz progress
  path. Fixed a debounce-vs-throttle bug found in the first screenshot:
  restarting the timer on every progress tick meant it never fired and
  the status bar sat at "0 of 0 downloads".
- First-run column widths that fit the content.
- tst_downloadfilterproxy: nodes filter to their own rows, the
  Finished/Unfinished split is correct, and the filter is dynamic (a row
  that finishes leaves the Unfinished node with no re-list). Verified
  against mockd --tasks 400 (screenshot: tree counts 75/84/89/83/69 sum
  to 400, status bar "400 of 400, 21 active").

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_016Ne28kx4VreeBWZv82Nksd
2026-09-10 15:41:29 +04:00
samiandClaude Sonnet 5 175185bbfc ext: capture/downloads-api.ts — downloads.onCreated safety net
Downloads that never reach the header hook (form POST results, service-worker
responses, clicks Firefox routes straight to its downloader) surface here.
If one looks like the daemon's, offer it FIRST and only cancel + erase
Firefox's copy on {action:"take"} — a failed or slow offer can never leave
the user with nothing. blob:/data: downloads are left to Firefox (the daemon
can't fetch a blob URL).

- offered-urls.ts: short-lived, bounded TTL set of URLs the header hook has
  already offered; the safety net checks it (via wasOffered) so nothing is
  double-handled. Hook gains an onOffered hook to populate it.
- background/index.ts: both paths share one offer(), getCookies, rules
  mirror, and OfferedUrls instance.

13 new tests incl. fail-open (offer rejects -> 'error', ignore ->
'offer_declined', cancel() throwing after take still returns 'taken').
101 tests green; web-ext lint clean.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_012Y9RU58hD1BuwP82DySUHk
2026-09-10 15:39:40 +04:00
samiandClaude Sonnet 5 d6cf1fe7dc core: post the engine API for DAEMON review (pre-stage-7)
The download entry point the AGENT-CORE brief asked for on day one and
that slipped. DAEMON has an RPC surface and a store and, until this is
agreed, nothing in velox::core to call.

vdm/task/download.hpp — DownloadSpec (the resolved subset DAEMON hands in:
absolute save_path, verbatim browser headers, requested segments/buffer,
optional probe_hint / checksum / auth, allow_resume), EngineState (the
CORE-owned subset of the wire TaskState), Progress / SegmentProgress,
DownloadCallbacks (on_progress <=4 Hz, on_state for every transition
incl. auto-pauses, on_auth_required, on_decision_needed, on_finished
last), DownloadHandle (pause/resume/cancel — idempotent per the ADR 0013
signature — plus provide_auth / decide / refresh_url, and synchronous
state()/progress() snapshots).

vdm/engine.hpp — Engine: start(spec, callbacks) -> handle, segment_budget()
(DAEMON's sched/ admission surface, ADR 0011), live connection.* setters,
a standalone probe() on the pool outside the segment budget.

core/docs/engine-api-m1.md — the review doc: field semantics, the state
machine, threading/lifetime rules (which thread callbacks arrive on, what
is legal from inside one, handle/engine lifetime), the shared-`paused`
idempotency contract as a signature, and five open questions for DAEMON.

Value types compile and are covered by api_compiles_test; Engine /
DownloadHandle bodies land in stage 8, built against whatever DAEMON
signs off here.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HPPSGhiArbvQgwC2DNiURS
2026-09-10 15:39:25 +04:00
samiandClaude Sonnet 5 23407974d2 ext: capture/index.ts — blocking onHeadersReceived hook, fail-open
For a response shouldCapture() likes: gather cookies, offer to the daemon
under a hard 750 ms budget, and return {cancel:true} ONLY on an explicit
{action:"take"}. Every other outcome — shouldCapture says no, daemon down /
slow / erroring, cookies fail, anything throws — resolves to {} and Firefox
downloads normally.

Fail-open tests written first (tests/capture/hook.test.ts): offer() rejects,
offer() never settles (resolves within budget), timeout-shaped rejection,
getRules() throws, malformed details. Plus the happy paths and a check that
the capture.offer payload is well-formed from details + stash + headers.

background/index.ts: wire the stash + hook onto browser.webRequest, mirror
capture rules via capture.getRules on connect and on a capture.* settings
change; DEFAULT_CAPTURE_RULES (enabled:false) until the first mirror lands.

84 tests green; web-ext lint clean.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_012Y9RU58hD1BuwP82DySUHk
2026-09-10 15:35:29 +04:00
samiandClaude Sonnet 5 1aed222ae2 docs: ADR-number convention in CLAUDE.md §7 — second merger renumbers
Six lanes pick ADR numbers with no allocator. This lane dodged one collision
by taking 0012 while DAEMON drafted 0011, and just hit a real one — two 0014s
in the same integration round. Codify what already happened in practice: take
the next free number in main's docs/adr/, and on collision whoever merges
second renumbers and fixes cross-refs rather than round-tripping.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_012fgjnqFCS5h5L7gZTZo3rV
2026-09-10 15:31:01 +04:00
samiandClaude Sonnet 5 dd1676bc0a proto: renumber ADR 0015 (0014 collided with PKG's), fix stale README versions
PKG landed docs/adr/0014-conformance-runs-through-ctest.md (dad88fc) and this
lane's 0014-generated-binding-changes-and-versioning.md landed in the same
integration round, both as 0014. Renumbered this one to 0015 — second merger
renumbers. Updated the two cross-references (contracts/README.md rule 4,
proto-answers-daemon-m1.md P1) and the in-file header.

Also while in contracts/README.md: the file-tree comment and the "Method
surface" heading still said v1.2.0; both now v1.4.0 to match VERSION.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_012fgjnqFCS5h5L7gZTZo3rV
2026-09-10 15:31:01 +04:00
samiandClaude Sonnet 5 e585113daf daemon: adopt HandlerResult<T> — real error codes from handlers (contracts/ 1.4.0)
Rebased onto main at 1.4.0. The regenerated Dispatcher returns
HandlerResult<T> = expected<T, HandlerError{code, message, data}>
(ADR 0014); the covariant-return break on all 39 overrides is the swap
predicted in daemon/docs/proto-requests-m1.md P1.

- dispatcher.hpp/.cpp: Result<T> -> HandlerResult<T> on every override;
  not_implemented() now returns HandlerError{InternalError, ...} rather
  than a ParseError forwarded as -32603.
- download.get: returns -32010 TaskNotFound with data.taskId. Not a
  placeholder — with no store, every id is genuinely not-found, which
  is the real answer for contracts/ fixture download.get.not-found. It
  becomes a store lookup when store/ is wired in.
- uds_roundtrip: the -32603-collapse guard is now a -32010 + data.taskId
  assertion, the regression guard the P1 note promised.

download.add (-32011) and download.probe (-32013) stay InternalError
until they have real bodies (canonicalization / probe); they get their
fixture codes when that logic lands.

All 24 tests green; uds_roundtrip TSan-clean.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
2026-09-10 15:31:00 +04:00
samiandClaude Sonnet 5 4b279e8271 daemon: store/ — SQLite WAL schema + forward-only migrator (build step 3)
The daemon's persistent state. SQLite in WAL mode, foreign keys on,
5 s busy timeout so a writer waits rather than SQLITE_BUSY under the
RPC loop.

- store/sqlite — RAII Db/Stmt over the C API; errors returned as
  DbResult<T> (std::expected), never thrown — the RPC loop must not
  unwind. transaction() helper: BEGIN / fn / COMMIT, ROLLBACK on error.
- store/migrations/0001_initial.sql — the eight tables from the brief:
  settings, categories, queues, tasks, segments, rules, history,
  pairings. Notable choices:
    * tasks columns project onto proto TaskSummary with no computation;
      requested vs effective segments/buffer split per ADR 0010/0012;
      pause_reason column per ADR 0013.
    * segments end_byte is NOT constrained >= 0 so a whole-file
      zero-length download is one row with end_byte = -1 (ADR 0010 B3a).
    * pairings stores only token_sha256 — the plaintext token is
      returned once from session.pair and never persisted (CLAUDE.md §4).
    * indices on tasks(state), (category_id), (queue_id, queue_position),
      (created_at), (completed_at) for the "1000 tasks, download.list
      under 50 ms" DoD.
    * six built-in categories + a Main queue seeded.
- store/migrations — runs every embedded migration past PRAGMA
  user_version, each in its own transaction, forward-only. SQL files
  are embedded at build time by cmake/embed_migrations.cmake.

Test veloxd.store_migrations (ASan+UBSan and TSan clean): fresh DB ->
head, all tables present, seed rows, FK cascade (segment orphan
rejected, task delete cascades), the end_byte=-1 zero-length case,
idempotent re-run, and forward-only from every released user_version.

Also: daemon/docs/proto-requests-m1.md — P1 marked landed on lane/proto
as 1.4.0 (HandlerError/HandlerResult), to be adopted in rpc/ once that
merges to main; P2 resolved.

Not linked into the running daemon yet — the store is wired to the
dispatcher when download.add/list/get get real bodies, next.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
2026-09-10 15:27:20 +04:00
samiandClaude Sonnet 5 0c7ce1437c cli: velox — add / ls / pause / resume / rm, with --json (build step 8, pulled forward)
The scriptable client, built now rather than last: it is how the daemon
gets exercised before the GUI is pointed at it (AGENT-DAEMON.md).

- src/client — synchronous blocking RPC over the Unix socket: resolve
  $XDG_RUNTIME_DIR/velox/velox.sock, connect, session.hello, one framed
  request/reply per call. Distinguishes transport failure (exit 3) from
  a daemon-returned error (exit 1).
- src/main — subcommands add/ls/pause/resume/rm; --json prints the raw
  JSON-RPC result or error; --dir/--out/--segments on add;
  --delete-file on rm. Usage errors exit 2.

ls works end to end against veloxd today (empty table). add and the
bulk verbs reach the daemon and surface its "not implemented" (-32603)
cleanly until the store lands — the plumbing is done, the commands
light up as handlers do.

Test velox.client: the real Client against an in-process UdsServer —
no-daemon path, session.hello, download.list, and a not-implemented
method surfacing as an RPC error rather than a transport error.
ASan+UBSan clean; full tree (21 tests, incl. conformance) green.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
2026-09-10 15:27:20 +04:00
samiandClaude Sonnet 5 e60d6669d8 daemon: rpc/ — Unix-socket transport + generated dispatch wiring (build step 1)
First real code in daemon/. veloxd now listens on
$XDG_RUNTIME_DIR/velox/velox.sock (0600, SO_PEERCRED same-UID check),
frames NDJSON, and routes every method through the generated
velox::proto::dispatch(). The CLI and GUI have a server to talk to.

Modules:
- rpc/ndjson.hpp    — newline-delimited framing, 8 MiB frame cap, CRLF-
                       tolerant, partial-tail buffering. Header-only, tested.
- rpc/event_loop    — single-threaded poll(2) reactor; never blocks the
                       loop. stop()/wake() are async-signal-safe (eventfd).
- rpc/runtime_dir   — $XDG_RUNTIME_DIR/velox resolution, 0700, owner-checked;
                       refuses an insecure fallback rather than using /tmp.
- rpc/uds_server    — listener + non-blocking per-conn read/write with
                       backpressure; handles session.hello (protocol-major
                       check -> -32001, sessionId, transport=uds) and
                       session.subscribe in the server layer; routes the
                       rest through dispatch().
- rpc/dispatcher    — VeloxDispatcher : proto::Dispatcher, all 39 methods.
                       download.list answers an empty table; the rest return
                       "not implemented" (-> -32603) until the store lands.
- main.cpp          — abstract-namespace single-instance lock, signal ->
                       clean shutdown, socket unlinked on exit.

Tests (ASan+UBSan and TSan clean):
- veloxd.ndjson         — framing edge cases
- veloxd.uds_roundtrip  — real socket: hello ok / version mismatch / empty
                          list / -32601 / -32700 / pipelined requests, and a
                          guard on the -32603 collapse documented in P1.

Known gap, filed not worked around: daemon/docs/proto-requests-m1.md P1 —
the generated Dispatcher has no error channel below -32603, so handlers
cannot yet return -32010/-32011/-32013 with their data payloads. The
server layer handles -32001/-32002/-32003 around dispatch(); genuine
in-handler errors collapse to -32603 until PROTO gives handlers a real
error return. Three error fixtures are non-conformant until then.

Not in this drop: rpc/ws_server (next; needs the store for hashed pairing
tokens), store/, sched/, cli/. WS reuses this event loop.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
2026-09-10 15:27:20 +04:00
sami 170bcfdb3e merge: ADR 0014 — generated binding changes and versioning 2026-09-10 15:26:30 +04:00
samiandClaude Sonnet 5 768f4f8e53 proto: ADR 0014 — versioning a generated-binding break with an unchanged wire
Records the rule 1.4.0 applied, so the next generated-binding retype cites it
instead of relitigating README rule 4's "retype → major + ADR" from scratch.

The rule: VERSION tracks the wire protocol, not any binding's API or ABI. A
change that leaves the wire byte-identical but breaks a generated binding's
source API (a C++ virtual's return type, a struct name) is a minor bump plus
a migration note — major would make session.hello refuse a client whose wire
behaviour is unchanged, which is worse than the problem. An ADR is still
required when the change encodes a design decision; "only one lane consumes
it" is not a reason to skip that, since GUI already links velox::proto and
the next such change starts with more than one consumer.

Rule 4 in contracts/README.md now points here so its "major + ADR" line
isn't read in isolation. proto-answers-daemon-m1.md's P1 writeup references
it as the durable home for the reasoning that was otherwise only in a commit
message.

Also names the nlohmann brace-init hazard the 1.4.0 work hit: json{nullptr}
is the array [null], not JSON null, so HandlerError::data is `= nullptr`. The
kind of thing a regeneration reintroduces; caught here only by an end-to-end
assertion on dispatch() output, which is called out to keep.

Docs only — no schema, VERSION, or generated-code change.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_012fgjnqFCS5h5L7gZTZo3rV
2026-09-10 15:24:53 +04:00
sami bb138169d5 merge: lane/pkg-qa 2026-09-10 15:18:43 +04:00
sami 6c94df0441 merge: lane/core 2026-09-10 15:18:43 +04:00
sami f1fb669209 merge: lane/proto 2026-09-10 15:18:43 +04:00
samiandClaude Sonnet 5 5d81b4cdae core: segment/segmenter + segment/budget (stage 6)
vdm/ids.hpp — TaskId, an opaque engine handle (DAEMON keeps the wire
UUID <-> TaskId map; the engine never sees the UUID).

segment/segmenter — per-download range management (docs/04 §3). Initial
lazy split; assign_slot() splits the largest remaining range when the
budget grants a slot; on_complete(may_steal) either *steals* the second
half of the largest remaining range for the same worker (slot-neutral) or
returns nullopt so the caller *yields* the slot (ADR 0011 A1); on_failed()
returns requeue only on the 3rd consecutive connection error with a mirror
present — the remaining range is orphaned and re-split. Non-resumable or
unknown-size => exactly 1 segment; never split below min_segment_bytes
(1 MiB). Resume ctor rebuilds from a persisted table (falls back to a
fresh layout if it doesn't tile [0,total)). One mutex == "the task lock";
segment fields are std::atomic and the store is a std::deque so a steal's
append never moves a worker's record.

segment/budget — the global allocator (ADR 0011). Owns exactly one
ceiling (maxActiveSegments) and min-1-before-seconds fairness: a two-pass
allocation (guarantee pass gives every wanting task 1 slot in DAEMON's
priority order, then a growth pass round-robins the rest up to each
task's effective cap = min(per_task_cap, host cap, 1 if non-resumable)),
recomputed from scratch on every edge so a live set_max_active_segments
cut naturally yields the excess lowest-priority-first, never a
mid-segment kill. DAEMON-facing surface exactly as promised in
daemon/docs/core-requests-m1.md / ADR 0011: budget(), segments_active(),
starved_tasks(), starved_since(), set_max_active_segments (drain),
set_host_segment_cap, set_task_order, on_budget_changed (a jthread
coalesces at <=4 Hz; the tasks_starved 0<->nonzero edge fires
immediately). Callbacks are copied out and run after the lock is
dropped.

Tests: segmenter split/steal/requeue/resume math + a concurrent
steal-and-advance run; budget min-1 under a tight budget, round-robin
growth, host-cap and non-resumable clamps, live-lower shedding
lowest-priority-first, starvation below the task count, starved-edge
notification, and a concurrent set_want hammer. Green under ASan/UBSan;
the steal path and the budget green under TSan.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HPPSGhiArbvQgwC2DNiURS
2026-09-10 15:14:14 +04:00
samiandClaude Sonnet 5 5e3e21543a proto: give the generated C++ Dispatcher a real error channel (P1, 1.4.0)
DAEMON's daemon/docs/proto-requests-m1.md P1: velox::proto::Dispatcher's
on_* methods returned Result<T> = expected<T, ParseError>, and dispatch()
mapped every handler error to -32603 InternalError. A handler had no way to
return -32010 (download.get not-found), -32011 (download.add invalid-path)
or -32013 (probe-failed) with their data payloads -- three error fixtures a
conformant server must satisfy were unreachable, blocking DAEMON's
"conformance as a server" M1 DoD.

Two error channels now, kept separate on purpose:
  - parse: Result<T> / ParseError -- dispatch() failing to turn the wire into
    typed params. Always -32602, always structural.
  - handler: HandlerResult<T> / HandlerError -- a handler deciding the request
    can't be fulfilled. Carries any ErrorCode + message + free-form data.

    struct HandlerError {
        ErrorCode code{ErrorCode::InternalError};  // bare {} is a valid -32603
        std::string message;
        nlohmann::json data = nullptr;             // straight into the error's data
    };
    template <class T> using HandlerResult = std::expected<T, HandlerError>;

dispatch()'s handler branch is now
  make_error(id, r.error().code, r.error().message, r.error().data)
instead of a hard-coded InternalError. -32001/-32002/-32003 stay the server
layer's to raise around dispatch(), as DAEMON already does.

Verified end to end against the real dispatch() path: a handler returning
TaskNotFound/InvalidPath/ProbeFailed produces -32010/-32011/-32013 with the
data object intact, and a bare HandlerError{} still yields a clean -32603
with no data field. The `= nullptr` on the member (not `{nullptr}`) matters:
brace-init of nlohmann::json from nullptr is the array [null], not JSON null.

FixtureDispatcher regenerated to HandlerResult; conformance_main.cpp only
inspects dispatch()'s JSON and needed no change. TS side is untouched beyond
the version string -- no server Dispatcher is generated there.

P2 also handled: session.hello.version-mismatch's data.expected was a stale
"1.0.0"; now $any, with a note that the error-fixture compare is on `code`
only so a server echoing kProtocolVersion there is fine.

Version: minor, 1.3.0 -> 1.4.0. Wire is byte-identical (no schema, fixture,
or OpenRPC change) but every Dispatcher implementer must swap Result ->
HandlerResult on regen, and the bump is how lanes are told to. Not an ADR:
one lane consumes this binding, it's the one that asked, and the shape is
the one they proposed. Answered in contracts/proto-answers-daemon-m1.md.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_012fgjnqFCS5h5L7gZTZo3rV
2026-09-10 15:10:13 +04:00
samiandClaude Sonnet 5 39867f4f92 pkg: document the GUI M1 DoD gate wiring (blocked on GUI's harness)
R3: the three GUI DoD gates (10k rows at 60fps, flat RSS over 10 min,
--slow/--flaky/--drop-connection recovery) have nowhere to run. GUI owns the
harness, PKG/QA owns the job. tests/integration/README.md records the wiring
contract — driver invocation, exit-code and --json semantics — and carries
the pre-drafted per-PR and nightly job stanzas with TODO(GUI) markers for the
harness path. Wire for real when GUI files the follow-up.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0143aKiohmDiyefJBwHDJJqw
2026-09-10 14:58:10 +04:00
samiandClaude Sonnet 5 dad88fce3f pkg: run conformance through ctest as the one canonical path
PROTO wired the suite into ctest (label "conformance": the end-to-end
`conformance` test that shells to run.sh, plus the native `conformance_cpp`);
the CI job called run.sh directly. Two entry points, and the required M0 gate
exercised only one of them, so the ctest registration could rot.

The conformance job now configures, builds velox_conformance_cpp, and runs
`ctest --preset dev -L conformance`. The dev test preset's
noTestsAction: error is the rot guard: an empty label match exits non-zero
instead of the old silent pass. build/sanitizers exclude the heavy e2e test
with -E '^conformance$' (the dedicated job owns that run; conformance_cpp
still runs under every sanitizer). ADR 0014 records the decision.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0143aKiohmDiyefJBwHDJJqw
2026-09-10 14:58:10 +04:00
samiandClaude Sonnet 5 57b06ea5b1 pkg: ignore libFuzzer crash artifacts
libFuzzer writes crash-* / oom-* / leak-* / timeout-* into CWD on a find and
each holds the crashing input verbatim. None were ignored; CORE caught two by
hand before committing. Prevention, not cleanup.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0143aKiohmDiyefJBwHDJJqw
2026-09-10 14:58:10 +04:00
samiandClaude Sonnet 5 82a2f11d7a pkg: make bootstrap --check validate apt names, and run it on 26.04
--check verified outcomes — binaries on PATH, pkg-config modules — on a box
that already installed everything. It never looked at the APT_* names, which
is the only part that breaks on a clean machine of the wrong release, as R1
just showed. Add a loop over the assembled PKGS array that fails on any name
with no installable candidate (apt-cache policy; no root, no network).
All-missing is treated as stale lists (warn), not 36 bad names.

The bootstrap-script job runs on a 24.04 runner, where the bad name still
resolves, so name validation there checks the wrong archive. Add
bootstrap-script-2604: a real --with-clang install in an ubuntu:26.04
container — the release the project ships on, and the first time the
fuzz-toolchain half of bootstrap is exercised anywhere (CORE had been running
clang++-21 directly). Also pass --with-clang/--packaging to the 24.04 --check
so the optional and M6 names can't rot unnoticed. BRANCH_PROTECTION.md gains
the 2604 row and the stale "when X merges" rows are corrected to "now".

gui/docs/pkg-qa-requests-m1.md R2 + the --with-clang note.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0143aKiohmDiyefJBwHDJJqw
2026-09-10 14:58:10 +04:00
samiandClaude Sonnet 5 99b429abfa pkg: fix qt6 svg dev package name — libqt6svg6-dev has no 26.04 candidate
`apt-cache policy libqt6svg6-dev` is "Candidate: (none)" on 26.04; the package
that carries the Svg headers and Qt6SvgConfig.cmake is qt6-svg-dev. Since
gui/CMakeLists.txt landed on main the root build's
find_package(Qt6 ... Svg REQUIRED) is live, so a clean 26.04 box could not
configure the project at all. Same name in README.md and AGENT-PKG-QA.md.

Reported in gui/docs/pkg-qa-requests-m1.md R1.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0143aKiohmDiyefJBwHDJJqw
2026-09-10 14:58:10 +04:00
samiandClaude Sonnet 5 9dce588456 proto: link velox::proto in the C++ conformance runner, per CORE's request
core/docs/proto-requests-conformance-cmake.md: the runner's header comment
said it links libveloxproto, but it actually compiled
core/generated/velox_proto.cpp straight into the executable and found
nlohmann_json itself — the only option while ADR 0009's target didn't exist.
It exists now on main as velox::proto (core/CMakeLists.txt, PUBLIC generated
include dir, PUBLIC nlohmann_json).

if(TARGET velox::proto): link it. else: fall back to compiling the generated
.cpp directly, for a configure with no core/ in the tree. Both paths verified
— full tree links libveloxproto.a (compiled once, by veloxproto's own
target); with core/CMakeLists.txt hidden the fallback compiles the .cpp and
finds nlohmann itself. conformance_cpp passes either way.

Beyond tidiness: once GUI links velox::proto too, the conformance runner
linking the same target is what guarantees the suite and the clients exercise
byte-identical generated code, rather than two compiles of one .cpp under two
warning configs — the exact skew a conformance suite exists to catch.

run.sh's own direct g++ compile is unaffected and stays independent by
design; this only changes the ctest-driven path CI and lanes use.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_012fgjnqFCS5h5L7gZTZo3rV
2026-09-10 14:50:26 +04:00
sami 1fabaf805d merge: lane/gui 2026-09-10 14:39:36 +04:00
sami e6f070f0dd merge: lane/ext 2026-09-10 14:39:36 +04:00
sami efd5532862 merge: lane/core 2026-09-10 14:39:36 +04:00
samiandClaude Sonnet 5 05b650b8ec ext: capture/rules.ts — shouldCapture() decision table
The header-hook manager's core call: capture when any positive signal holds
and none of the vetoes do (docs/05 §2). Vetoes are absolute and checked
first — a monitored .zip on an excluded host is not captured.

Decision table written first (tests/capture/rules.test.ts, 22 rows); every
branch here exists to satisfy one. Covers the M1 DoD set: attachment,
monitored extension, monitored MIME, size threshold, excluded host (exact +
wildcard), HTML navigation, blob:/data: origin, bypass modifier, streaming
media (HLS MIME and resourceType 'media'), a page-issued range request, plus
non-GET, redirect status, sub-threshold, and large-but-renderable.

Pure function of (candidate, rules); rules are the daemon's, mirrored via
capture.getRules, so the decision never drifts from daemon policy.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_012Y9RU58hD1BuwP82DySUHk
2026-09-10 13:55:05 +04:00
samiandClaude Sonnet 5 8844dd616b gui: amend PKG/QA request (R1 escalation, R3 restated) + EXT grep-check note
Review feedback on the filed request:

- R1: gui/CMakeLists.txt is on main now, so root's
  find_package(Qt6 ... Svg REQUIRED) is live — a missing SVG dev package
  is a hard configure failure for the whole project, not a skipped guard.
  Added the reason CI hasn't caught it: ubuntu-latest is 24.04 (where
  libqt6svg6-dev likely resolves), the project targets 26.04 (where it
  does not). Wrong name + runner/target release mismatch = the class of
  bug PKG/QA owns is currently unobservable in CI. That's the argument
  for R2, folded in.
- R3: corrected — CI does build the GUI and runs its three ctests under
  the ci preset. What has no home is the non-unit-test DoD: 10k-row
  60fps, flat RSS over a 10-minute run, and mockd
  --slow/--flaky/--drop-connection recovery. Asked for those specifically.
- gui/docs/ext-requests-m1.md: CLAUDE.md §3 says the no-download-logic
  rule applies to extension/ too; GUI made its half an executable ctest,
  EXT's half is still prose. Suggested the ESLint equivalent for the
  existing extension-lint job.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_016Ne28kx4VreeBWZv82Nksd
2026-09-10 13:53:51 +04:00
samiandClaude Sonnet 5 092e99f7a0 core: meta/veloxpart — resume sidecar, reader first + fuzzed (stage 5)
util/crc32.hpp — header-only CRC-32 (zlib polynomial, reflected), used to
integrity-check the sidecar.

meta/veloxpart — the <name>.veloxpart.meta resume file (docs/04 §5).
Little-endian, versioned, CRC-32 over the whole record. Layout: magic,
version, flags, total_size, downloaded, url set (original/effective/
mirrors), etag/last-modified/content-type, segment records (start, end
INCLUSIVE, completed), optional sha256 streaming-hash blob.

parse_veloxpart() is the attacker-facing surface (the file sits in a
world-writable-ish download dir) and is total on any byte string: CRC
checked before any field is interpreted; magic, a version it understands,
every count and length bounded by a hard cap AND checked against the
remaining buffer; ByteReader latches on overrun; trailing bytes rejected.
Every malformation is meta_corrupt / meta_version_unsupported, never a
crash or an unbounded allocation. serialize_veloxpart() is deterministic
(unchanged sidecar isn't rewritten). File helpers write atomically
(temp + rename) and fdatasync the file and its directory.

Tests: crc32 known vector; full + minimal round-trips; deterministic
serialize; file round-trip; and a truncation/corruption table — bad
magic, CRC mismatch (payload and CRC-field flips), future version,
truncation at every stage, hostile url_count / segment_count / lp_string
length (the case the brief singles out), trailing bytes, impossible
segment.completed.

tools/fuzz/fuzz_veloxpart — feeds raw bytes and bytes-with-valid-CRC
(so the field parser and ByteReader bounds checks are actually reached),
and round-trip-stability-checks anything accepted. Ran 1.1M execs clean
under ASan+UBSan+libFuzzer (clang++-21); fuzz_content_disposition and
fuzz_url likewise re-run to 1.1M. tools/fuzz gains a -runs=0 seed-replay
CTest smoke per target (regression tripwire; the campaign stays manual).

Fuzz-found and fixed: parse_content_disposition could emit a filename
containing NUL / control bytes from a mangled filename* ext-value —
strip_path only removed path separators. Now sanitize_leaf() also drops
C0 controls and DEL (rules/ still owns the authoritative sanitize; `..`
and printable-unsafe content pass through as before).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HPPSGhiArbvQgwC2DNiURS
2026-09-10 13:52:34 +04:00
samiandClaude Sonnet 5 801bcaae12 ext: capture/headers.ts — request-header stash
onHeadersReceived (where shouldCapture runs) doesn't carry the headers the
browser actually sent; signed-URL and referrer-gated CDNs need them. Stash
from onBeforeSendHeaders keyed by requestId, read back at capture time.

This map sees every request, so it is bounded both ways — oldest-out past a
size cap (default 2048) and a 5-minute TTL, swept on a 60 s timer and on read
— and attach() clears an entry the moment its request completes or errors.

normalizeHeaders(): Array<{name,value}> -> lower-cased map, repeats joined
with ", ". 13 tests: eviction order, redirect re-put refresh, TTL expiry,
sweep, and the onBeforeSendHeaders/onCompleted/onErrorOccurred wiring.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_012Y9RU58hD1BuwP82DySUHk
2026-09-10 13:52:07 +04:00
samiandClaude Sonnet 5 c5d596d93b ext: MV3 manifest + esbuild build; drop polyfill for Firefox's browser global
Firefox-only extension, so webextension-polyfill (a Chrome shim) is dead
weight and forces a bundler just to resolve one bare import. Use the native
`browser.*` global with @types/firefox-webext-browser instead.

  - manifest.json: MV3, event-page background (dist/background.js), the
    docs/05 §7 permission set (<all_urls> in host_permissions),
    strict_min_version 128.0, data_collection_permissions none.
  - scripts/build.mjs: esbuild bundle of src/background/index.ts -> dist/,
    esm, target firefox128. Wired to `prepare` so `npm ci` produces the
    bundle and CI's `web-ext lint` (which needs it to exist) passes with no
    added CI step. dist/ stays gitignored.
  - src/background/index.ts: event-page entry — brings the transport up,
    holds the shared reference. Capture surfaces attach here next.
  - transport/storage.ts, transport/index.ts: use the browser global.
  - tests/setup.ts: stub the browser global instead of mocking a module.

web-ext lint clean (0/0/0). typecheck clean. 38 tests still green.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_012Y9RU58hD1BuwP82DySUHk
2026-09-10 13:50:23 +04:00
samiandClaude Sonnet 5 90e2580b05 ext: S1 native-messaging spike (ADR 0003) + WebSocket transport
Spike S1 — run on the target machine through real snap confinement
(apparmor snap.firefox.firefox enforced; web-ext's direct-exec of the inner
binary bypasses it, so runs were forced through `snap run firefox`):

  - manifest in ~/.mozilla/native-messaging-hosts/  -> WORKS; host launched
    unconfined with real $HOME and real $XDG_RUNTIME_DIR, bound a socket in
    the real /run/user/<uid>. Corroborated by the machine's 1Password host.
  - ~/snap/firefox/common/.mozilla/native-messaging-hosts/  -> not read
  - /usr/lib/mozilla/native-messaging-hosts/               -> not read
  - flatpak path                                           -> N/A (snap Firefox)

Decision: WebSocket stays the default; native messaging is an opportunistic
upgrade taken only when its handshake succeeds. docs/05 §4 corrected in this
commit to point the snap manifest at ~/.mozilla and mark /usr/lib as
deb/tarball-only. ADR carries a self-contained reproduction; the scratch
harness has been removed.

transport/ (build order item 1):
  - types.ts        VeloxTransport interface + error taxonomy
  - rpc.ts          JSON-RPC id correlation, per-call deadline, AbortSignal
  - backoff.ts      exponential backoff with jitter
  - discovery.ts    52000-52016 scan ordering (last-good port first)
  - websocket.ts    scan -> session.hello -> auto-pair (token in
                    storage.local) -> reconnect; -32001 fatal, refused/
                    rate-limited pairing latches needsPairing (no retry storm);
                    a mid-handshake drop aborts hello immediately
  - native.ts       connectNative(); distinguishes "not installed" (fatal,
                    lets the picker fall through) from a crash (reconnect)
  - index.ts        createTransport() runtime picker + persisted Options override

Toolchain: package.json / tsconfig (strict) / vitest; webextension-polyfill
mocked. 38 tests, incl. the WS suite against a real loopback ws server.
No manifest.json yet, so CI's extension-lint guard stays a no-op.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_012Y9RU58hD1BuwP82DySUHk
2026-09-10 01:17:43 +04:00
sami 2e3251f0b5 merge: GUI real checks — RTL, no-download-logic, model patches 2026-09-10 01:15:46 +04:00
samiandClaude Sonnet 5 2959b0f707 gui: real RTL + no-download-logic checks; split into velox-gui-lib
Follow-up hardening after a review noted the RTL "check" verified nothing
(a .ts stub that no test loads), matching a session-wide pattern of
checks written against what should be true rather than what would break.

- Split the non-main() code into velox-gui-lib (STATIC) so tests link the
  real widgets/models, not a reimplementation.
- tst_rtl: builds the real MainWindow, flips layoutDirection, asserts the
  direction propagates to the central widget AND that the offline-banner
  QHBoxLayout actually mirrors (label x-position LTR vs RTL differs by
  >100px). Verified it fails when the banner is pinned LtR.
- gui_no_download_logic: a ctest that greps gui/src for curl_*/pwrite/
  sqlite/QSqlDatabase/QNetworkAccessManager and fails on a hit — CLAUDE.md
  §3 as an executable check. Verified it fails when a curl_ token is added.
- Still uncovered (noted, not claimed): that the translation catalogue
  loads and the right context/strings resolve at runtime.

Not covered here because the files are PKG/QA-owned: tools/bootstrap.sh
ships a package name that does not exist on 26.04 (libqt6svg6-dev; the
real one is qt6-svg-dev), and --check validates pkg-config outcomes
rather than the apt names it would install. Both, plus the same name in
AGENT-PKG-QA.md and the README, are written up apply-ready in
gui/docs/pkg-qa-requests-m1.md.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_016Ne28kx4VreeBWZv82Nksd
2026-09-10 01:12:06 +04:00
sami 9025085d82 merge: GUI rpc client, table model, main window 2026-09-10 01:03:24 +04:00
samiandClaude Sonnet 5 2a87abe96d gui: RPC client, download table model, and a live main window
First vertical slice of velox-gui, built entirely against tools/mockd
(no daemon dependency):

- rpc/: RpcConnection runs a QLocalSocket on a worker thread with
  newline-delimited JSON-RPC framing, drives the session.hello /
  session.subscribe handshake, and reconnects with exponential backoff
  (250 ms -> 8 s). RpcClient is the main-thread face: marshals calls onto
  the worker, delivers replies as main-thread callbacks, re-emits server
  notifications as typed Qt signals, and issues the one-shot download.list
  on reaching Connected.
- models/DownloadTableModel: QAbstractTableModel over TaskSummary. A
  progress batch is a row patch with a narrow dataChanged over the value
  columns only; beginResetModel() is reserved for the initial load and a
  reconnect resync.
- widgets/ProgressDelegate: in-cell progress bar for the Status column.
- mainwindow/MainWindow: the table, a status-bar connection dot, an
  offline banner instead of a modal, dialog-free pause/resume/stop
  actions, and QSettings column/geometry persistence.
- gui/CMakeLists.txt links velox::proto (never velox::core, ADR 0009) and
  self-guards on the veloxproto target so main keeps configuring if it is
  ever absent again.
- i18n from the first commit: every string via tr(), plus an Arabic .ts
  stub for the RTL check.
- tests/: headless QTest for the model — proves the progress patch is a
  narrow dataChanged and never resets the model.

Verified end-to-end against `mockd --tasks 300`: handshake, initial list,
live progress batches applied to the model, and a clean
Reconnecting -> Connected recovery when mockd is bounced mid-run.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_016Ne28kx4VreeBWZv82Nksd
2026-09-10 00:58:53 +04:00
sami 850e85de2a merge: libveloxproto target and stage 4 io layer 2026-09-10 00:46:12 +04:00
samiandClaude Sonnet 5 bd0a24c87a core: io/sparse_file + io/write_buffer (stage 4)
io/sparse_file — the single O_WRONLY output file (docs/04 §4). open()
posix_fallocate's the full size (falls back to ftruncate on
EOPNOTSUPP/ENOSYS, reported via preallocated()); write_at() pwrites at an
absolute offset, looping short writes and retrying EINTR; sync() is
fdatasync (timer/pause only); advise_dontneed() is
posix_fadvise(DONTNEED); resize() trims a preallocated tail or sizes a
chunked download. errno -> vdm::Error (ENOSPC->disk_full,
EACCES->permission_denied, ENOENT/ENOTDIR/...->path_rejected). No lock on
write_at — POSIX makes each pwrite atomic for a regular file, so N
segment threads writing disjoint ranges is safe (tested, TSan-clean).

io/write_buffer — per-segment accumulate-and-flush buffer, preallocated
at construction; append() only memcpys (no allocation on the write-
callback hot path, docs/04 §8 — asserted by a global-new counter in the
test). Flushes on fill via a caller-supplied FlushFn; a chunk >= capacity
arriving on an empty buffer writes straight through. On a flush error
next_offset() stays at the last durable position. Single-threaded; the
disk-writer-thread handoff is stage 8.

Also: vtest.hpp VT_CHECK_EQ/NE now copy operands (auto, not auto&&) — an
assertion must not outlive a temporary the expression returned a
reference into (ASan caught this on Result<void>{}.error().code).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HPPSGhiArbvQgwC2DNiURS
2026-09-10 00:41:31 +04:00
samiandClaude Sonnet 5 e8a6b64c2f core: file conformance-cpp CMake correction for PROTO
tests/conformance/cpp/CMakeLists.txt (PROTO's lane) compiles
core/generated/velox_proto.cpp directly while its comment claims it links
libveloxproto. Now that the veloxproto target exists it should link
velox::proto, with a TARGET-guarded fallback to the direct-compile for
standalone configures. Filed, not edited — not CORE's file.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HPPSGhiArbvQgwC2DNiURS
2026-09-10 00:31:50 +04:00
samiandClaude Sonnet 5 bd98fe42f9 core: add the libveloxproto target (ADR 0009)
core/ now produces two libraries as ADR 0009 specifies:
 - veloxcore  — the engine; still links only Threads + CURL, no JSON.
 - veloxproto — generated/velox_proto.cpp, generated/ as a PUBLIC include
   dir, nlohmann_json linked PUBLIC. velox::proto alias.

Consumed by veloxd / CLI / GUI / the conformance runner; veloxcore must
never link it. nlohmann_json is found here too (the root only finds it
when daemon/ has landed) so core builds standalone. Generated code is
built -Wall -Wextra -Wno-error — it is committed and never hand-edited, so
a codegen quirk must not break the build. A configure-time FATAL_ERROR
trips if veloxcore ever links veloxproto.

Verified: libveloxproto.a builds clean; veloxcore's link deps contain no
proto/nlohmann; full suite green.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HPPSGhiArbvQgwC2DNiURS
2026-09-10 00:31:02 +04:00
sami 3588e3b0db merge: lane/pkg-qa 2026-09-10 00:09:32 +04:00
sami 09b6945bd1 merge: lane/daemon 2026-09-10 00:09:32 +04:00
sami 42d5fbb8fe merge: lane/core 2026-09-10 00:09:32 +04:00
sami 6d44767373 merge: lane/proto 2026-09-10 00:09:32 +04:00
samiandClaude Sonnet 5 27e9ce9fb5 daemon: mark ADR 0013 fully accepted — PROTO landed the error-on-paused widening
PROTO closed the one remaining contract gap as contracts/ 1.3.0
(lane/proto commit 6db304a): event.task.state.error / TaskSummary.error
now populate on a paused transition CORE entered unilaterally, not just
on failed/retry_wait. Minor widening of an existing field's presence
condition, no retype, no new field, per contracts/README.md rule 4.

Updates every place in the ADR that referred to this as an open
question or unresolved gap: the status line, the pause-reason
bookkeeping in §2, the alternatives-considered pointer, and the
contract-gap section itself (renamed "surfaced, now closed"). Notes
1.3.0 is on lane/proto but not yet merged to main (still 1.1.0) —
daemon/src/sched/'s pause/resume logic should be written once that
merge lands, not before.

All four of ADR 0013's open items are now resolved: CORE confirmed
tasks_starved's structural exclusion and pause()/resume() idempotency
explicitly (verdict: "accept as written", not hedged), and adopted
"auto-pause" with no new wire term.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
2026-09-10 00:07:28 +04:00
samiandClaude Sonnet 5 d3ab00d67c proto: fix daemon/engine attribution in two buffer-clamp descriptions
DAEMON's rebase audit caught it: TaskDetail.effectiveBufferBytes said "the
daemon reduces every live segment's buffer" to fit maxTotalBufferBytes, but
ADR 0011's ownership table (line 55) assigns bufferBytes/maxTotalBufferBytes
to CORE in bytes-units -- DAEMON counts tasks, CORE counts segments and
bytes. "The engine" is correct.

Same error, same root cause, in DownloadSpec.segments: "the daemon lowers it
to the per-host cap" attributes the per-host *segment* cap to DAEMON, but
that's CORE's (ADR 0011 line 54, "CORE enforces per-host segment caps -- it
owns the connections and is the only place segments are counted"). DAEMON's
own per-host cap is a *task*-level admission cap (line 50), a different
thing entirely -- conflating the two in the schema's own prose is exactly
how the clamp ends up implemented twice, once in each lane, disagreeing.

Description-only, no version bump: the JSON Schema shape is untouched, only
which component the prose names as doing the reducing. Regenerated code
diffs are comment-only (doc comments in the generated header and TS types).

Checked every other buffer/segment-clamp description for the same mistake;
the rest either already said "CORE"/"the engine" or used passive voice that
doesn't misattribute (Settings.connection.maxTotalBufferBytes,
Settings.connection.bufferBytes, docs/04, ADR 0012).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_012fgjnqFCS5h5L7gZTZo3rV
2026-09-10 00:07:05 +04:00
samiandClaude Sonnet 5 62cda074c1 daemon: fold in CORE's sign-off on ADR 0013
CORE accepted ADR 0013 as written, no amendments
(core/docs/adr-0013-core-response.md, lane/core@c65e664). Folds in:

- pause()/resume() idempotency contract, precisely: no-op success on
  an already-paused task, ALSO on a terminal task (pause racing
  completion isn't an error), resume() no-op on a non-paused task, the
  only error is task_not_found, and no state-change event fires for a
  no-op call.
- tasks_starved pinned as {connecting, downloading} AND
  segments_active == 0 — a structural exclusion of retry_wait and
  auto-paused tasks rather than a special case, with the full state
  table CORE gave.
- restart handling confirmed fully; two non-blocking notes from CORE
  about work-interruption during verifying/assembling.
- "auto-pause" adopted as the term, no new wire/API surface.

Status updated: accepted by CORE; PROTO's item 3 (permit `error` on
event.task.state when state=="paused") is the one remaining blocker
before daemon/src/sched/'s pause/resume logic can be written
correctness-preservingly.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
2026-09-10 00:05:11 +04:00
samiandClaude Sonnet 5 6db304a0ae proto: widen error-on-paused for ADR 0013's auto-pause signal (1.3.0)
DAEMON's docs/adr/0013-task-state-machine-ownership.md needs a wire signal
for the difference between a paused task the daemon entered unilaterally
(auth_required, server_file_changed, disk_full) and one that was requested
(user, schedule, queue stop, admission reconcile) -- without it, DAEMON's §3
resume rule ("resume only when the reason matches the event that justifies
resuming") has nothing correctness-preserving to key on, and would have to
guess from timing. CORE has already accepted the ADR; this was the sole
remaining blocker per DAEMON's own status line on it.

No retype, no new field -- error was already TaskError | null on both
event.task.state and TaskSummary, exactly as DAEMON characterized the ask.
Only the *description* of when it is populated widens: previously "failed or
retry_wait", now also "paused, when the daemon entered it on its own
initiative". A deliberate pause still carries error: null. TaskError's own
top-level description gets the same widening, since it previously also said
"failed or retry_wait" and would otherwise contradict the field that embeds
it.

New fixture (event.task.state.auto-paused.json) exercises the case directly:
an auth_required pause with error populated, contrasted in its own
description against download.pause.json's error: null for a requested pause.
The existing event.task.state.json fixture's first assertion was stale
("error is present exactly when failed or retry_wait") and is corrected.

Minor bump, 1.2.0 -> 1.3.0: a description widening on an already-nullable,
already-optional field changes no JSON Schema shape, but it is a real
behavioral commitment change worth a version bump so downstream regenerates
and notices, per the same reasoning ADR 0010 applied to TaskErrorCode.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_012fgjnqFCS5h5L7gZTZo3rV
2026-09-10 00:04:35 +04:00
samiandClaude Sonnet 5 203d4a662f proto: wire the conformance suite into ctest so CI actually runs it
Verified PKG's landed CI (.github/workflows/ci.yml, db7650b/8f45815) against
this lane's actual tree, per the standing instruction that conformance as a
required check is this lane's DoD to verify, not PKG's. It wasn't running:
the `conformance` job's presence-check looks for tests/conformance/CMakeLists.txt
or tests/conformance/package.json, and neither existed -- the job was
silently short-circuiting to a green "skipped" on every PR, forever. The M0
exit gate was not gating anything.

tests/conformance/CMakeLists.txt registers one ctest entry, labeled
"conformance", that shells out to run.sh -- the exact command
tests/conformance/README.md tells a human to run locally, so there is one
definition of "the suite passed", not a CMake-flavoured near-duplicate of it.
cpp/CMakeLists.txt's existing conformance_cpp test gets the same label, for a
lane iterating on core/generated/ who wants the fast native-only path.

Fixed a second landmine found while wiring this: the root CMakeLists.txt only
find_package(nlohmann_json)'s when daemon/CMakeLists.txt exists, since
daemon is its real consumer -- but daemon hasn't landed yet, so
add_subdirectory(tests/conformance) would have failed to configure the
moment this file existed, on every machine, until daemon merges. Fixed inside
tests/conformance/cpp/CMakeLists.txt with an if(NOT TARGET) guard rather than
widening the root file's condition, which is PKG's to change.

run.sh now installs its own Python deps (jsonschema, referencing) on demand:
they aren't in tools/bootstrap.sh's apt list -- that's PKG's script, these
are this suite's own dependency -- so a bare CI image would otherwise fail
check_contract.py with an ImportError before this suite even started.

Verified end to end: `cmake --preset dev && ctest --test-dir build/dev -R
'^conformance$'` passes in 23.8s, exercising the exact command and label the
CI job uses.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_012fgjnqFCS5h5L7gZTZo3rV
2026-09-10 00:02:13 +04:00
samiandClaude Sonnet 5 c65e664c29 core: sign off on ADR 0013 (task state-machine ownership)
Accept as written — no amendments. Answers to the four open items:

1. tasks_starved / starved_tasks() count ONLY tasks in {connecting,
   downloading} with segments_active == 0 (the allocator owes a slot to a
   task that is asking). retry_wait and paused (either-initiated) are
   outside that state set, so they are excluded by construction, not by a
   special case. Design commitment; the accessor is stage 6/8.
2. pause() is idempotent: no-op success on an already-paused or terminal
   task; resume() no-op success on a non-paused task; only task_not_found
   errors. No state-change event for a no-op.
3. PROTO's item — CORE confirms its half: auto-pause reports the
   transition with ErrorInfo populated (auth_required / server_file_changed
   / disk_full / path_rejected), already in util/error.hpp. Ready once
   PROTO permits error on state=="paused".
4. CORE adopts "auto-pause"; the wire/API discriminator stays
   state==paused + presence of the Error code.

Notes back: pause during verifying re-hashes from scratch on resume;
pause during assembling is M4; restart handling in §5 agreed —
start(TaskId) re-derives resume position from .veloxpart.meta + If-Range.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HPPSGhiArbvQgwC2DNiURS
2026-09-10 00:02:02 +04:00
samiandClaude Sonnet 5 2d36e9fef0 proto: land F2 — download.provideAuth (1.2.0)
The last contract gap blocking an M1 definition-of-done item: CORE's "401
handled" has no return path without it, and B2a's sibling F2 was accepted in
proto-answers-m1.md but never actually landed.

download.provideAuth {taskId, username, password, save?} -> {ok}, exactly as
proposed there. Privileged and Unix-socket-only: a credential-bearing method
must never be reachable from the browser, which is the other half of the
promise event.auth.required's own description already makes ("never back
through this event, never into a log"). It answers the challenge; it does not
itself resume the task -- the daemon retries with the credential attached and
the ordinary event.task.state reports the task leaving retry_wait, the same
as any other state change.

save only tells the daemon whether to persist the credential in the Secret
Service for next time, or use it for this attempt alone -- it never touches
SQLite or a log either way, in keeping with CLAUDE.md's secrets rule.

Three fixtures: the success path, -32010 for a task that no longer exists
(credentials submitted for it are simply discarded), and -32003 confirming
the extension has no path to this method under any transport.

mockd gets a real handler rather than falling through to the generic fixture
responder: it validates the taskId exists (so the -32010 fixture is
replayable) and actually transitions the task out of retry_wait.

Minor bump, 1.1.0 -> 1.2.0: additive method, no existing type touched.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_012fgjnqFCS5h5L7gZTZo3rV
2026-09-10 00:01:58 +04:00
samiandClaude Sonnet 5 f60070c420 mockd: add --tasks N — a plausible large synthetic table for GUI's DoD
GUI's M1 definition of done is "10 000 synthetic rows scroll at 60 fps with
flat memory over 10 minutes (mockd --tasks 10000)". This flag was missing from
the four unhappy-path flags that did land; the brief's own flag list omitted
it, which is corrected here too.

--tasks seeds a plausible population rather than N copies of one row: varied
state, size (log-uniform 50 KB - 20 GB), category, queue position and
description, drawn from the same category.list / queue.list fixtures the rest
of mockd already serves so a synthetic task can never name a category or
queue those methods don't also return. State distribution is roughly
55% complete / 8% failed / 4% cancelled / 6% paused / 2% retry_wait / 25%
queued, using the new TaskErrorCode taxonomy for failures.

"Progress advances across the whole set, not a handful of live rows" ruled
out the obvious cheap answer. A bounded, rotating pool of concurrently-active
downloads (--active-cap, default 24) is fed continuously from each queue's
FIFO — with the rest of that queue's queuePosition renumbered on every
promotion, as a real scheduler would — and a small fraction of active tasks
hit a transient failure and cycle through retry_wait before rejoining, so the
pool keeps rotating through new rows for the whole run instead of draining
once. Verified over a 10000-task, 60-second run: 61.5 MB RSS flat, and the
active pool's membership meaningfully different after 60s.

tick() only ever walks the active pool plus due retry-wait entries, never the
full task list, so its cost stays flat regardless of --tasks. A manual
download.add is still admitted immediately regardless of --active-cap — a
human driving the GUI by hand must never wait behind synthetic load.

Fixed a latent double-push while building this: any task 'connecting' at the
top of a tick was pushed to the progress batch once for the transition and
again at the loop's unconditional final push, inflating event.task.progress
payloads with a duplicate entry for that taskId. It predates this change (the
original tick() had the same shape) but only became visible once several
tasks are legitimately 'connecting' in the same tick, which --active-cap's
continuous promotion now does routinely.

--seed makes a run reproducible, which matters when a GUI bug only shows up
at a particular row.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_012fgjnqFCS5h5L7gZTZo3rV
2026-09-10 00:01:44 +04:00
samiandClaude Sonnet 5 19d3cd2b1d daemon: draft ADR 0013 — task state-machine ownership (D1)
Drafts the CORE/DAEMON split PROTO raised as D1 and CORE already agreed
to in contracts/proto-answers-m1.md, since neither lane would write it
down alone. DAEMON owns new/queued and pause-for-schedule; CORE owns
probing through complete|failed and cancelled-from-anywhere is
DAEMON-driven; paused is shared.

Ties into ADR 0011 in two places:
- retry_wait looks identical to segment starvation from the budget
  accessor's point of view (zero segments, deliberately) and must be
  excluded from tasks_starved by construction, not by DAEMON guessing
  from timing.
- restart handling: CORE holds no persistent state, so any CORE-owned
  TaskState reloads as queued and re-admits through the scheduler;
  paused tasks reload with their pauseReason intact.

Surfaces one real contract gap while drafting, not just an open
question: event.task.state's error field is schema-scoped to
failed/retry_wait only, so CORE auto-pausing for auth_required or
server_file_changed currently has no wire signal telling DAEMON why —
needed before the resume-must-not-cross-reasons rule (§3) can be
implemented at all. Filed as a PROTO follow-up in the ADR itself.

Status: proposed, needs CORE + PROTO sign-off (four open items at the
end) before daemon/src/sched/'s pause/resume logic is written against
it.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
2026-09-10 00:00:08 +04:00
samiandClaude Sonnet 5 201ebc55d4 core: net/probe + Content-Disposition parser + URL splitter (stage 3)
net/content_disposition — total parser for the mojibake-prone header:
RFC 6266 filename (quoted/token), RFC 5987 filename* ext-values
(charset'lang'pct-encoded, incl. RFC 2231 continuations), legacy RFC 2047
encoded-words (=?UTF-8?B?..?= / ?Q?), and raw Latin-1 bytes; prefers
filename* over filename; strips path components AFTER decoding (a base64
payload can hold '/'). 22-case test table.

net/text_codec (internal) — percent-decode, UTF-8 validation, Latin-1->
UTF-8, base64, RFC 2047 — shared by the CD parser and the URL splitter.

net/url — a small total URL splitter (scheme/userinfo/host/port/path/
query/fragment, http(s) validity) and url_filename() for the last path
segment; used for the filename fallback.

net/probe — HEAD then a ranged GET bytes=0-0 that PROVES resumability
(206 + matching Content-Range + a validator), rather than trusting
Accept-Ranges which servers lie about; the ranged GET is also the HEAD-
refused (403/405/501) fallback. 401/407 -> success result with
requires_auth, not an error. Runs on its own pool (max_concurrent,
default 4) outside the segment budget per ADR 0011 §5. suggest_filename()
does the resolution order (explicit -> disposition -> URL -> download.bin)
with a light strip; rules/ (stage 9) owns the authoritative sanitize.

tools/fuzz — libFuzzer targets for the CD parser and the URL splitter,
compiling the parser sources directly so they're fully instrumented;
self-guards on VELOX_BUILD_FUZZ + Clang (the top-level CMake adds every
tools/* unconditionally). Seed corpora included.

Fixed on the way: a p -> Transfer -> State -> cbs -> p reference cycle in
Prober that leaked every probe (drop the stored Transfer; the worker
keeps State alive). Tests green under ASan/UBSan and TSan.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HPPSGhiArbvQgwC2DNiURS
2026-09-09 23:53:55 +04:00
samiandClaude Sonnet 5 db7650b2fd pkg: make the conformance CI check run and be able to fail
The conformance job was a required check that ran nothing. Two bugs:

  1. Its guard tested for tests/conformance/CMakeLists.txt or package.json.
     The suite ships as tests/conformance/run.sh; neither file exists, so the
     guard was always false and the job took the "skipped" (success) branch.
  2. Even forced true, it ran `ctest --preset dev -L conformance` — no test
     carries that label, so ctest reported "Total Tests: 0" and exited 0.

Replace the job body with PROTO's intended wiring from
tests/conformance/README.md: bootstrap the toolchain, pin Node 22 (apt ships
< 20; the TS replay runner needs >= 20), and run ./tests/conformance/run.sh
directly. The suite starts its own mockd and builds its own C++ runner, so no
cmake configure is needed. Verified it goes red: an enum-invalid fixture makes
run.sh exit 1; reverting it returns to green.

check_contract.py imports jsonschema and referencing, which bootstrap.sh did
not install. Add python3-jsonschema / python3-referencing to the apt set and
to --check, so one command still provisions the whole suite.

Guards now fail loudly instead of passing quietly:

  - conformance has no skip branch any more. run.sh has landed; the job runs
    it unconditionally and errors if the entrypoint is missing.
  - extension-lint keyed "has EXT landed?" to extension/package.json — the
    same single-filename trap. Key it to a manifest instead, and once a
    manifest exists, treat a missing package.json as a hard failure rather
    than a green skip.

Mark conformance required now in BRANCH_PROTECTION.md — it is the M0 exit gate.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0143aKiohmDiyefJBwHDJJqw
2026-09-09 23:48:30 +04:00
383 changed files with 48448 additions and 412 deletions
+20 -9
View File
@@ -14,11 +14,13 @@ policy so it can be re-applied or audited.
|---|---|
| `clang-format` | now |
| `testserver` | now |
| `bootstrap-script` | now |
| `build (gcc)` / `build (clang)` | when the first C++ lane merges |
| `sanitizers (dev)` / `sanitizers (tsan)` | when the first C++ lane merges |
| `conformance` | **when `tests/conformance/` lands — this is the M0 exit gate** |
| `extension-lint` | when `extension/` lands |
| `bootstrap-script` | now (validates package names against the 24.04 runner archive) |
| `bootstrap-script-2604` | now — real `--with-clang` install in a 26.04 container; the release the project ships on |
| `build (gcc)` / `build (clang)` | now — core, daemon and gui have merged |
| `sanitizers (dev)` / `sanitizers (tsan)` | now — core, daemon and gui have merged |
| `conformance` | **now — `tests/conformance/` has landed; this is the M0 exit gate.** Includes the live-`veloxd` runner (step 3b of `run.sh`), unconditional in the script — see `docs/adr/0019-live-veloxd-conformance-is-required.md`. |
| `extension-lint` | now — `extension/` has merged (MV3 manifest + esbuild build) |
| `gui-dod` | now — `gui/tests/dod/` has landed (GUI M1 DoD gates R3: `scroll-60fps`, `unhappy-path`); see `tests/integration/README.md#gui-m1-definition-of-done-gates-r3`. `gui-dod-nightly` (`rss-flat`) is schedule-only and cannot be a required PR check. |
`clang-tidy` is intentionally **not** required through M1 (`continue-on-error: true`,
`.clang-tidy` has `WarningsAsErrors: ''`). Make it required at M2.
@@ -30,7 +32,16 @@ policy so it can be re-applied or audited.
## Note on the "skipped" job steps
Several jobs (`conformance`, `extension-lint`, `clang-tidy`) short-circuit to a "skipped"
echo when their lane hasn't landed. They still report **success**, so they can be marked
required now without blocking — they start doing real work automatically on the commit
that adds the lane.
`extension-lint` and `clang-tidy` short-circuit to a "skipped" echo when their lane
hasn't landed. They still report **success**, so they can be marked required now without
blocking — they start doing real work automatically on the commit that adds the lane.
Their guards fail **loudly** (non-zero) once the lane is half-present — e.g. an
`extension/manifest.json` with no lintable `package.json`. A guard keyed to a single
filename is how a required check ends up green over nothing; the skip branch is only for
a lane that is genuinely absent.
`conformance` has no skip branch. It runs `ctest -L conformance` (see
`docs/adr/0014-conformance-runs-through-ctest.md`); the `dev` test preset's
`noTestsAction: error` fails the job if that label ever matches nothing, so a deleted or
renamed registration goes red instead of passing vacuously.
+162 -25
View File
@@ -4,6 +4,9 @@ on:
push:
branches: [main]
pull_request:
schedule:
- cron: '17 3 * * *' # nightly-integration only; every other job stays PR/push-triggered
workflow_dispatch: # lets a human fire nightly-integration on demand
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
@@ -41,34 +44,81 @@ jobs:
run: python3 tools/testserver/selftest.py
bootstrap-script:
# Keeps tools/bootstrap.sh honest: it must run clean and its --check must pass.
# Keeps tools/bootstrap.sh honest on the runner image: it must run clean and its
# --check must pass. ubuntu-latest is 24.04; the project ships on 26.04, so this
# exercises the 24.04 archive only. bootstrap-script-2604 below is what validates
# the package names against the release the project actually targets.
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: sudo ./tools/bootstrap.sh --with-clang
- run: ./tools/bootstrap.sh --check
- run: ./tools/bootstrap.sh --check --with-clang
# Cheap: apt-cache only. Validates the M6 packaging names now so they can't rot
# unnoticed until M6.
- run: ./tools/bootstrap.sh --check --with-clang --packaging
bootstrap-script-2604:
# The project targets 26.04 and GitHub has no 26.04 runner image yet, so the one
# automated place bootstrap.sh runs is on the wrong release to catch a name that is
# valid on 24.04 and gone on 26.04 — which is exactly how libqt6svg6-dev reached a
# contributor's VM (gui/docs/pkg-qa-requests-m1.md R1/R2). Run the real install in a
# 26.04 container, with --with-clang: the fuzz toolchain had never been exercised
# anywhere (CORE ran clang++-21 directly because it can't sudo).
runs-on: ubuntu-latest
container: ubuntu:26.04
steps:
- name: Base tools for checkout
run: |
apt-get update -qq
apt-get install -y --no-install-recommends ca-certificates git sudo
- uses: actions/checkout@v4
- name: Full bootstrap on 26.04 (--with-clang)
run: ./tools/bootstrap.sh --with-clang
- name: Re-verify
run: ./tools/bootstrap.sh --check --with-clang
extension-lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- id: check
# "Has EXT landed?" is answered by a manifest, not by extension/package.json:
# a guard keyed to one filename passes vacuously the day EXT ships the lane
# under any other name. Skip only when the lane genuinely is not here; once a
# manifest exists, a missing lint entrypoint is a hard failure, not a skip.
run: |
if [ -f extension/package.json ]; then echo "present=true" >> "$GITHUB_OUTPUT"
else echo "present=false" >> "$GITHUB_OUTPUT"; fi
manifest=""
for m in extension/manifest.json extension/src/manifest.json extension/public/manifest.json; do
if [ -f "$m" ]; then manifest="$m"; break; fi
done
if [ -z "$manifest" ]; then
echo "extension/ has not landed yet (no manifest.json) — skipping web-ext lint."
echo "present=false" >> "$GITHUB_OUTPUT"
exit 0
fi
echo "EXT has landed: $manifest"
echo "present=true" >> "$GITHUB_OUTPUT"
if [ ! -f extension/package.json ]; then
echo "::error::$manifest exists but extension/package.json does not — this job" \
"cannot lint the extension. Wire web-ext lint in here; do not let the check" \
"pass green over an unlinted lane."
exit 1
fi
- uses: actions/setup-node@v4
if: steps.check.outputs.present == 'true'
with:
node-version: '20'
- name: web-ext lint
node-version: '22'
- name: eslint (no-download-logic gate + general rules)
if: steps.check.outputs.present == 'true'
working-directory: extension
run: |
npm ci
npx eslint .
- name: web-ext lint
if: steps.check.outputs.present == 'true'
working-directory: extension
run: |
npx web-ext lint --source-dir .
- name: skipped
if: steps.check.outputs.present == 'false'
run: echo "extension/ has not landed yet — skipping web-ext lint"
# --- build + test matrix ----------------------------------------------------------
build:
@@ -89,7 +139,10 @@ jobs:
- name: Build
run: cmake --build --preset ci
- name: Test
run: ctest --preset ci --output-on-failure
# -E '^conformance$' drops the end-to-end run.sh test (npm installs, its own
# mockd, ~24 s); the dedicated `conformance` job owns that one run. The native
# `conformance_cpp` test is not excluded and still runs on every matrix leg.
run: ctest --preset ci --output-on-failure -E '^conformance$'
sanitizers:
runs-on: ubuntu-latest
@@ -106,7 +159,10 @@ jobs:
- name: Build
run: cmake --build --preset ${{ matrix.preset }}
- name: Test
run: ctest --preset ${{ matrix.preset }} --output-on-failure
# See the build job: the end-to-end run.sh test is the dedicated `conformance`
# job's; sanitizing a suite that shells out to its own unsanitized g++ build and
# a node process buys nothing. `conformance_cpp` still runs here under the sanitizer.
run: ctest --preset ${{ matrix.preset }} --output-on-failure -E '^conformance$'
env:
ASAN_OPTIONS: detect_leaks=1:halt_on_error=1
UBSAN_OPTIONS: print_stacktrace=1:halt_on_error=1
@@ -143,24 +199,105 @@ jobs:
run: echo "no C++ lane has landed a CMakeLists yet — skipping clang-tidy"
conformance:
# Required check on every PR once tests/conformance/ lands (branch protection is
# configured in the repo settings, not here — see .github/BRANCH_PROTECTION.md).
# The M0 exit gate. Proves the generated C++ daemon surface and the generated TS
# extension surface agree with contracts/fixtures without either side having run
# against the other. Required on every PR — branch protection is a repo setting,
# recorded in .github/BRANCH_PROTECTION.md.
#
# Canonical entry point is `ctest -L conformance`. tests/conformance/CMakeLists.txt
# (owned by PROTO) registers two tests under that label: `conformance`, which shells
# out to run.sh end to end, and `conformance_cpp`, the finer-grained native runner.
# CI drives it exactly as a developer does — one definition of "the suite passed",
# and PROTO's registration is on the exercised path so it cannot rot. See
# docs/adr/0014-conformance-runs-through-ctest.md.
#
# `noTestsAction: error` in the dev test preset is the rot guard: if the label ever
# matches nothing (registration deleted, typo), ctest exits non-zero instead of
# passing vacuously.
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- id: check
run: |
if [ -f tests/conformance/CMakeLists.txt ] || [ -f tests/conformance/package.json ]; then
echo "present=true" >> "$GITHUB_OUTPUT"
else echo "present=false" >> "$GITHUB_OUTPUT"; fi
- name: Bootstrap toolchain
if: steps.check.outputs.present == 'true'
run: sudo ./tools/bootstrap.sh
- name: Run conformance suite
if: steps.check.outputs.present == 'true'
- uses: actions/setup-node@v4
with:
node-version: '22' # apt ships < 20; run.sh's TS replay runner needs >= 20
- name: Configure
run: cmake --preset dev
- name: Build the native conformance runner
run: cmake --build --preset dev --target velox_conformance_cpp
- name: Run conformance (ctest -L conformance)
run: ctest --preset dev -L conformance --output-on-failure
nightly-integration:
# Real veloxd + tools/testserver, 50 concurrent downloads mixing hostile modes,
# every completed file's SHA-256 checked against testserver's own /sha256/ route,
# veloxd's open-FD count checked flat across the run. Nightly, not per-PR: it's
# ~2 minutes of real network I/O against a local server, not a schema check.
# See tests/integration/README.md#nightly-integration-run for what each assertion
# catches and the forced-failure transcript proving it isn't vacuous.
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Bootstrap toolchain
run: sudo ./tools/bootstrap.sh
- name: Configure
run: cmake --preset dev
- name: Build veloxd
run: cmake --build --preset dev --target veloxd
- name: Nightly integration run
run: python3 tests/integration/nightly_run.py --veloxd build/dev/bin/veloxd --tasks 50 --timeout 180
gui-dod:
# Per-PR GUI M1 DoD gates (gui/docs/pkg-qa-requests-m1.md R3): scroll-60fps and
# unhappy-path. The 10-minute rss-flat gate is gui-dod-nightly, not here. GUI's
# harness defaults QT_QPA_PLATFORM=offscreen itself, so no Xvfb/compositor needed.
# See tests/integration/README.md#gui-m1-definition-of-done-gates-r3 for what each
# gate catches and the forced-failure transcript proving it isn't vacuous.
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Bootstrap toolchain
run: sudo ./tools/bootstrap.sh
- uses: actions/setup-node@v4
with:
node-version: '22' # tools/mockd
- name: Configure + build
run: |
cmake --preset dev
ctest --preset dev --output-on-failure -L conformance
- name: skipped
if: steps.check.outputs.present == 'false'
run: echo "tests/conformance/ has not landed yet — skipping"
cmake --build --preset dev --target gui-dod-harness
- name: Install mockd
run: cd tools/mockd && npm ci
- name: Gates
run: |
gui/tests/dod/run.sh scroll-60fps --json scroll.json
gui/tests/dod/run.sh unhappy-path --json unhappy.json
- uses: actions/upload-artifact@v4
if: always()
with:
name: gui-dod-${{ github.run_id }}
path: "*.json"
gui-dod-nightly:
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Bootstrap toolchain
run: sudo ./tools/bootstrap.sh
- uses: actions/setup-node@v4
with:
node-version: '22'
- name: Configure + build
run: |
cmake --preset dev
cmake --build --preset dev --target gui-dod-harness
- run: cd tools/mockd && npm ci
- name: RSS soak (10 min)
run: gui/tests/dod/run.sh rss-flat --json rss.json
- uses: actions/upload-artifact@v4
if: always()
with:
name: gui-dod-rss-${{ github.run_id }}
path: rss.json
+8
View File
@@ -37,3 +37,11 @@ massif.out.*
*.log
*.veloxpart
*.veloxpart.meta
# Fuzzing crash artifacts — libFuzzer writes these to CWD on a find and each holds the
# crashing input verbatim. Ignore so they are never committed by accident (CORE caught
# two by hand before this rule existed).
crash-*
oom-*
leak-*
timeout-*
+5
View File
@@ -68,3 +68,8 @@ you change observable behaviour, update the doc in `docs/` that describes it in
Ask in the PR rather than guessing at the interface. A day of clarification is cheaper than
an M2 integration rewrite. And record real decisions as an ADR in `docs/adr/` — the next
agent to touch this will have none of your context.
**ADR numbers:** there is no allocator. Take the next free number in `main`'s
`docs/adr/` (gaps from reserved-but-unwritten entries are fine to fill). Lanes draft in
parallel, so collisions happen: whoever merges **second** renumbers, updates any
cross-references, and keeps going — it is not worth a round trip.
+2 -1
View File
@@ -97,12 +97,13 @@ Surveyed on this machine 2026-09-09 — **most of it is already installed**:
| libcurl4-openssl-dev · libsqlite3-dev · nlohmann-json3-dev · libssl-dev | ✓ |
| libavformat-dev · libavcodec-dev · ffmpeg | ✓ |
| clang-format · clang-tidy · python3 · pkg-config | ✓ |
| python3-jsonschema · python3-referencing (conformance static runner) | ✓ |
**Only these four are missing:**
```bash
sudo apt update && sudo apt install -y \
libqt6svg6-dev \ # GUI: SVG icon rendering
qt6-svg-dev \ # GUI: SVG icon rendering
libsecret-1-dev \ # DAEMON: Secret Service for site logins
nodejs npm \ # EXT + PROTO: extension build, mockd, conformance runner
clang # optional: libFuzzer targets in M7
+23
View File
@@ -0,0 +1,23 @@
# cli/ produces the `velox` binary — the scriptable RPC client. Owned by lane DAEMON.
# Built early (AGENT-DAEMON.md build step 8, pulled forward): it is how the daemon is
# tested before the GUI exists.
#
# Links velox::proto for the wire types and error codes. It speaks the same NDJSON Unix
# socket veloxd listens on; no daemon code is linked.
if(NOT TARGET nlohmann_json::nlohmann_json)
find_package(nlohmann_json 3.11 REQUIRED)
endif()
add_executable(velox
src/main.cpp
src/client.cpp
)
target_include_directories(velox PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/src)
target_compile_features(velox PRIVATE cxx_std_23)
target_compile_options(velox PRIVATE -Wall -Wextra -Wpedantic -Werror)
target_link_libraries(velox PRIVATE velox::proto nlohmann_json::nlohmann_json)
if(VELOX_BUILD_TESTS AND EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/tests/CMakeLists.txt)
add_subdirectory(tests)
endif()
View File
+130
View File
@@ -0,0 +1,130 @@
#include "client.hpp"
#include <sys/socket.h>
#include <sys/un.h>
#include <unistd.h>
#include <cerrno>
#include <cstdlib>
#include <cstring>
#include "velox_proto.hpp"
namespace velox::cli {
namespace {
std::string read_error_message(int e) { return std::strerror(e); }
} // namespace
std::string default_socket_path() {
std::string base;
if (const char* xdg = ::getenv("XDG_RUNTIME_DIR"); xdg != nullptr && xdg[0] != '\0') {
base = xdg;
} else {
base = "/run/user/" + std::to_string(::geteuid());
}
if (!base.empty() && base.back() == '/') base.pop_back();
return base + "/velox/velox.sock";
}
Client::~Client() {
if (fd_ >= 0) ::close(fd_);
}
std::optional<CallError> Client::connect() {
socket_path_ = default_socket_path();
if (socket_path_.size() + 1 > sizeof(sockaddr_un::sun_path)) {
return CallError{CallError::kConnect, "socket path too long: " + socket_path_, {}};
}
fd_ = ::socket(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0);
if (fd_ < 0) return CallError{CallError::kConnect, read_error_message(errno), {}};
sockaddr_un addr{};
addr.sun_family = AF_UNIX;
std::memcpy(addr.sun_path, socket_path_.c_str(), socket_path_.size());
if (::connect(fd_, reinterpret_cast<sockaddr*>(&addr), sizeof(addr)) != 0) {
const int e = errno;
::close(fd_);
fd_ = -1;
return CallError{CallError::kConnect,
"cannot reach veloxd at " + socket_path_ + ": " + read_error_message(e),
{}};
}
nlohmann::json hello = {
{"clientType", "cli"},
{"clientName", std::string("velox ") + std::string(velox::proto::kProtocolVersion)},
{"protocolVersion", std::string(velox::proto::kProtocolVersion)},
};
auto r = call("session.hello", hello);
if (!r) return r.error();
hello_result_ = *r;
return std::nullopt;
}
std::expected<nlohmann::json, CallError> Client::call(const std::string& method,
const nlohmann::json& params) {
const nlohmann::json request = {
{"jsonrpc", "2.0"},
{"id", next_id_++},
{"method", method},
{"params", params},
};
return round_trip(request);
}
std::expected<nlohmann::json, CallError> Client::round_trip(const nlohmann::json& request) {
if (fd_ < 0) return std::unexpected(CallError{CallError::kConnect, "not connected", {}});
std::string out = request.dump();
out.push_back('\n');
std::size_t off = 0;
while (off < out.size()) {
const ssize_t n = ::write(fd_, out.data() + off, out.size() - off);
if (n > 0) {
off += static_cast<std::size_t>(n);
continue;
}
if (n < 0 && errno == EINTR) continue;
return std::unexpected(CallError{CallError::kConnect,
"write to daemon failed: " + read_error_message(errno), {}});
}
// Read until a newline completes a frame.
for (;;) {
if (const auto nl = inbuf_.find('\n'); nl != std::string::npos) {
const std::string line = inbuf_.substr(0, nl);
inbuf_.erase(0, nl + 1);
nlohmann::json reply = nlohmann::json::parse(line, nullptr, false);
if (reply.is_discarded()) {
return std::unexpected(
CallError{CallError::kProtocol, "daemon sent a malformed reply", {}});
}
if (reply.contains("error")) {
const auto& e = reply.at("error");
return std::unexpected(CallError{e.value("code", 0), e.value("message", ""),
e.contains("data") ? e.at("data") : nlohmann::json()});
}
return reply.contains("result") ? reply.at("result") : nlohmann::json(nullptr);
}
char chunk[8192];
const ssize_t n = ::read(fd_, chunk, sizeof(chunk));
if (n > 0) {
inbuf_.append(chunk, static_cast<std::size_t>(n));
continue;
}
if (n == 0) {
return std::unexpected(CallError{CallError::kConnect,
"daemon closed the connection", {}});
}
if (errno == EINTR) continue;
return std::unexpected(CallError{CallError::kConnect,
"read from daemon failed: " + read_error_message(errno), {}});
}
}
} // namespace velox::cli
+55
View File
@@ -0,0 +1,55 @@
#pragma once
// A synchronous, blocking RPC client for the Unix socket. The CLI does one request at a
// time and waits for the reply, so none of the daemon's async machinery is needed here —
// just connect, session.hello, call, read one NDJSON frame back.
#include <expected>
#include <optional>
#include <string>
#include <nlohmann/json.hpp>
namespace velox::cli {
struct CallError {
int code; // JSON-RPC error code, or a negative transport code below
std::string message;
nlohmann::json data; // may be null
static constexpr int kConnect = -1000; // could not reach the daemon
static constexpr int kProtocol = -1001; // malformed reply / framing error
};
class Client {
public:
~Client();
// Resolve the socket path ($XDG_RUNTIME_DIR/velox/velox.sock), connect, and complete
// session.hello with clientType "cli". On failure returns the error and leaves the
// client unusable.
std::optional<CallError> connect();
// Send one request and return its result, or the error. `params` is passed through
// verbatim as the JSON-RPC params.
std::expected<nlohmann::json, CallError> call(const std::string& method,
const nlohmann::json& params);
const std::string& socket_path() const noexcept { return socket_path_; }
const nlohmann::json& hello_result() const noexcept { return hello_result_; }
private:
std::expected<nlohmann::json, CallError> round_trip(const nlohmann::json& request);
int fd_ = -1;
int next_id_ = 1;
std::string socket_path_;
std::string inbuf_;
nlohmann::json hello_result_;
};
// $XDG_RUNTIME_DIR/velox/velox.sock, or /run/user/<uid>/velox/velox.sock when the env var
// is unset. Mirrors daemon/src/rpc/runtime_dir.cpp; kept in sync by being trivial.
std::string default_socket_path();
} // namespace velox::cli
+179
View File
@@ -0,0 +1,179 @@
// velox — the command-line client for veloxd.
//
// velox add <url> [--dir D] [--out NAME] [--segments N] [--json]
// velox ls [--json]
// velox pause <id>... velox resume <id>...
// velox rm <id>... [--delete-file]
//
// Exit codes: 0 ok, 1 daemon returned an error, 2 usage error, 3 cannot reach the daemon.
#include <cstdio>
#include <cstdlib>
#include <string>
#include <string_view>
#include <vector>
#include <nlohmann/json.hpp>
#include "client.hpp"
namespace {
using nlohmann::json;
using velox::cli::CallError;
using velox::cli::Client;
constexpr int kOk = 0;
constexpr int kRpcError = 1;
constexpr int kUsage = 2;
constexpr int kNoDaemon = 3;
struct Args {
std::vector<std::string> positional;
bool json = false;
bool delete_file = false;
std::string dir;
std::string out;
long segments = 0;
};
[[noreturn]] void usage(int code) {
std::fprintf(code == kOk ? stdout : stderr,
"usage: velox <command> [options]\n\n"
" add <url> [--dir DIR] [--out NAME] [--segments N]\n"
" ls\n"
" pause <id>...\n"
" resume <id>...\n"
" rm <id>... [--delete-file]\n\n"
" --json print the raw JSON-RPC result\n");
std::exit(code);
}
Args parse_args(int argc, char** argv) {
Args a;
for (int i = 2; i < argc; ++i) {
const std::string_view arg = argv[i];
if (arg == "--json") {
a.json = true;
} else if (arg == "--delete-file") {
a.delete_file = true;
} else if (arg == "--dir" && i + 1 < argc) {
a.dir = argv[++i];
} else if (arg == "--out" && i + 1 < argc) {
a.out = argv[++i];
} else if (arg == "--segments" && i + 1 < argc) {
a.segments = std::strtol(argv[++i], nullptr, 10);
} else if (arg == "-h" || arg == "--help") {
usage(kOk);
} else if (!arg.empty() && arg.front() == '-') {
std::fprintf(stderr, "velox: unknown option %s\n", argv[i]);
usage(kUsage);
} else {
a.positional.emplace_back(arg);
}
}
return a;
}
int report_error(const CallError& e, bool as_json) {
if (as_json) {
json j = {{"error", {{"code", e.code}, {"message", e.message}}}};
if (!e.data.is_null()) j["error"]["data"] = e.data;
std::printf("%s\n", j.dump(2).c_str());
} else {
std::fprintf(stderr, "velox: %s\n", e.message.c_str());
}
return e.code == CallError::kConnect ? kNoDaemon : kRpcError;
}
void print_task_table(const json& items) {
std::printf("%-38s %-10s %-9s %s\n", "ID", "STATE", "PROGRESS", "NAME");
for (const auto& t : items) {
const std::string id = t.value("taskId", "");
const std::string state = t.value("state", "");
const std::string name = t.value("filename", "");
const long long total = t.value("sizeBytes", 0LL);
const long long done = t.value("downloadedBytes", 0LL);
char pct[12] = "-";
if (total > 0) std::snprintf(pct, sizeof(pct), "%lld%%", done * 100 / total);
std::printf("%-38s %-10s %-9s %s\n", id.c_str(), state.c_str(), pct, name.c_str());
}
}
int cmd_ls(Client& c, const Args& a) {
auto r = c.call("download.list", json::object());
if (!r) return report_error(r.error(), a.json);
if (a.json) {
std::printf("%s\n", r->dump(2).c_str());
return kOk;
}
const auto& items = r->contains("items") ? r->at("items") : json::array();
if (items.empty()) {
std::printf("no downloads\n");
return kOk;
}
print_task_table(items);
return kOk;
}
int cmd_add(Client& c, const Args& a) {
if (a.positional.empty()) {
std::fprintf(stderr, "velox add: a URL is required\n");
return kUsage;
}
json params = {{"url", a.positional.front()}};
if (!a.dir.empty()) params["saveDir"] = a.dir;
if (!a.out.empty()) params["filename"] = a.out;
if (a.segments > 0) params["segments"] = a.segments;
auto r = c.call("download.add", params);
if (!r) return report_error(r.error(), a.json);
if (a.json) {
std::printf("%s\n", r->dump(2).c_str());
} else {
std::printf("added %s\n", r->value("taskId", "?").c_str());
}
return kOk;
}
int cmd_bulk(Client& c, const Args& a, const char* method) {
if (a.positional.empty()) {
std::fprintf(stderr, "velox: at least one task id is required\n");
return kUsage;
}
json params = {{"taskIds", a.positional}};
if (std::string_view(method) == "download.remove" && a.delete_file) params["deleteFile"] = true;
auto r = c.call(method, params);
if (!r) return report_error(r.error(), a.json);
if (a.json) {
std::printf("%s\n", r->dump(2).c_str());
} else {
std::printf("ok\n");
}
return kOk;
}
} // namespace
int main(int argc, char** argv) {
if (argc < 2) usage(kUsage);
const std::string command = argv[1];
if (command == "-h" || command == "--help") usage(kOk);
const Args args = parse_args(argc, argv);
Client client;
if (const auto err = client.connect()) {
return report_error(*err, args.json);
}
if (command == "ls") return cmd_ls(client, args);
if (command == "add") return cmd_add(client, args);
if (command == "pause") return cmd_bulk(client, args, "download.pause");
if (command == "resume") return cmd_bulk(client, args, "download.resume");
if (command == "rm") return cmd_bulk(client, args, "download.remove");
std::fprintf(stderr, "velox: unknown command '%s'\n", command.c_str());
usage(kUsage);
}
+13
View File
@@ -0,0 +1,13 @@
# CLI integration test: a real in-process UdsServer on a temp socket, the real Client
# against it. Links veloxd_rpc for the server half.
add_executable(velox_client_test client_test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/../src/client.cpp)
target_include_directories(velox_client_test PRIVATE
${CMAKE_CURRENT_SOURCE_DIR}/../src
${CMAKE_CURRENT_SOURCE_DIR}/../../daemon/tests
)
target_compile_features(velox_client_test PRIVATE cxx_std_23)
target_compile_options(velox_client_test PRIVATE -Wall -Wextra -Wpedantic -Werror)
target_link_libraries(velox_client_test PRIVATE veloxd_rpc velox::proto nlohmann_json::nlohmann_json)
add_test(NAME velox.client COMMAND velox_client_test)
set_tests_properties(velox.client PROPERTIES TIMEOUT 30)
+132
View File
@@ -0,0 +1,132 @@
// The CLI's Client against a real in-process daemon RPC server.
#include <sys/stat.h>
#include <unistd.h>
#include <cstdlib>
#include <string>
#include <thread>
#include "check.hpp"
#include "client.hpp"
#include "rpc/dispatcher.hpp"
#include "rpc/event_loop.hpp"
#include "rpc/event_hub.hpp"
#include "rpc/uds_server.hpp"
#include "store/migrations.hpp"
#include "store/settings.hpp"
#include "store/sqlite.hpp"
namespace rpc = velox::daemon::rpc;
static std::string g_allowed_root;
using velox::cli::CallError;
using velox::cli::Client;
void run() {
// connect() with no daemon -> a kConnect error, not a crash.
{
::setenv("XDG_RUNTIME_DIR", "/tmp/velox-cli-test-nonexistent-xyz", 1);
Client c;
auto err = c.connect();
CHECK(err.has_value());
if (err) CHECK_EQ(err->code, CallError::kConnect);
}
// Point XDG_RUNTIME_DIR at a fresh temp dir; the client derives
// <XDG_RUNTIME_DIR>/velox/velox.sock and the server binds exactly that.
char tmpl[] = "/tmp/velox-cli-test-XXXXXX";
const char* xdg = ::mkdtemp(tmpl);
CHECK(xdg != nullptr);
if (xdg == nullptr) return;
::setenv("XDG_RUNTIME_DIR", xdg, 1);
const std::string velox_dir = std::string(xdg) + "/velox";
::mkdir(velox_dir.c_str(), 0700);
const std::string server_sock = velox_dir + "/velox.sock";
rpc::EventLoop loop;
auto db = velox::daemon::store::Db::open(":memory:");
CHECK(db.has_value());
if (!db) return;
CHECK(velox::daemon::store::migrate_to_head(*db).has_value());
char root_tmpl[] = "/tmp/velox-cli-root-XXXXXX";
g_allowed_root = ::mkdtemp(root_tmpl);
CHECK(!g_allowed_root.empty());
{
velox::daemon::store::Settings settings(*db);
CHECK(settings.set_raw("saveTo.allowedRoots",
"[\"" + g_allowed_root + "\"]").has_value());
CHECK(settings.set_raw("saveTo.defaultDir",
"\"" + g_allowed_root + "\"").has_value());
}
rpc::EventHub hub;
rpc::VeloxDispatcher dispatcher(*db, hub);
rpc::UdsServer server(loop, dispatcher, hub, server_sock);
const auto ec = server.start();
CHECK(!ec);
if (ec) return;
std::thread th([&loop] { loop.run(); });
{
Client c;
auto err = c.connect();
CHECK(!err.has_value());
if (err) {
loop.stop();
th.join();
return;
}
CHECK_EQ(c.hello_result().value("transport", ""), std::string("uds"));
auto ls = c.call("download.list", nlohmann::json::object());
CHECK(ls.has_value());
if (ls) {
CHECK_EQ(ls->value("total", -1), 0);
CHECK(ls->at("items").is_array());
}
// download.add outside every allowed root -> -32011, original saveDir echoed.
auto bad = c.call("download.add",
{{"url", "https://example.com/x"}, {"saveDir", "/etc"}});
CHECK(!bad.has_value());
if (!bad) {
CHECK_EQ(bad.error().code, -32011);
CHECK_EQ(bad.error().data.value("path", ""), std::string("/etc"));
}
// download.add into an allowed root -> a task id that then shows up in the list.
auto ok = c.call(
"download.add",
{{"url", "https://example.com/movie.mp4"}, {"saveDir", g_allowed_root}});
CHECK(ok.has_value());
std::string task_id;
if (ok) {
task_id = ok->value("taskId", "");
CHECK(!task_id.empty());
CHECK_EQ(ok->value("state", ""), std::string("queued"));
}
auto ls2 = c.call("download.list", nlohmann::json::object());
CHECK(ls2.has_value());
if (ls2) {
CHECK_EQ(ls2->value("total", -1), 1);
CHECK_EQ(ls2->at("items").at(0).value("filename", ""), std::string("movie.mp4"));
}
auto detail = c.call("download.get", {{"taskId", task_id}});
CHECK(detail.has_value());
if (detail) CHECK_EQ(detail->at("summary").value("taskId", ""), task_id);
auto missing =
c.call("download.get", {{"taskId", "00000000-0000-4000-8000-000000000000"}});
CHECK(!missing.has_value());
if (!missing) CHECK_EQ(missing.error().code, -32010);
}
loop.stop();
th.join();
::unlink(server_sock.c_str());
}
TEST_MAIN()
+19 -9
View File
@@ -3,13 +3,18 @@
**This directory is the interface between every lane.** Owner: agent **PROTO**.
Nobody else commits here. Everybody else *generates from* here.
> ## Status: **v1.0.0 — FROZEN** (2026-09-09)
> ## Status: **v1.4.0** (frozen at v1.0.0 on 2026-09-09; minor bumps since)
>
> **v1.0.0** froze 38 methods, 9 events, 26 named types. **v1.1.0** (current) is a minor
> bump on top of it: `bufferBytes` bounds widened to 64 KiB - 16 MiB across all four
> locations, two new settings keys (`connection.maxTotalBufferBytes`,
> `connection.maxActiveSegments`), and `TaskDetail.effectiveBufferBytes` — see
> `docs/adr/0012-buffer-and-segment-budget.md`. See also `docs/adr/0005-...` for the
> **v1.0.0** froze 38 methods, 9 events, 26 named types. **v1.1.0** widened `bufferBytes`
> bounds and added the segment-budget settings. **v1.2.0** added `download.provideAuth`
> (F2). **v1.3.0** widened when `error` is populated on a state change to cover a
> daemon-initiated `paused` (for `docs/adr/0013-...`). **v1.4.0** (current) is a
> **C++-binding-only** change: the generated `Dispatcher` gains a `HandlerError` /
> `HandlerResult<T>` error channel so a handler can return `-32010` / `-32011` /
> `-32013` with their `data` payloads instead of collapsing to `-32603`. The wire is
> byte-identical — no schema or fixture change — but any `Dispatcher` implementer
> must swap `Result` → `HandlerResult` on regen. Answered in
> `contracts/proto-answers-daemon-m1.md`. See also `docs/adr/0005-...` for the
> versioning rule and `docs/adr/0010-...` for the failure taxonomy and segment ranges.
>
> Lane requests are answered in writing: `contracts/proto-answers-m1.md` responds to
@@ -30,7 +35,7 @@ Nobody else commits here. Everybody else *generates from* here.
```
contracts/
├── VERSION # protocol semver — frozen at 1.0.0
├── VERSION # protocol semver — v1.4.0, minor-bumped from the v1.0.0 freeze
├── openrpc.json # human-readable API doc (generated from schema/)
├── schema/
│ ├── envelope.schema.json # JSON-RPC 2.0 envelope + our error codes
@@ -59,6 +64,10 @@ subset, `fixtures/` documents the fixture shape and the placeholder rules.
renaming, retyping, or changing a default → **major** bump and a written migration note
in `docs/adr/`. `session.hello` rejects a major mismatch with error `-32001` and a
message the GUI renders as "Velox needs updating".
"Retype → major" is about the **wire** — a field a client parses off the socket. A
change that leaves the wire byte-identical but breaks a *generated binding's* source
API (a C++ virtual's return type, a struct name) is a **minor** bump plus a migration
note — see `docs/adr/0015-generated-binding-changes-and-versioning.md`.
5. **Changes arrive as a PR to `contracts/` alone**, containing: schema edit + fixtures +
regenerated code + `VERSION` bump. Lanes rebase onto it. This is the only synchronization
point in the whole project — keep it cheap and frequent rather than big and rare.
@@ -76,7 +85,7 @@ on rather than as prose a reader has to honour:
| `x-errors` | the error codes this method is documented to return |
| `x-wsRestrictions` | extra limits when the call arrives from the extension |
19 of the 38 methods are privileged: everything that reconfigures the daemon, destroys user
20 of the 39 methods are privileged: everything that reconfigures the daemon, destroys user
data, or names an arbitrary destination path. The extension may *request* a download; it
may not choose where the bytes land.
@@ -93,7 +102,7 @@ All four carry **the same JSON-RPC 2.0 payloads**. The framing differences stop
transport layer; no method behaves differently depending on how it arrived — except that
methods marked `"privileged": true` in the schema are refused over the WebSocket transport.
## Method surface (v1.0.0 target — expand only via PR)
## Method surface (v1.4.0 — expand only via PR)
### Session
| Method | Params → Result |
@@ -114,6 +123,7 @@ methods marked `"privileged": true` in the schema are refused over the WebSocket
| `download.remove` | `{taskIds[], deleteFile:bool}``{removed[]}` |
| `download.update` | `{taskId, patch:{filename?, saveDir?, categoryId?, queueId?, description?, segments?, bufferBytes?}}``TaskSummary` |
| `download.refreshUrl` | `{taskId, url, headers?}``{ok}` *(IDM's "Refresh Download Address")* |
| `download.provideAuth` | `{taskId, username, password, save?}``{ok}` — answers `event.auth.required`. UDS only; privileged. Credentials go to the Secret Service, never SQLite, never logs |
### Organisation
`category.list` · `category.upsert` · `category.remove` · `queue.list` · `queue.upsert` ·
+1 -1
View File
@@ -1 +1 @@
1.1.0
1.4.0
+37 -5
View File
@@ -7,6 +7,13 @@ Design notes that matter to the CORE and DAEMON lanes:
not used and not emitted. Parsing goes through `velox::proto::parse<T>(json)` which
returns `std::expected<T, ParseError>`, so a malformed frame from the wire is an ordinary
value the RPC loop handles, not a throw unwinding through the transfer path.
* **Two error channels, kept separate.** `parse<T>() -> Result<T>` (i.e. `expected<T,
ParseError>`) is the wire failing to become typed params — always `-32602`, always
structural. A `Dispatcher::on_*` handler returns `HandlerResult<T>` (i.e. `expected<T,
HandlerError>`), which carries any contract error code plus a free-form `data` object,
so a handler can answer `-32010` `{taskId}`, `-32011` `{path}`, `-32013` `{httpStatus}`
and so on. `dispatch()` forwards the handler's code/message/data straight into the
JSON-RPC error object.
* **Serialisation is ADL `to_json`,** so `nlohmann::json j = task;` works as expected.
Only the outbound direction is allowed to be implicit; the wire is never trusted.
* **`libveloxproto`, not `libveloxcore`.** This code includes nlohmann/json, which
@@ -285,16 +292,37 @@ def emit_header(c: Contract) -> str:
"nlohmann::json make_result(const nlohmann::json& id, nlohmann::json result);",
"nlohmann::json make_notification(Event e, nlohmann::json params);",
"",
"/// A handler's own failure — as opposed to ParseError, which is the wire failing",
"/// to become typed params. Carries any contract error code, a message, and a",
"/// free-form `data` object that goes straight into the JSON-RPC error's `data`",
"/// field: `{\"taskId\": ...}` for TaskNotFound, `{\"path\": ...}` for InvalidPath,",
"/// `{\"httpStatus\": ...}` for ProbeFailed. `code` defaults to InternalError so a",
"/// handler that sets only a message still produces a valid error response.",
"///",
"/// -32001/-32002/-32003 are the server layer's to raise around dispatch(), not a",
"/// handler's: they are decided before or without reference to method params.",
"struct HandlerError {",
" ErrorCode code{ErrorCode::InternalError};",
" std::string message;",
" // `= nullptr`, not `{nullptr}`: brace-init of nlohmann::json from nullptr",
" // yields the array [null], not JSON null. make_error() drops a null data.",
" nlohmann::json data = nullptr;",
"};",
"",
"template <class T>",
"using HandlerResult = std::expected<T, HandlerError>;",
"",
"/// One virtual per method. The daemon implements this; `dispatch` below does the",
"/// envelope handling, the transport check and the parameter parsing, so a handler",
"/// only ever sees a validated, typed params struct.",
"/// only ever sees a validated, typed params struct. Return `std::unexpected(",
"/// HandlerError{...})` to answer with a specific error code and data.",
"class Dispatcher {",
"public:",
" virtual ~Dispatcher() = default;",
""]
for m in c.methods:
o += doc_comment(m.doc, " ")
o.append(f" virtual Result<{cpp_type(m.result)}> {handler_name(m.name)}(const {cpp_type(m.params)}& params) = 0;")
o.append(f" virtual HandlerResult<{cpp_type(m.result)}> {handler_name(m.name)}(const {cpp_type(m.params)}& params) = 0;")
o.append("")
o += ["};", "",
"/// Parse one JSON-RPC request, route it, and return the response to write back.",
@@ -478,7 +506,12 @@ def emit_field_parse(f: Field, indent: str) -> list[str]:
f'{i} const auto it = j.find("{f.name}");']
if f.optional:
# Absent and null mean the same thing: the field is not set. A client that omits
# a nullable field and one that sends null are treated identically on purpose.
# a nullable field and one that sends null are treated identically on purpose --
# correct for create-style params, where there is no existing value to distinguish
# "never set" from "explicitly cleared". Patch-style fields need the distinction
# (download.update's patch: "an explicit null clears a nullable field") and get an
# opt-in exception via x-clearable per ADR 0018 (not implemented yet: this is the
# decision record, not the generator change).
o.append(f"{i} if (it != j.end() && !it->is_null()) {{")
o += emit_value_parse(f.type, "(*it)", "val", "fp", i + " ")
o.append(f"{i} out.{m} = std::move(val);")
@@ -579,8 +612,7 @@ def emit_dispatch(c: Contract) -> list[str]:
' nlohmann::json{{"path", p.error().path}});',
f" auto r = handler.{handler_name(m.name)}(*p);",
" if (!r)",
" return make_error(id, ErrorCode::InternalError, r.error().message,",
' nlohmann::json{{"path", r.error().path}});',
" return make_error(id, r.error().code, r.error().message, r.error().data);",
" nlohmann::json out = *r;",
" return make_result(id, std::move(out));",
" }",
+13 -4
View File
@@ -44,7 +44,7 @@ def main() -> int:
# velox::conformance, so the names need qualifying.
pt, rt = "proto::" + cpp_type(m.params), "proto::" + cpp_type(m.result)
o += [
f" proto::Result<{rt}> {handler_name(m.name)}(const {pt}& params) override {{",
f" proto::HandlerResult<{rt}> {handler_name(m.name)}(const {pt}& params) override {{",
" (void)params;",
f' return golden<{rt}>("{m.name}");',
" }",
@@ -53,12 +53,21 @@ def main() -> int:
o += [
"private:",
" // Every fixture-backed handler only ever succeeds. A missing or unparseable",
" // fixture is a bug in the suite, not a contract outcome, so it surfaces as",
" // InternalError rather than being dressed up as a real error code.",
" template <class T>",
" proto::Result<T> golden(const std::string& method) {",
" proto::HandlerResult<T> golden(const std::string& method) {",
" const nlohmann::json* value = results_(method);",
" if (value == nullptr)",
' return std::unexpected(proto::ParseError{method, "no fixture for this method"});',
" return proto::parse<T>(*value, method);",
" return std::unexpected(proto::HandlerError{",
' proto::ErrorCode::InternalError, "no fixture for method " + method});',
" auto parsed = proto::parse<T>(*value, method);",
" if (!parsed)",
" return std::unexpected(proto::HandlerError{",
" proto::ErrorCode::InternalError,",
' "fixture for " + method + " failed to parse: " + parsed.error().message});',
" return std::move(*parsed);",
" }",
"",
" std::function<const nlohmann::json*(const std::string&)> results_;",
+25 -1
View File
@@ -21,7 +21,7 @@ fixtures/
```jsonc
{
"name": "download.add — start an ISO now, into the Programs category",
"name": "download.add — add an ISO for later, into the Programs category",
"description": "Why this case is worth pinning.",
"transport": "uds", // optional: replay only on this transport
"requires": "...", // optional: a condition a plain server cannot produce
@@ -81,3 +81,27 @@ cases in `tests/integration/`.
correct response is *no response*: past 750 ms the extension must abandon the offer and let
Firefox download normally. A download manager that eats downloads when its daemon is down
is worse than no download manager.
## No fixture may pair a real external URL with `startMode: "now"`
This suite replays every fixture against a real, live `veloxd` (`tests/conformance/run.sh`),
not just `mockd`. `mockd` never actually fetches anything, so it hid this for a while: a
fixture with `startMode: "now"` (or `"queue"` into a running queue — anything that gets
admitted to the scheduler right away) and a real, resolvable URL makes a **real** daemon
actually start downloading it, for real, onto whatever machine runs the suite. This
happened — twice, with `download.add.json` pointed at a ~6 GB Ubuntu ISO, straight into the
developer's real `~/Downloads`.
The fix in each case is one of:
- `startMode: "later"` — exercises the add path (validation, category assignment, the
event) without ever handing the task to the engine;
- a URL under `example.org`/`example.com` (IANA-reserved for exactly this, RFC 2606) —
resolvable enough to validate as a URL, never a real download source;
- `requires`, if the fixture's entire point needs a real transfer to fail in a specific way
(see `errors/download.add.disk-full.json`) — skipped by default, so it only ever runs
where the condition has actually been arranged.
A real `saveDir` gets the same treatment for the same reason: an absolute path like
`/home/sami/Downloads/...` only means anything on the machine that fixture was written on.
Omit `saveDir` and let `saveTo.defaultDir` apply, or use a relative-feeling path under a
root the runner controls.
+6 -5
View File
@@ -1,22 +1,23 @@
{
"name": "capture.offer — attachment on a monitored type is taken",
"description": "Golden fixture. tests/conformance replays this against the real daemon AND the TS client. If either side drifts, this goes red before the lanes ever integrate.",
"description": "Golden fixture. tests/conformance replays this against the real daemon AND the TS client. If either side drifts, this goes red before the lanes ever integrate. url is example.org (RFC 2606), not a real download source: 'take' against a real veloxd (tests/conformance/run.sh) admits a real task and hands it to the engine for real, and no fixture may do that against a real external URL. contentLength is a plausible-but-small 5 MiB rather than a real ISO's size: the 'Programs' category's saveDir is a migration-seeded builtin (~/Downloads/Programs, daemon/src/store/migrations/0001_initial.sql), not something an isolated test run's settings can redirect, so 'take' always sparse-preallocates into that real path on whatever machine runs this suite -- keeping the declared size small keeps that footprint trivial instead of a real ISO's worth of disk. transport is uds only: a real 'take' persists an active task, so replaying this same fixture again on a second live transport against the same daemon would correctly dedupe against it (capture.offer dedupes by exact URL) and get 'ignore' instead -- an artifact of replaying one fixture against one shared daemon over two transports, not a behaviour to golden.",
"transport": "uds",
"request": {
"jsonrpc": "2.0",
"id": 42,
"method": "capture.offer",
"params": {
"url": "https://releases.ubuntu.com/26.04/ubuntu-26.04-desktop-amd64.iso",
"url": "https://example.org/dl/ubuntu-26.04-desktop-amd64.iso",
"method": "GET",
"tabUrl": "https://releases.ubuntu.com/26.04/",
"tabUrl": "https://example.org/26.04/",
"headers": {
"User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:154.0) Gecko/20100101 Firefox/154.0",
"Referer": "https://releases.ubuntu.com/26.04/",
"Referer": "https://example.org/26.04/",
"Accept": "*/*"
},
"cookies": [],
"contentType": "application/octet-stream",
"contentLength": 6228541440,
"contentLength": 5242880,
"contentDisposition": "attachment; filename=\"ubuntu-26.04-desktop-amd64.iso\"",
"filename": "ubuntu-26.04-desktop-amd64.iso",
"origin": "moz-extension://11111111-2222-3333-4444-555555555555"
+6 -7
View File
@@ -1,6 +1,6 @@
{
"name": "download.add \u2014 start an ISO now, into the Programs category",
"description": "The ordinary add path. saveDir is canonicalized and checked against the allowed roots before anything is written.",
"name": "download.add — add an ISO for later, into the Programs category",
"description": "The ordinary add path. saveDir is canonicalized and checked against the allowed roots before anything is written. startMode is 'later' deliberately: this suite replays against a real veloxd (tests/conformance/run.sh), and a real daemon given startMode 'now' would actually start fetching url for real. No fixture may pair a real external URL with startMode 'now' -- see contracts/fixtures/README.md.",
"request": {
"jsonrpc": "2.0",
"id": 11,
@@ -8,10 +8,9 @@
"params": {
"url": "https://releases.ubuntu.com/26.04/ubuntu-26.04-desktop-amd64.iso",
"filename": "ubuntu-26.04-desktop-amd64.iso",
"saveDir": "/home/sami/Downloads/Programs",
"categoryId": "programs",
"segments": 8,
"startMode": "now"
"startMode": "later"
}
},
"response": {
@@ -19,13 +18,13 @@
"id": 11,
"result": {
"taskId": "$uuid",
"state": "connecting",
"state": "paused",
"duplicate": null
}
},
"assertions": [
"the .veloxpart file is created sparse and preallocated at the final size",
"saveDir resolves inside saveTo.allowedRoots, or the call fails -32011 having written nothing",
"saveDir is omitted here on purpose: it resolves to saveTo.defaultDir, which is itself checked against saveTo.allowedRoots the same way an explicit saveDir would be -- see errors/download.add.invalid-path.json for the -32011 case",
"startMode 'later' lands the task in 'paused' and never hands it to the engine, so nothing is fetched and no .veloxpart is created yet -- that only happens once the task is actually started (download.start.json, or startMode 'now'/'queue' against a source this suite controls)",
"event.task.added is emitted to every subscriber before this reply is sent"
]
}
+1 -1
View File
@@ -20,7 +20,7 @@
],
"defaults": {
"url": "https://example.org/",
"categoryId": "compressed",
"categoryId": "programs",
"startMode": "queue",
"queueId": "main"
}
@@ -0,0 +1,29 @@
{
"name": "download.provideAuth \u2014 answer a 401 challenge and remember it",
"description": "The task was sitting in retry_wait after event.auth.required. This does not restart the transfer itself: the daemon retries with the credential attached and the task's state.transition to connecting/downloading happens on its own, reported the normal way through event.task.state.",
"transport": "uds",
"request": {
"jsonrpc": "2.0",
"id": 80,
"method": "download.provideAuth",
"params": {
"taskId": "$taskId",
"username": "svc-releases",
"password": "hunter2-not-a-real-password",
"save": true
}
},
"response": {
"jsonrpc": "2.0",
"id": 80,
"result": {
"ok": true
}
},
"assertions": [
"the password never appears in a log line, ever, on either side of this call",
"save true stores the credential in the Secret Service keyed by host and realm, not in SQLite",
"the task itself is not touched synchronously by this call \u2014 it moves out of retry_wait when the daemon's own retry succeeds, reported via event.task.state",
"this method is refused with -32003 over the WebSocket transport"
]
}
@@ -0,0 +1,29 @@
{
"name": "download.provideAuth \u2014 the task no longer exists",
"description": "The ordinary stale-client case: the user typed credentials into a dialog for a task that was removed in the meantime.",
"transport": "uds",
"request": {
"jsonrpc": "2.0",
"id": 81,
"method": "download.provideAuth",
"params": {
"taskId": "00000000-0000-4000-8000-000000000000",
"username": "x",
"password": "y"
}
},
"response": {
"jsonrpc": "2.0",
"id": 81,
"error": {
"code": -32010,
"message": "no such task",
"data": {
"taskId": "00000000-0000-4000-8000-000000000000"
}
}
},
"assertions": [
"credentials submitted for a task that no longer exists are discarded, never persisted anywhere"
]
}
@@ -0,0 +1,26 @@
{
"name": "download.provideAuth \u2014 refused over the WebSocket transport",
"description": "The other half of event.auth.required's own promise: a credential-bearing method must never be reachable from the browser.",
"transport": "ws",
"request": {
"jsonrpc": "2.0",
"id": 82,
"method": "download.provideAuth",
"params": {
"taskId": "$taskId",
"username": "x",
"password": "y"
}
},
"response": {
"jsonrpc": "2.0",
"id": 82,
"error": {
"code": -32003,
"message": "method is not permitted on this transport"
}
},
"assertions": [
"the extension has no path to this method under any circumstance"
]
}
@@ -18,7 +18,7 @@
"code": -32001,
"message": "protocol major version mismatch: daemon speaks 1.x, client speaks 2.x",
"data": {
"expected": "1.0.0",
"expected": "$any",
"actual": "2.0.0"
}
}
@@ -27,7 +27,9 @@
"the connection is closed after this reply; no method is served on a mismatched major",
"a differing minor or patch is accepted, never refused",
"the message is safe to show a user verbatim",
"the version check is transport-independent; this is replayed on the Unix socket so it is not masked by -32002"
"the version check is transport-independent; this is replayed on the Unix socket so it is not masked by -32002",
"data.expected is the daemon's own current protocol version string (kProtocolVersion), not a bare major and not pinnable in a golden file -- the conformance compare on error payloads is on `code` only, structural elsewhere, so echoing the live version is fine"
],
"transport": "uds"
"transport": "uds",
"closesConnection": true
}
@@ -0,0 +1,57 @@
{
"name": "event.task.state \u2014 the daemon auto-pauses on a 401",
"description": "The CORE-auto-pause case ADR 0013 (docs/adr/) needs a wire signal for: the daemon paused this task on its own initiative -- not because the user clicked pause, a schedule window closed, or admission control reconciled a lowered cap -- and error explains why. Compare download.pause.json, where the same target state (paused) carries error: null because that pause was requested.",
"notification": {
"jsonrpc": "2.0",
"method": "event.task.state",
"params": {
"taskId": "8c1d4e5f-6a7b-4c8d-9e0f-1a2b3c4d5e6f",
"state": "paused",
"previousState": "connecting",
"summary": {
"taskId": "8c1d4e5f-6a7b-4c8d-9e0f-1a2b3c4d5e6f",
"filename": "film.mkv",
"saveDir": "/home/sami/Downloads/Video",
"url": "https://example.org/film.mkv",
"effectiveUrl": "https://example.org/film.mkv",
"sizeBytes": 1503238553,
"downloadedBytes": 0,
"state": "paused",
"speedBps": 0,
"etaSeconds": null,
"resumable": true,
"segments": 4,
"categoryId": "video",
"queueId": null,
"queuePosition": null,
"description": null,
"createdAt": "$isoDate",
"lastTryAt": "$isoDate",
"completedAt": null,
"error": {
"code": "auth_required",
"message": "the server asked for credentials (401)",
"httpStatus": 401,
"retryable": false,
"cause": null,
"attempt": 1,
"nextRetryAt": null
}
},
"error": {
"code": "auth_required",
"message": "the server asked for credentials (401)",
"httpStatus": 401,
"retryable": false,
"cause": null,
"attempt": 1,
"nextRetryAt": null
}
}
},
"assertions": [
"error is set here specifically because the daemon paused this task itself, not the user -- the trigger is event.auth.required on the same task shortly before",
"the scheduler must not resume this task on a schedule window or queue restart: only download.provideAuth (or the user explicitly resuming) may clear it -- resuming blindly re-fails immediately and looks like a flapping bug",
"a client distinguishes an auto-pause from a deliberate one by this field being non-null, not by inspecting previousState or any other heuristic"
]
}
@@ -50,7 +50,7 @@
}
},
"assertions": [
"error is present exactly when state is failed or retry_wait",
"error is present on every failed or retry_wait transition, and also on a paused transition the daemon entered unilaterally -- never on a paused transition the user or scheduler requested",
"error.code is a TaskErrorCode, never a JSON-RPC ErrorCode \u2014 the two are different spaces",
"retryable false means the scheduler will not pick this up again on its own"
]
+4 -4
View File
@@ -1,6 +1,6 @@
{
"name": "limiter.get \u2014 the limiter is off",
"description": "globalBps still carries the last configured value so the GUI can restore it when the user re-enables the limit.",
"description": "globalBps still carries the last configured value so the GUI can restore it when the user re-enables the limit. applyToRunning is a write-only instruction on limiter.set (\"retune already-running transfers now\", not a persisted setting), so it never comes back from get.",
"request": {
"jsonrpc": "2.0",
"id": 52,
@@ -12,11 +12,11 @@
"id": 52,
"result": {
"enabled": false,
"globalBps": 2097152,
"applyToRunning": false
"globalBps": 2097152
}
},
"assertions": [
"enabled false means no throttling regardless of globalBps"
"enabled false means no throttling regardless of globalBps",
"applyToRunning is absent, not false: it's meaningless outside a limiter.set call"
]
}
+84 -9
View File
@@ -2,7 +2,7 @@
"openrpc": "1.2.6",
"info": {
"title": "Velox Download Manager",
"version": "1.1.0",
"version": "1.4.0",
"description": "The wire contract between veloxd and every client: the Qt GUI, the CLI, the native-messaging host and the Firefox extension. One JSON-RPC 2.0 payload set over four framings; only the framing differs.\n\nGENERATED from contracts/schema/ by contracts/codegen/gen_openrpc.py. Do not edit by hand.",
"license": {
"name": "See repository LICENSE"
@@ -487,9 +487,9 @@
],
"minimum": 1,
"maximum": 32,
"description": "The REQUESTED connection count. An upper bound, not a promise: the daemon lowers it to the per-host cap, and to 1 when the source turns out not to be resumable. What is actually in use comes back as TaskSummary.segments. null means use connection.maxSegmentsPerDownload."
"description": "The REQUESTED connection count. An upper bound, not a promise: the engine lowers it to the per-host cap, and to 1 when the source turns out not to be resumable. What is actually in use comes back as TaskSummary.segments. null means use connection.maxSegmentsPerDownload."
},
"description": "The REQUESTED connection count. An upper bound, not a promise: the daemon lowers it to the per-host cap, and to 1 when the source turns out not to be resumable. What is actually in use comes back as TaskSummary.segments. null means use connection.maxSegmentsPerDownload."
"description": "The REQUESTED connection count. An upper bound, not a promise: the engine lowers it to the per-host cap, and to 1 when the source turns out not to be resumable. What is actually in use comes back as TaskSummary.segments. null means use connection.maxSegmentsPerDownload."
},
{
"name": "bufferBytes",
@@ -1031,6 +1031,79 @@
}
]
},
{
"name": "download.provideAuth",
"summary": "Answer an event.",
"description": "Answer an event.auth.required challenge. The task sits in retry_wait until this arrives; on success the daemon retries with the credentials attached and the task resumes on its own \u2014 this method does not itself start the transfer. Privileged and Unix-socket-only: a credential-bearing method must never be reachable from the browser, which is exactly the boundary event.auth.required's own description draws ('never back through this event, never into a log') \u2014 this is the other half of that promise. Credentials are handed to the Secret Service, never to SQLite and never logged; save only tells the daemon whether to persist them there for next time, or use them for this attempt alone.",
"paramStructure": "by-name",
"params": [
{
"name": "taskId",
"schema": {
"type": "string",
"format": "uuid"
},
"required": true
},
{
"name": "username",
"schema": {
"type": "string",
"maxLength": 256
},
"required": true
},
{
"name": "password",
"schema": {
"type": "string",
"maxLength": 1024
},
"required": true
},
{
"name": "save",
"schema": {
"type": [
"boolean",
"null"
],
"description": "true persists the credential in the Secret Service, keyed by host and realm, for future downloads from the same site. false or null uses it for this task's retry only. Never affects SQLite or the daemon's logs either way."
},
"description": "true persists the credential in the Secret Service, keyed by host and realm, for future downloads from the same site. false or null uses it for this task's retry only. Never affects SQLite or the daemon's logs either way."
}
],
"result": {
"name": "download.provideAuthResult",
"schema": {
"type": "object",
"additionalProperties": false,
"required": [
"ok"
],
"properties": {
"ok": {
"type": "boolean"
}
}
}
},
"x-privileged": true,
"x-transports": [
"uds"
],
"x-deadlineMs": 5000,
"errors": [
{
"code": -32003,
"message": "Method is privileged and was called over a transport that may not use it."
},
{
"code": -32010,
"message": "No task with that id."
}
]
},
{
"name": "download.refreshUrl",
"summary": "IDM's 'Refresh Download Address'.",
@@ -3134,7 +3207,7 @@
],
"minimum": 1,
"maximum": 32,
"description": "The REQUESTED connection count. An upper bound, not a promise: the daemon lowers it to the per-host cap, and to 1 when the source turns out not to be resumable. What is actually in use comes back as TaskSummary.segments. null means use connection.maxSegmentsPerDownload."
"description": "The REQUESTED connection count. An upper bound, not a promise: the engine lowers it to the per-host cap, and to 1 when the source turns out not to be resumable. What is actually in use comes back as TaskSummary.segments. null means use connection.maxSegmentsPerDownload."
},
"bufferBytes": {
"type": [
@@ -4070,7 +4143,7 @@
],
"minimum": 65536,
"maximum": 16777216,
"description": "The write buffer actually in use per live segment, right now. May be well below bufferBytes: the daemon reduces every live segment's buffer to fit connection.maxTotalBufferBytes across connection.maxActiveSegments concurrently-transferring segments, and reports the reduced value here so the GUI can show '16 MiB (using 4 MiB)'. null before the task has started its first segment."
"description": "The write buffer actually in use per live segment, right now. May be well below bufferBytes: the engine reduces every live segment's buffer to fit connection.maxTotalBufferBytes across connection.maxActiveSegments concurrently-transferring segments, and reports the reduced value here so the GUI can show '16 MiB (using 4 MiB)'. null before the task has started its first segment."
},
"partPath": {
"type": [
@@ -4111,7 +4184,7 @@
"title": "TaskDetail"
},
"TaskError": {
"description": "Why a task is in the failed or retry_wait state. Distinct from the JSON-RPC Error, which describes a failed call rather than a failed download \u2014 the two live in different code spaces on purpose, and `code` here is a TaskErrorCode string, never a JSON-RPC integer.",
"description": "Why a task is in the failed, retry_wait, or (when the daemon paused it on its own initiative rather than the user) paused state. Distinct from the JSON-RPC Error, which describes a failed call rather than a failed download \u2014 the two live in different code spaces on purpose, and `code` here is a TaskErrorCode string, never a JSON-RPC integer. A pause the user or the scheduler requested carries no error: this field only explains a paused state the daemon entered unilaterally (auth_required, server_file_changed, disk_full and the like), never a deliberate one.",
"type": "object",
"additionalProperties": false,
"required": [
@@ -4496,7 +4569,8 @@
{
"type": "null"
}
]
],
"description": "Set when state is failed or retry_wait, and also when state is paused and the daemon entered that state on its own initiative rather than at a user's or scheduler's request. null on every other state, including a deliberate pause."
}
},
"title": "TaskSummary"
@@ -4811,7 +4885,7 @@
},
{
"name": "event.task.state",
"description": "A task changed lifecycle state. Carries the summary so the row can be repainted in full without a round trip, and error whenever the new state is failed or retry_wait.",
"description": "A task changed lifecycle state. Carries the summary so the row can be repainted in full without a round trip, and error whenever the daemon has something to say about why: on every failed or retry_wait transition, and on a paused transition the daemon entered unilaterally rather than at a user's or scheduler's request.",
"params": {
"type": "object",
"additionalProperties": false,
@@ -4855,7 +4929,8 @@
{
"type": "null"
}
]
],
"description": "Set when the new state is failed or retry_wait, and also when it is paused and the daemon entered that state on its own initiative \u2014 auth_required, server_file_changed, disk_full and the like \u2014 rather than because of a user action, a schedule window closing, or an admission-control decision. null on every other transition, including every deliberately-requested pause. A client must not assume a paused task has no error just because it usually doesn't; check this field rather than the state name alone."
}
}
},
+82
View File
@@ -0,0 +1,82 @@
# PROTO → DAEMON — answers to `daemon/docs/proto-requests-m1.md`
Status: **answered**. Against `contracts/` at **1.4.0** (`lane/proto`).
Raised by DAEMON while building `rpc/` against 1.3.0.
---
## P1 — the generated `Dispatcher` has no error channel below `-32603` · **landed in 1.4.0**
Done, essentially as sketched. The generated C++ binding now has two error channels,
kept deliberately separate:
| Channel | Type | Raised by | Always |
|---|---|---|---|
| parse | `Result<T>` = `expected<T, ParseError>` | `dispatch()` turning the wire into typed params | `-32602`, structural, `data.path` a JSON pointer |
| handler | `HandlerResult<T>` = `expected<T, HandlerError>` | a `Dispatcher::on_*` method | any contract code + free-form `data` |
```cpp
struct HandlerError {
ErrorCode code{ErrorCode::InternalError}; // default: a bare HandlerError{} is a valid -32603
std::string message;
nlohmann::json data = nullptr; // forwarded straight into the JSON-RPC error's data
};
template <class T> using HandlerResult = std::expected<T, HandlerError>;
```
Every `Dispatcher::on_*` now returns `HandlerResult<T>`. `dispatch()`'s handler-error
branch went from a hard-coded `InternalError` to:
```cpp
if (!r) return make_error(id, r.error().code, r.error().message, r.error().data);
```
So the three in-handler fixtures are now satisfiable by a conformant server:
| Fixture | `return std::unexpected(HandlerError{ ... })` |
|---|---|
| `download.get.not-found` | `ErrorCode::TaskNotFound, "no such task", {{"taskId", id}}` |
| `download.add.invalid-path` | `ErrorCode::InvalidPath, "outside allowed roots", {{"path", p}}` |
| `download.probe.probe-failed` | `ErrorCode::ProbeFailed, "HTTP 403", {{"httpStatus", 403}}` |
`session.pair.rate-limited` (`-32014`) is a handler result too if you want it there —
nothing stops a handler returning `HandlerError{ErrorCode::RateLimited, ...,
{{"retryAfterSec", 60}}}`. `-32001/-32002/-32003` stay yours to raise in the server layer
around `dispatch()`, as you're already doing; they're decided before or without reference
to method params, and `HandlerError`'s own doc comment says so.
Verified end to end: a handler returning each of the above through the real `dispatch()`
path produces the right code with the `data` payload intact, and a bare `HandlerError{}`
still yields a clean `-32603` with no `data` field. (Watch the nlohmann brace-init trap:
`HandlerError{code, msg, {{"k", v}}}` gives an object, but a lone `{nullptr}` would give
the array `[null]` — the struct's member initializer is `= nullptr` for exactly that
reason.)
`FixtureDispatcher` and `conformance_main.cpp`: the generated dispatcher swapped
`Result``HandlerResult` automatically; `conformance_main.cpp` only ever inspects
`dispatch()`'s JSON output and needed no change.
**Version:** minor, 1.3.0 → 1.4.0. The wire is byte-identical — no schema, fixture,
or OpenRPC change — but every implementer of `Dispatcher` must swap `Result`
`HandlerResult` on their `on_*` overrides or they won't compile, and a version bump is
how lanes are told to regenerate and adapt. Minor, not major: a major would make
`session.hello` refuse a client whose wire behaviour is unchanged. The rule — a
generated-binding API break with an unchanged wire is minor + migration note, because
`VERSION` is the protocol version, not the C++ ABI — is written up as
`docs/adr/0015-generated-binding-changes-and-versioning.md` (this instance is
mechanical; the ADR records the rule for the next one, which GUI will also consume
since it already links `velox::proto`). `kProtocolVersion` moves to `"1.4.0"` with it.
## P2 — clarifications
**`session.hello.version-mismatch` `data.expected`.** You're right that `"1.0.0"` in the
fixture is stale. Fixed: it's now `$any`. Echo `kProtocolVersion` (`"1.4.0"`) there — the
conformance compare on an error fixture is on `code` only, structural elsewhere, so the
live version string is fine and can't be pinned in a golden file that outlives version
bumps anyway. `actual` stays the concrete bad version the fake client sent (`"2.0.0"`).
**`SessionHelloResult.transport` always populated.** No change requested, noted. The
field's own description already invites it (`"Lets a client know up front which privileged
methods will be refused"`), so always setting `"uds"` / `"ws"` is using it as intended.
`std::optional` stays because a hand-rolled or older server may legitimately omit it and a
client must tolerate that.
+1 -1
View File
@@ -100,7 +100,7 @@ Agreed with your ranking: these are minor under rule 4 and land as small PRs to
| # | Item | Verdict | Shape |
|---|---|---|---|
| **B2a** | readable effective buffer size | **landed in 1.1.0** | `TaskDetail.effectiveBufferBytes` (placed on `TaskDetail`, not `TaskSummary``bufferBytes` itself was already `TaskDetail`-only, so the pair stays together). See `docs/adr/0012-buffer-and-segment-budget.md`, which also lands B4's bounds and the two new settings keys in the same PR. |
| **F2** | credential return path for 401/407 | **accepted as proposed** | `download.provideAuth {taskId, username, password, save?}``{ok}`. Unix socket only, privileged: a credential-bearing method must never be reachable from the browser. Secrets go to the Secret Service; `save` only tells DAEMON whether to persist. |
| **F2** | credential return path for 401/407 | **landed in 1.2.0** | `download.provideAuth {taskId, username, password, save?}``{ok}`, exactly as proposed: Unix socket only, privileged. It answers the challenge; it does not itself resume the task — the daemon retries with the credential attached and the usual `event.task.state` reports the task leaving `retry_wait`. |
| **F1** | "needs user decision" carrier | **the simple option** | `state: paused` + `event.notify` is the intended carrier for M1: CORE reports `server_file_changed`, DAEMON pauses and notifies, GUI offers restart. A dedicated `event.task.decision` + `download.decide` is a real design with a state machine attached, and it should not be invented in a hurry — raise it again in M3 if the notify path proves too thin. A string comparison on `error.code` covers the engine side either way, which is now a `TaskErrorCode` comparison rather than a magic number. |
| **F3** | `checksum` string format | **already frozen, differently** | `download.add {checksum}` is **not** a string. It is a `Checksum` object: `{algorithm: "md5"\|"sha1"\|"sha256"\|"sha512", value: "<hex>"}`, with `value` patterned `^[0-9a-fA-F]{32,128}$`. Parse your `"<algo>:<hex>"` form at the CLI or GUI edge, not on the wire. Note `sha512` is accepted by the contract even though the appendix lists MD5/SHA-256 — reject it in the engine if you do not implement it, rather than the contract forbidding it. |
@@ -2,7 +2,7 @@
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://velox.dev/schema/events/event.task.state.schema.json",
"title": "event.task.state",
"description": "A task changed lifecycle state. Carries the summary so the row can be repainted in full without a round trip, and error whenever the new state is failed or retry_wait.",
"description": "A task changed lifecycle state. Carries the summary so the row can be repainted in full without a round trip, and error whenever the daemon has something to say about why: on every failed or retry_wait transition, and on a paused transition the daemon entered unilaterally rather than at a user's or scheduler's request.",
"x-direction": "server-to-client",
"type": "object",
"properties": {
@@ -49,7 +49,8 @@
{
"type": "null"
}
]
],
"description": "Set when the new state is failed or retry_wait, and also when it is paused and the daemon entered that state on its own initiative \u2014 auth_required, server_file_changed, disk_full and the like \u2014 rather than because of a user action, a schedule window closing, or an admission-control decision. null on every other transition, including every deliberately-requested pause. A client must not assume a paused task has no error just because it usually doesn't; check this field rather than the state name alone."
}
}
}
@@ -0,0 +1,35 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://velox.dev/schema/methods/download.provideAuth.schema.json",
"title": "download.provideAuth",
"description": "Answer an event.auth.required challenge. The task sits in retry_wait until this arrives; on success the daemon retries with the credentials attached and the task resumes on its own — this method does not itself start the transfer. Privileged and Unix-socket-only: a credential-bearing method must never be reachable from the browser, which is exactly the boundary event.auth.required's own description draws ('never back through this event, never into a log') — this is the other half of that promise. Credentials are handed to the Secret Service, never to SQLite and never logged; save only tells the daemon whether to persist them there for next time, or use them for this attempt alone.",
"x-privileged": true,
"x-transports": ["uds"],
"x-deadlineMs": 5000,
"x-errors": [-32003, -32010],
"type": "object",
"properties": {
"params": {
"type": "object",
"additionalProperties": false,
"required": ["taskId", "username", "password"],
"properties": {
"taskId": { "type": "string", "format": "uuid" },
"username": { "type": "string", "maxLength": 256 },
"password": { "type": "string", "maxLength": 1024 },
"save": {
"type": ["boolean", "null"],
"description": "true persists the credential in the Secret Service, keyed by host and realm, for future downloads from the same site. false or null uses it for this task's retry only. Never affects SQLite or the daemon's logs either way."
}
}
},
"result": {
"type": "object",
"additionalProperties": false,
"required": ["ok"],
"properties": {
"ok": { "type": "boolean" }
}
}
}
}
@@ -80,7 +80,7 @@
],
"minimum": 1,
"maximum": 32,
"description": "The REQUESTED connection count. An upper bound, not a promise: the daemon lowers it to the per-host cap, and to 1 when the source turns out not to be resumable. What is actually in use comes back as TaskSummary.segments. null means use connection.maxSegmentsPerDownload."
"description": "The REQUESTED connection count. An upper bound, not a promise: the engine lowers it to the per-host cap, and to 1 when the source turns out not to be resumable. What is actually in use comes back as TaskSummary.segments. null means use connection.maxSegmentsPerDownload."
},
"bufferBytes": {
"type": [
@@ -65,7 +65,7 @@
],
"minimum": 65536,
"maximum": 16777216,
"description": "The write buffer actually in use per live segment, right now. May be well below bufferBytes: the daemon reduces every live segment's buffer to fit connection.maxTotalBufferBytes across connection.maxActiveSegments concurrently-transferring segments, and reports the reduced value here so the GUI can show '16 MiB (using 4 MiB)'. null before the task has started its first segment."
"description": "The write buffer actually in use per live segment, right now. May be well below bufferBytes: the engine reduces every live segment's buffer to fit connection.maxTotalBufferBytes across connection.maxActiveSegments concurrently-transferring segments, and reports the reduced value here so the GUI can show '16 MiB (using 4 MiB)'. null before the task has started its first segment."
},
"partPath": {
"type": [
+52 -9
View File
@@ -2,17 +2,60 @@
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://velox.dev/schema/types/TaskError.schema.json",
"title": "TaskError",
"description": "Why a task is in the failed or retry_wait state. Distinct from the JSON-RPC Error, which describes a failed call rather than a failed download the two live in different code spaces on purpose, and `code` here is a TaskErrorCode string, never a JSON-RPC integer.",
"description": "Why a task is in the failed, retry_wait, or (when the daemon paused it on its own initiative rather than the user) paused state. Distinct from the JSON-RPC Error, which describes a failed call rather than a failed download \u2014 the two live in different code spaces on purpose, and `code` here is a TaskErrorCode string, never a JSON-RPC integer. A pause the user or the scheduler requested carries no error: this field only explains a paused state the daemon entered unilaterally (auth_required, server_file_changed, disk_full and the like), never a deliberate one.",
"type": "object",
"additionalProperties": false,
"required": ["code", "message", "retryable"],
"required": [
"code",
"message",
"retryable"
],
"properties": {
"code": { "$ref": "https://velox.dev/schema/types/TaskErrorCode.schema.json" },
"message": { "type": "string", "description": "Human-readable, safe to show a user. Never carries a credential, a token or a full local path outside the download roots." },
"httpStatus": { "type": ["integer", "null"], "minimum": 100, "maximum": 599, "description": "Set for the codes listed in TaskErrorCode's x-carriesHttpStatus, and null otherwise." },
"retryable": { "type": "boolean", "description": "Whether the scheduler will pick this task up again on its own. Carried per-occurrence rather than derived from the code, because 'probe_failed' is retryable or not depending on what the probe hit." },
"cause": { "oneOf": [{ "$ref": "https://velox.dev/schema/types/TaskErrorCode.schema.json" }, { "type": "null" }], "description": "The underlying failure, for codes that wrap one. max_retries_exhausted sets it to whatever the last attempt actually failed with, so a user learns the reason rather than just that Velox gave up." },
"attempt": { "type": ["integer", "null"], "minimum": 0, "description": "How many attempts have been made so far." },
"nextRetryAt":{ "type": ["string", "null"], "format": "date-time" }
"code": {
"$ref": "https://velox.dev/schema/types/TaskErrorCode.schema.json"
},
"message": {
"type": "string",
"description": "Human-readable, safe to show a user. Never carries a credential, a token or a full local path outside the download roots."
},
"httpStatus": {
"type": [
"integer",
"null"
],
"minimum": 100,
"maximum": 599,
"description": "Set for the codes listed in TaskErrorCode's x-carriesHttpStatus, and null otherwise."
},
"retryable": {
"type": "boolean",
"description": "Whether the scheduler will pick this task up again on its own. Carried per-occurrence rather than derived from the code, because 'probe_failed' is retryable or not depending on what the probe hit."
},
"cause": {
"oneOf": [
{
"$ref": "https://velox.dev/schema/types/TaskErrorCode.schema.json"
},
{
"type": "null"
}
],
"description": "The underlying failure, for codes that wrap one. max_retries_exhausted sets it to whatever the last attempt actually failed with, so a user learns the reason rather than just that Velox gave up."
},
"attempt": {
"type": [
"integer",
"null"
],
"minimum": 0,
"description": "How many attempts have been made so far."
},
"nextRetryAt": {
"type": [
"string",
"null"
],
"format": "date-time"
}
}
}
@@ -132,7 +132,8 @@
{
"type": "null"
}
]
],
"description": "Set when state is failed or retry_wait, and also when state is paused and the daemon entered that state on its own initiative rather than at a user's or scheduler's request. null on every other state, including a deliberate pause."
}
}
}
+52 -16
View File
@@ -1,11 +1,12 @@
# libveloxcore the download engine. Lane CORE.
#
# No JSON, no SQL, no Qt, no RPC in this tree (CLAUDE.md §3, AGENT-CORE brief).
# This file is self-contained; it is wired into the build by PKG uncommenting
# `add_subdirectory(core)` in the root CMakeLists.txt (see core/docs/pkg-requests-m1.md).
# core/ produces TWO targets (ADR 0009):
# veloxcore the download engine. No JSON, no SQL, no Qt, no RPC. Ever (CLAUDE.md §3).
# veloxproto the generated wire types, which ARE JSON. NOT linked by veloxcore.
# The `no JSON in core/` rule constrains core/src/ and core/include/; core/generated/ is
# the sanctioned exception. Wired in by PKG via add_subdirectory(core) in the root file.
find_package(Threads REQUIRED)
find_package(CURL 8.0 REQUIRED)
find_package(OpenSSL REQUIRED)
add_library(veloxcore STATIC
src/util/error.cpp
@@ -13,6 +14,21 @@ add_library(veloxcore STATIC
src/util/thread_pool.cpp
src/net/curl_error.cpp
src/net/http_client.cpp
src/net/text_codec.cpp
src/net/content_disposition.cpp
src/net/url.cpp
src/net/probe.cpp
src/io/sparse_file.cpp
src/io/write_buffer.cpp
src/meta/veloxpart.cpp
src/segment/segmenter.cpp
src/segment/budget.cpp
src/task/digest.cpp
src/task/download_task.cpp
src/engine.cpp
src/rules/filename.cpp
src/rules/collision.cpp
src/rules/match.cpp
)
add_library(velox::core ALIAS veloxcore)
@@ -29,21 +45,41 @@ target_compile_options(veloxcore PRIVATE
-Wall -Wextra -Wpedantic -Werror
)
target_link_libraries(veloxcore PUBLIC Threads::Threads CURL::libcurl)
target_link_libraries(veloxcore PUBLIC Threads::Threads CURL::libcurl PRIVATE OpenSSL::Crypto)
# Later stages add: find_package(OpenSSL) for meta/ (streaming SHA-256 + resume CRC).
# --- libveloxproto generated wire code (ADR 0009) --------------------------------------
# Its own target so libveloxcore stays JSON-free. Consumed by veloxd, the CLI, the GUI and
# the conformance runner. The root CMakeLists only find_package(nlohmann_json)'s when
# daemon/ has landed, so find it here too this must build even if core is the only lane.
if(NOT TARGET nlohmann_json::nlohmann_json)
find_package(nlohmann_json 3.11 REQUIRED)
endif()
add_library(veloxproto STATIC generated/velox_proto.cpp)
add_library(velox::proto ALIAS veloxproto)
target_include_directories(veloxproto PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/generated)
target_compile_features(veloxproto PUBLIC cxx_std_23)
target_link_libraries(veloxproto PUBLIC nlohmann_json::nlohmann_json)
# Generated code is committed and never hand-edited (CLAUDE.md §2); do not fail the build
# on a codegen quirk that trips -Werror. Warnings stay on for visibility.
target_compile_options(veloxproto PRIVATE -Wall -Wextra -Wno-error)
# A build-time tripwire for the split ADR 0009 exists to protect: veloxcore must never end
# up linking veloxproto.
get_target_property(_core_links veloxcore LINK_LIBRARIES)
if(_core_links AND "veloxproto" IN_LIST _core_links)
message(FATAL_ERROR "veloxcore links veloxproto — ADR 0009 violation (engine sees JSON).")
endif()
if(VELOX_BUILD_TESTS)
add_subdirectory(tests)
endif()
# libFuzzer is clang-only; a GCC configure with -DVELOX_BUILD_FUZZ=ON (the `ci` preset)
# must not hard-fail. No fuzz targets exist yet they arrive with net/probe (stage 3)
# and meta/veloxpart (stage 5).
if(VELOX_BUILD_FUZZ AND NOT CMAKE_CXX_COMPILER_ID MATCHES "Clang")
message(STATUS "veloxcore: VELOX_BUILD_FUZZ set but compiler is "
"${CMAKE_CXX_COMPILER_ID}; fuzz targets need Clang and will be skipped.")
endif()
# When fuzz targets land (stage 3: Content-Disposition, URL; stage 5: .veloxpart.meta),
# they are added here under `if(VELOX_BUILD_FUZZ AND CMAKE_CXX_COMPILER_ID MATCHES "Clang")`
# and live in ${CMAKE_SOURCE_DIR}/tools/fuzz (owned by lane CORE).
# Fuzz targets live in ${CMAKE_SOURCE_DIR}/tools/fuzz (lane CORE) and are wired in by the
# top-level CMakeLists.txt, which add_subdirectory()s every tools/* with a CMakeLists.
# tools/fuzz/CMakeLists.txt self-guards on VELOX_BUILD_FUZZ + a Clang compiler.
# Present: Content-Disposition, URL (stage 3). Coming: .veloxpart.meta (stage 5).
+97
View File
@@ -0,0 +1,97 @@
# CORE response to ADR 0013 (task state-machine ownership)
**Verdict: accept as written.** No amendments to the decision. `daemon/src/sched/`'s
pause/resume logic is unblocked from CORE's side once PROTO lands open item 3 (the
`error`-on-`paused` widening — see below).
The split in §1, the shared-`paused` + idempotency contract in §2, the cross-reason resume
rule in §3, the `retry_wait ≠ starvation` resolution in §4, and the restart handling in §5
all match what CORE agreed in `contracts/proto-answers-m1.md` D1 and the ADR 0011 response.
Everything below is a **design commitment** — CORE's state machine is stage 8 and
`starved_tasks()` is stage 6/8; none of it is built yet. So "does CORE already do X" is
answered as "CORE will do X, and here is the spec it will be built to", not as a report on
existing code.
---
## Answers to the four open items
### 1 — `starved_tasks()` / `tasks_starved` exclude `retry_wait` and auto-paused, by construction
Not a behavior change (nothing to change yet); a commitment baked into the accessor's
definition. Pinning ADR 0011's "running task with `segments_active == 0`":
`tasks_starved` counts **only tasks whose task-state is `connecting` or `downloading` and
whose `segments_active == 0`.** That is precisely "the segment allocator has not granted a
slot to a task that is asking for one." Consequences:
| Task state | In `tasks_starved`? | Why |
|---|---|---|
| `probing` | no | uses the probe pool (ADR 0011 §5), not the segment budget |
| `connecting`, `segments_active == 0` | **yes** | admitted + probed, waiting on the allocator's first grant — the real starvation case |
| `connecting`/`downloading`, `segments_active ≥ 1` | no | a segment in its own `connecting` sub-state counts as a held slot (ADR 0011 A3) |
| `downloading`, `segments_active == 0` | **yes** | held slots, lost them all (e.g. every segment failed and is being re-requested) — transient, still a real "allocator owes this task a slot" |
| `retry_wait` | no | CORE's backoff timer holds it at zero *deliberately*; it is not asking the allocator for anything until it re-enters `connecting` |
| `paused` (DAEMON- **or** CORE-initiated) | no | a paused task is not asking for a slot |
| `new`, `queued`, `assembling`, `verifying`, `complete`, `failed`, `cancelled` | no | not in the `{connecting, downloading}` set |
So `retry_wait` and auto-paused tasks fall outside `tasks_starved` **because they are not
in the counted state set**, not because of a special case that could rot. `starved_tasks()`
returns exactly the TaskIds in that count; `starved_since(id)` is defined only for them.
A `retry_wait` task still counts against `connection.maxConcurrentDownloads` from DAEMON's
side (it is running, not requeued) and contributes nothing to the allocator's guarantee
pass until it re-enters `connecting` — matches §4 exactly.
### 2 — `pause()` is idempotent (committed)
- `pause(TaskId)` on a task already in `paused` (however it got there) → **no-op, returns
success**. DAEMON never needs to know CORE auto-paused first.
- `pause(TaskId)` on a task in a terminal state (`complete`/`failed`/`cancelled`) →
**no-op, returns success** — a pause racing a completion is not an error.
- `resume(TaskId)` on a task that is not paused → **no-op, returns success**.
- The only error `pause()`/`resume()` return is `task_not_found`.
- CORE does not report a state-change event for a no-op pause (nothing changed). DAEMON
reads the resulting state via the normal state-change callback / `TaskDetail`, not from
`pause()`'s return.
- If CORE is mid-transition to `paused` (its own auto-pause) when DAEMON's `pause()`
arrives, the task ends up `paused` once, with one state-change event.
### 3 — PROTO: `error`-on-`paused` widening
Not CORE's to land, but CORE confirms its half: when CORE auto-pauses, it reports the
transition through the same state-change callback every CORE transition uses, with the
`ErrorInfo` populated — `auth_required` (401/407), `server_file_changed`, `disk_full`,
`path_rejected` for a mid-run destination failure. Those are the `vdm::Error` values from
the B1 taxonomy (`core/docs/proto-requests-m1.md`), already implemented in
`core/include/vdm/util/error.hpp`. **CORE is ready**; PROTO only needs to permit `error` to
be present on `event.task.state` when `state == "paused"`, and DAEMON to project CORE's
`ErrorInfo` onto it. `error: null` on a DAEMON-initiated pause is correct and sufficient.
Until PROTO lands it, DAEMON cannot implement §3's cross-reason resume rule
correctness-preservingly — agree it should not guess from timing.
### 4 — "auto-pause" naming
CORE has no established internal term (the state machine is unbuilt). **CORE adopts
"auto-pause"** for the informal concept. On the wire and in the API there is no new term:
the discriminator is `state == paused` plus the `Error` code (present ⇒ CORE-initiated,
absent ⇒ DAEMON-initiated), exactly as the ADR's §2 and the rejected-alternatives section
describe.
---
## Two notes back to DAEMON (not objections)
- **§1 table, `paused` from `assembling`/`verifying`:** the *state transition* is honoured
from any CORE state as the table says. The *work* interruption is best-effort: a pause
during `verifying` discards the in-progress hash and re-hashes from the start of the
file on resume (cheap, bounded); a pause during `assembling` (HLS/DASH mux) is an M4
concern and may not be cleanly interruptible mid-mux. Neither affects M1 or the API
shape.
- **§5 restart:** CORE agrees fully. CORE holds no persistent state; `start(TaskId)`
transparently checks for a valid `.veloxpart.meta` sidecar (stage 5), re-validates with
`If-Range` (`docs/04` §5), and either resumes from the recorded offsets or restarts if
the validator failed / the sidecar is corrupt. DAEMON does nothing special on restart
beyond rewriting CORE-owned states to `queued` and re-admitting — which is what §5 says.
+186
View File
@@ -0,0 +1,186 @@
# CORE → DAEMON — the engine API (M1)
**Status: reviewed and signed off** by DAEMON on 2026-09-10
(`daemon/docs/engine-api-review.md`). Nothing forced a `sched/` or dispatch rewrite. The
five open questions are resolved at the bottom; the review's four confirms are folded into
the headers (`sha512` added to `Checksum::Algo`; the parent-directory contract made
explicit; `cancel()` ordering documented; `rate_limiter()` accessor added).
Integration order: DAEMON wires this in after `daemon/src/sched/` lands (the scheduler is
what calls `start`/`pause`/`resume`/`cancel` and drives `set_task_order`); `sched/` builds
against these headers in parallel with CORE stage 8 and does not need the bodies.
---
This is the `libveloxcore` public surface `veloxd` links and calls
to actually run a download. It is the thing the AGENT-CORE brief asked for on day one and
that slipped: DAEMON has an RPC surface and a store and, until this is agreed, nothing to
call. Sketch headers: `core/include/vdm/engine.hpp`, `core/include/vdm/task/download.hpp`
(both compile now; `Engine` / `DownloadHandle` bodies land in CORE stage 8). Nothing here
touches `contracts/` — DAEMON projects these callbacks onto `TaskSummary` / `TaskDetail` /
`event.*`.
Please review the field semantics, the threading/lifetime rules, and the
pause/resume/cancel contract, and raise anything that would force a `daemon/src/sched/` or
RPC-dispatch rewrite later. Open questions are at the bottom.
---
## 1. `DownloadSpec` — what DAEMON hands the engine
DAEMON has already run the rules engine, picked the category folder, canonicalised the
path and checked it against the allowed roots (`-32011` is DAEMON's error, raised before
`start()`), and resolved the filename. The engine's spec is the concrete result.
| field | who fills it | notes |
|---|---|---|
| `url`, `mirrors` | DAEMON | `mirrors` are alternative URLs for the *same bytes*; the segmenter's requeue prefers a different host after 3 connection failures (docs/04 §3). |
| `headers`, `cookies`, `referrer`, `user_agent` | DAEMON, verbatim from the capture | replayed on every segment request and on the probe. |
| `save_path` | DAEMON | **absolute and final.** The engine never canonicalises or root-checks. `<save_path>.veloxpart` and `.veloxpart.meta` sit beside it; on success the part file is renamed in place. |
| `segments` | user / `connection.maxSegmentsPerDownload` | requested upper bound 132; the engine lowers it to the per-host cap and to 1 for a non-resumable source. Effective value comes back in `Progress.effective_segments`. |
| `buffer_bytes` | user / `connection.bufferBytes` | requested per-segment; silently reduced to fit `maxTotalBufferBytes` across all live segments. Effective value in `Progress.effective_buffer_bytes` → your `TaskDetail.effectiveBufferBytes`. |
| `proxy`, `auth` | DAEMON | `auth` carries credentials only if known up front (Secret Service). Leave `scheme == none` to get an `on_auth_required` on a 401/407 instead. |
| `checksum` | user (`download.add.checksum`) | `{algo, hex}`. Verified during `verifying`; a mismatch is a terminal `failed` with `Error::checksum_mismatch`. |
| `probe_hint` | DAEMON | the `ProbeResult` you already got for the File Info dialog. Supplying it skips the engine's own probe — the task starts in `connecting`, not `probing`. The engine still revalidates with `If-Range` on resume. |
| `allow_resume` | DAEMON | `true`: if a CRC-valid `.veloxpart.meta` sits beside `save_path`, resume from it. `false`: start fresh, overwrite. Your restart flow (ADR 0013 §5) sets this per task. |
| `max_retries` | user / default 10 | per segment, before `failed` with `Error::max_retries_exhausted`. |
`start()` returns immediately. It never throws and never blocks on the network; a bad URL,
DNS failure, or unwritable `save_path` is delivered through `on_finished`.
## 2. The state machine the engine drives
`EngineState` is the CORE-owned subset of the wire `TaskState` (ADR 0013 §1):
```
probing ─▶ connecting ⇄ downloading ─▶ assembling ─▶ verifying ─▶ complete
│ │ ▲ │ │ (M4 mux; a no-op rename in M1)
│ │ └─ retry_wait ┘
▼ ▼
(any) ──────▶ paused ──(resume)──▶ connecting
(any CORE state) ──────────────────▶ failed (terminal, engine-initiated)
(any state, on cancel()) ──────────▶ cancelled (terminal, DAEMON-initiated)
```
`new` and `queued` are yours; the engine never emits them. `start()` corresponds to your
`queued → probing`. Every transition is reported through `on_state(from, to, error?)`,
including the auto-pauses (§4) and the terminals. `previousState` on your
`event.task.state` maps straight from the `from` argument.
## 3. Threading and lifetime
- **Callbacks run on an engine thread** — a transfer worker, the progress timer, or a
dispatch thread — **never** the thread that called `start()` / `pause()` / etc.
- **Per task, callbacks are serialised.** You will never get two callbacks for the same
handle at once. Across tasks they run concurrently.
- **A callback must not block.** It runs on a thread doing real transfer work; a slow
callback stalls that work. Hand off to your own queue/loop.
- **A callback must not re-enter the same handle synchronously** — no `pause()` from
inside `on_state`, etc. Post it. (Calling into a *different* handle, or into
`segment_budget()`, is fine.)
- **`on_finished` is always the last callback** for a task. After it returns the engine
makes no further callbacks for that handle and the handle's control methods are no-ops.
- **The handle is copyable and thread-safe.** Dropping the last copy does **not** cancel —
the task runs on. Call `cancel()` to stop it. (DAEMON holds the handle for the task's
life anyway.)
- **`Engine` must outlive every handle.** `~Engine()` cancels all running tasks and joins
their workers before returning — expect it to block briefly.
- **Logging**: the engine writes through `vdm::set_log_sink()` (a `core/util` global).
Install your sink once at startup; the engine never opens a file itself.
## 4. `paused` is shared, and idempotency is the contract (ADR 0013 §2)
Both sides put a task in `paused`, for disjoint reasons:
- **DAEMON-initiated**: `handle.pause()` — user pause, a schedule window closing, a queue
stop, `Queue.onComplete`, the admission governor reconciling a lowered
`maxActiveSegments`.
- **Engine auto-pause**: `on_auth_required` (401/407), `on_decision_needed`
(`server_file_changed` / stale range), disk full. The engine transitions to `paused`
on its own and fires `on_state(_, paused, ErrorInfo{...})` — the same path as any other
transition. `ErrorInfo.code` present ⇒ engine-initiated; absent ⇒ you did it. That is
the only discriminator, and it is what your `error`-on-`paused` widening (open item 3 of
ADR 0013) carries on the wire.
**Idempotency, now a signature:**
| call | already in that state / terminal | otherwise |
|---|---|---|
| `pause()` | no-op, no error | stop new segment requests, flush + `fdatasync` in-flight buffers, write `.veloxpart.meta`, release the budget slots, `on_state(_, paused, nullopt)`. Bounded by the slowest in-flight flush. |
| `resume()` | no-op if not `paused`; no-op if terminal | revalidate with `If-Range`, re-acquire budget slots, `paused → connecting`, resume from the sidecar offsets. |
| `cancel(discard_partial)` | no-op if already terminal | stop everything, `on_state(_, cancelled, nullopt)`, `on_finished(Err{cancelled})`. `discard_partial` also unlinks `.veloxpart[.meta]` — wire this to `download.remove {deleteFile}`. |
`resume()` after an auto-pause for `auth_required` **without** a preceding
`provide_auth()` is a no-op — the task stays paused. This is ADR 0013 §3's "resume must
not cross reasons", enforced on CORE's side: the scheduler cannot accidentally un-pause a
task waiting on credentials.
- **`provide_auth(user, pass, remember)`** — acts only while the task is auto-paused for
auth; supplies the credential for the retry and resumes. `remember` asks *you* to
persist to the Secret Service; the engine never stores it. No-op otherwise.
- **`decide(Decision)`** — acts only while auto-paused for a decision. `restart` discards
the partial and re-downloads; `keep_partial` continues against what is on disk (the
user's stated risk); `abort``failed`. No-op otherwise.
- **`refresh_url(url, headers)`** — IDM's "Refresh Download Address": the engine re-probes
the new URL to validate it, then **restarts every segment on it** (the signed-URL-expiry
case the wire method exists for), keeping the bytes already on disk. Mirror rotation is
*not* this — that is `spec.mirrors` + the segmenter's requeue-to-a-different-host.
- **`on_decision_needed`** is only for the cases the engine cannot resolve itself. A
routine 416 / stale range is the engine's own re-probe + re-split loop; it escalates
`DecisionRequest{range_metadata_stale}` **only when that loop fails**, at which point
"retry the same range" is already exhausted — so `{restart, keep_partial, abort}` is the
whole choice set.
## 5. Progress
`on_progress` is coalesced to **≤ 4 Hz per task** inside the engine — the same cadence as
`event.task.progress`, so your batcher can forward without re-throttling. It carries
aggregate `downloaded` / `speed_bps` / `eta_seconds`, the effective segment count and
buffer size, and a `SegmentProgress[]` (index, inclusive `[start,end]`, `completed`,
per-segment speed, state) for the GUI's segment bars. `total` is absent for a chunked
source until the stream ends.
`handle.state()` and `handle.progress()` are synchronous lock-guarded snapshots for
`download.get` / `download.list` — call them any time, including from your RPC thread.
## 6. What the engine does NOT do
- No filename resolution, no category matching, no path canonicalisation, no allowed-root
check — all DAEMON, before `start()`.
- No queueing, scheduling, priority, or "when queue completes" — DAEMON, via
`segment_budget().set_task_order()` and by choosing when to call `start()` / `pause()`.
- No persistence beyond `.veloxpart.meta`. On a daemon restart the engine knows nothing;
you reload from SQLite, rewrite CORE-owned states to `queued`, and re-`start()` with
`allow_resume = true` (ADR 0013 §5).
- No credential storage. Ever (`CLAUDE.md` §4).
---
## Resolved (DAEMON review, 2026-09-10)
1. **`probe_hint` stays optional.** DAEMON has a `ProbeResult` only on the File-Info path;
capture-take, `velox add`, `addBatch` and restart have none. The engine probes when
it's absent.
2. **One `cancel(discard_partial)`.** `download.cancel` = `cancel(false)`;
`download.remove` = `cancel(true)` for a live task (plus DAEMON's row/file cleanup), or
pure DAEMON-side for an already-terminal one. No `handle.remove()`.
3. **`{restart, keep_partial, abort}` is the whole set.** The engine owns the routine
416 re-probe/re-split and only escalates `on_decision_needed` when that loop fails —
"retry the same range" is exhausted by then.
4. **Per-task 4 Hz is fine.** DAEMON coalesces across tasks for `event.task.progress`
regardless; `on_progress_batch` is a nice-to-have and must not block stage 8.
5. **`refresh_url` restarts all segments on the new URL** after a validating re-probe (the
signed-URL case). Mirror rotation is `spec.mirrors` + the segmenter, not this.
## Review confirms, folded in
- **(a)** `vdm::TaskId` is a cheap-copy hashable value; DAEMON never constructs one — it
only receives it from `start()` / callbacks and passes it back to `set_task_order()`
etc. ✔ (`vdm/ids.hpp`)
- **(b)** DAEMON `mkdir -p`s `save_path`'s parent before `start()`. The engine opens the
file and fails with `Error::path_rejected` if the directory is missing. ✔ (documented on
`DownloadSpec`)
- **(c)** `Checksum::Algo` now has `sha512`, matching the wire `Checksum` set. ✔
- **(d)** `cancel()` always fires `on_state(_, cancelled, nullopt)` then
`on_finished(Err{Error::canceled})` (note: the taxonomy value is `canceled`), in that
order. ✔ (documented on `DownloadHandle::cancel`)
+104
View File
@@ -0,0 +1,104 @@
# M7 performance baseline
Measured against `docs/04-engine-design.md` §8's targets, via `tools/bench/vdm_bench`
(see that file's header comment for the exact commands — reproduced below with their
actual output). `--preset release`, this machine, 2026-09-11 through 2026-09-12. This is
a baseline record, not a sign-off: one of the three numbers below is loopback-only rather
than measured against a real 1 Gbit link, stated plainly rather than rounded away — see
"Open gaps". The RSS number *did* fail the DoD line in the first pass through this file;
it's since been root-caused and fixed (`docs/adr/0017`), not just re-measured.
## Commands and results
```
$ cmake --preset release && cmake --build --preset release
$ bin/vdm_bench throughput --size 5G --require-mbps 940 --max-cpu-pct 8
throughput: 5368709120 bytes in 2.99s
throughput 14371.00 Mbps
cpu 196.52 % of one core
peak RSS 22.81 MiB
```
Against `tools/bench/support/local_server.hpp`'s busybox loopback server, not a real 1
Gbit link — no such link was available to test against in this environment, so
`--require-mbps`/`--max-cpu-pct` weren't meaningfully exercised here (loopback trivially
clears 940 Mbps; the 196% CPU figure reflects driving a link far faster than 1 Gbit, not
the 1-Gbit-saturated cost the target is about). This needs re-running against a real
1 Gbit peer before it can stand as the actual M1/M7 sign-off number.
```
$ bin/vdm_bench heap-profile --tasks 20 --task-size 4M --top 5
segment budget: 32 live across tasks, engine reports total=32 active=32 starved=9
heap-profile: peak RSS 45.41 MiB, 20 tasks, 8 segments assumed
```
Default `Config` (`default_segments=8`, `max_active_segments=32`, `default_buffer_bytes=1
MiB`), default `--task-size 4M`. **Now clears the 60 MB line** (45.41 MiB), after the real
fix below — root-caused, not just re-measured. Superseded the original `load` run's ~70
MiB number quoted in earlier drafts of this file; see `docs/adr/0017`.
```
$ bin/vdm_bench alloc-check --size 512M --window-s 2
alloc-check: 4 allocations in 2.00s (budget 15)
```
Clears the no-allocation-on-the-hot-path bar (docs/agents/AGENT-CORE.md) comfortably.
This number is *after* a real fix landed in the same change:
`net::HttpClient::Impl::drain_commands` was constructing an (always-allocating, in
libstdc++) `std::deque` on every worker-loop iteration regardless of whether any command
was actually pending — once per curl_multi_poll wake, i.e. on the transfer hot path. Fixed
by checking `w.queue.empty()` under the lock before touching `local` at all. Before the
fix this bench reported thousands of allocations/sec under any sustained transfer.
## ASan / UBSan / TSan (M1 DoD: "20-task load test... clean")
- `--preset dev` (ASan+UBSan) and `--preset tsan`: the full `core/` test suite (40 ctest
cases, including `veloxcore_engine_test`'s hostile-mode suite and `veloxcore_budget_test`)
and all three `tools/bench` smoke tests pass clean on both presets.
- Two real bugs were caught and fixed getting here:
- `DownloadTaskState::quiesce()` (engine shutdown / `Engine`'s destructor) cleared the
`workers` map synchronously right after issuing an async `transfer.cancel()`, racing
the HttpClient worker thread's still-in-flight write callback into a heap-use-after-free
on the segment's ring buffer — ASan-caught via `alloc-check`, which (by design) drops
its `Engine` while a download is still active. Fixed by having `quiesce()` wait for
each worker to drain itself through the same `seg_finished` path every other exit uses,
instead of tearing the map down itself.
- `SegWorker::speed_bps` (the polled-progress fix, see `engine_polled_progress_reports_
nonzero_speed`) was written only by a segment's own curl callback and, before this
session, only ever read from that same thread (`emit_progress_if_due`, called from the
same callback) — safe without synchronization. Reading it from `snapshot_progress()`
(any thread calling `DownloadHandle::progress()`) broke that invariant: `workers_mu`'s
shared_lock protects the `workers` map's structure, not an individual `SegWorker`'s
mutable fields. TSan-caught. Fixed with `std::atomic<double>` (relaxed: this is an
informational EMA, nothing synchronizes real state on it) rather than adding a lock to
the write side.
- The `tools/bench load` ctest registration still runs at reduced concurrency
(`--tasks 8 --segments 2`) under sanitizer presets (`tools/bench/CMakeLists.txt`) from
when this was written against `docs/adr/0016`'s postscript — see `docs/adr/0017`'s "Open
gaps" note: that straggler is now suspected to have been the *same* root cause as the RSS
bug, not re-verified at the DoD's full shape under `--preset tsan` in this change.
## Open gaps
1. ~~RSS is ~70 MB against a 60 MB target~~ **Fixed — see `docs/adr/0017`.** Root cause
was not, as first guessed here, an `ADR 0012` arithmetic gap or per-queued-handle curl
overhead: `SegmentBudget::confirm_slot()` only checked a task's *own* target against
its own held count, never the engine-wide `active_` sum, so it could (and under real
20-task/8-segment contention, reliably did) admit segments well past
`max_active_segments``heap-profile` caught it directly: `budget.active` reading
5686 against a `total` of 32. Fixed at the budget level (the one place that can
actually enforce the invariant); `docs/adr/0012`'s own arithmetic was fine all along.
2. **Throughput/CPU numbers are loopback-only.** No 1 Gbit link was available to test
against; re-run `throughput --size 5G --require-mbps 940 --max-cpu-pct 8` against a real
one before treating this as signed off.
3. **`docs/adr/0016`**: `rate::RateLimiter`'s global-limit path (byte-rate pacing, a
different subsystem from the segment-admission bug in `docs/adr/0017`) has no fairness
ordering under heavy segment contention (a shared `TokenBucket`'s peek/commit race can
starve a waiter indefinitely) — a real, separate, still-open gap for the "global
bandwidth cap with many concurrent downloads" scenario.
4. **The TSan-only load-test straggler** noted in `docs/adr/0016`'s postscript, found
before `docs/adr/0017`'s fix landed: plausibly the *same* root cause (a segment denied
admission with nothing to wake it, worse under TSan's slowdown widening the window a
deferred yield can sit in) rather than the `CURLOPT_LOW_SPEED_TIME` guess that ADR
originally offered — not reverified at the DoD's full 20-task/8-segment shape under
`--preset tsan` in this change (the sanitizer-preset smoke registration still runs at
reduced concurrency; see `tools/bench/CMakeLists.txt`). Worth re-running before treating
it as closed.
@@ -0,0 +1,69 @@
# CORE → PROTO — `tests/conformance/cpp/CMakeLists.txt` should link `veloxproto` now
Status: **open**. Small, mechanical. Filed rather than fixed because `tests/conformance/`
is PROTO's lane.
## What's stale
`tests/conformance/cpp/CMakeLists.txt` says in its header comment:
> Links libveloxproto (the generated protocol code in core/generated/), not libveloxcore
…but it actually **compiles `core/generated/velox_proto.cpp` straight into the
executable** and finds `nlohmann_json` itself:
```cmake
add_executable(velox_conformance_cpp
conformance_main.cpp
${CMAKE_SOURCE_DIR}/core/generated/velox_proto.cpp)
target_include_directories(velox_conformance_cpp PRIVATE
${CMAKE_SOURCE_DIR}/core/generated
${CMAKE_CURRENT_SOURCE_DIR})
target_link_libraries(velox_conformance_cpp PRIVATE nlohmann_json::nlohmann_json)
```
That was the only option while ADR 0009's `libveloxproto` target didn't exist. **It exists
now** — `core/CMakeLists.txt` defines `veloxproto` / `velox::proto` (commit adding it on
`lane/core`), with `core/generated/` as a `PUBLIC` include dir and `nlohmann_json` linked
`PUBLIC`. The comment and the code now agree only if the runner links the target.
## Requested change
```cmake
if(TARGET velox::proto)
add_executable(velox_conformance_cpp conformance_main.cpp)
target_link_libraries(velox_conformance_cpp PRIVATE velox::proto)
else()
# Standalone configure of tests/conformance/ (no core/ in the tree): fall back to
# compiling the generated source directly, as today.
if(NOT TARGET nlohmann_json::nlohmann_json)
find_package(nlohmann_json 3.11 REQUIRED)
endif()
add_executable(velox_conformance_cpp
conformance_main.cpp
${CMAKE_SOURCE_DIR}/core/generated/velox_proto.cpp)
target_include_directories(velox_conformance_cpp PRIVATE
${CMAKE_SOURCE_DIR}/core/generated)
target_link_libraries(velox_conformance_cpp PRIVATE nlohmann_json::nlohmann_json)
endif()
target_include_directories(velox_conformance_cpp PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
target_compile_features(velox_conformance_cpp PRIVATE cxx_std_23)
add_test(NAME conformance_cpp COMMAND velox_conformance_cpp ${CMAKE_SOURCE_DIR})
set_tests_properties(conformance_cpp PROPERTIES LABELS "conformance")
```
The `if(TARGET ...)` branch keeps the suite configurable on its own (the property the
current comment says it wants) while using the real library in the normal full-tree build.
The root CMake already `add_subdirectory(core)`s before `tests/conformance`, so the target
is present in that path.
## Why it matters beyond tidiness
GUI is blocked on `libveloxproto` being a real link target (it can't `add_subdirectory` a
sibling lane's `core/generated/` and re-guess the nlohmann find). Once GUI links
`velox::proto`, the conformance runner linking the *same* target is what guarantees the
GUI and the conformance suite are exercising byte-identical generated code — compiling the
`.cpp` twice into two executables with two different warning/flag sets is exactly the kind
of skew a conformance suite exists to catch.
+127 -77
View File
@@ -3,7 +3,7 @@
//
// Source: contracts/schema/**
// Generator: contracts/codegen/gen_cpp.py
// Contract: v1.1.0
// Contract: v1.4.0
//
// Hand-editing this file is a merge blocker. Fix the schema and regenerate:
// python3 contracts/codegen/gen_cpp.py
@@ -4369,6 +4369,78 @@ template <> Result<DownloadProbeResult> parse<DownloadProbeResult>(const nlohman
return out;
}
void to_json(nlohmann::json& j, const DownloadProvideAuthParams& v) {
j = nlohmann::json::object();
j["taskId"] = v.taskId;
j["username"] = v.username;
j["password"] = v.password;
if (v.save.has_value()) j["save"] = *v.save;
}
template <> Result<DownloadProvideAuthParams> parse<DownloadProvideAuthParams>(const nlohmann::json& j, std::string_view path) {
if (!j.is_object()) return std::unexpected(ParseError{std::string(path), "expected an object"});
DownloadProvideAuthParams out;
{
const std::string fp = join(path, "taskId");
const auto it = j.find("taskId");
if (it == j.end() || it->is_null())
return std::unexpected(ParseError{fp, "required field is missing"});
if (!(*it).is_string()) return std::unexpected(ParseError{std::string(fp), "expected a string"});
auto val = (*it).get<std::string>();
out.taskId = std::move(val);
}
{
const std::string fp = join(path, "username");
const auto it = j.find("username");
if (it == j.end() || it->is_null())
return std::unexpected(ParseError{fp, "required field is missing"});
if (!(*it).is_string()) return std::unexpected(ParseError{std::string(fp), "expected a string"});
auto val = (*it).get<std::string>();
if (val.size() > 256u) return std::unexpected(ParseError{std::string(fp), "value is longer than 256 characters"});
out.username = std::move(val);
}
{
const std::string fp = join(path, "password");
const auto it = j.find("password");
if (it == j.end() || it->is_null())
return std::unexpected(ParseError{fp, "required field is missing"});
if (!(*it).is_string()) return std::unexpected(ParseError{std::string(fp), "expected a string"});
auto val = (*it).get<std::string>();
if (val.size() > 1024u) return std::unexpected(ParseError{std::string(fp), "value is longer than 1024 characters"});
out.password = std::move(val);
}
{
const std::string fp = join(path, "save");
const auto it = j.find("save");
if (it != j.end() && !it->is_null()) {
if (!(*it).is_boolean()) return std::unexpected(ParseError{std::string(fp), "expected a boolean"});
auto val = (*it).get<bool>();
out.save = std::move(val);
}
}
return out;
}
void to_json(nlohmann::json& j, const DownloadProvideAuthResult& v) {
j = nlohmann::json::object();
j["ok"] = v.ok;
}
template <> Result<DownloadProvideAuthResult> parse<DownloadProvideAuthResult>(const nlohmann::json& j, std::string_view path) {
if (!j.is_object()) return std::unexpected(ParseError{std::string(path), "expected an object"});
DownloadProvideAuthResult out;
{
const std::string fp = join(path, "ok");
const auto it = j.find("ok");
if (it == j.end() || it->is_null())
return std::unexpected(ParseError{fp, "required field is missing"});
if (!(*it).is_boolean()) return std::unexpected(ParseError{std::string(fp), "expected a boolean"});
auto val = (*it).get<bool>();
out.ok = std::move(val);
}
return out;
}
void to_json(nlohmann::json& j, const DownloadRefreshUrlParams& v) {
j = nlohmann::json::object();
j["taskId"] = v.taskId;
@@ -6895,6 +6967,7 @@ std::string_view to_string(Method m) noexcept {
case Method::DownloadList: return "download.list";
case Method::DownloadPause: return "download.pause";
case Method::DownloadProbe: return "download.probe";
case Method::DownloadProvideAuth: return "download.provideAuth";
case Method::DownloadRefreshUrl: return "download.refreshUrl";
case Method::DownloadRemove: return "download.remove";
case Method::DownloadResume: return "download.resume";
@@ -6938,6 +7011,7 @@ std::optional<Method> method_from_string(std::string_view s) noexcept {
if (s == "download.list") return Method::DownloadList;
if (s == "download.pause") return Method::DownloadPause;
if (s == "download.probe") return Method::DownloadProbe;
if (s == "download.provideAuth") return Method::DownloadProvideAuth;
if (s == "download.refreshUrl") return Method::DownloadRefreshUrl;
if (s == "download.remove") return Method::DownloadRemove;
if (s == "download.resume") return Method::DownloadResume;
@@ -6981,6 +7055,7 @@ bool is_privileged(Method m) noexcept {
case Method::DownloadList: return false;
case Method::DownloadPause: return false;
case Method::DownloadProbe: return false;
case Method::DownloadProvideAuth: return true;
case Method::DownloadRefreshUrl: return false;
case Method::DownloadRemove: return true;
case Method::DownloadResume: return false;
@@ -7025,6 +7100,7 @@ bool is_allowed_on(Method m, Transport t) noexcept {
case Method::DownloadList: return t == Transport::Uds ? true : true;
case Method::DownloadPause: return t == Transport::Uds ? true : true;
case Method::DownloadProbe: return t == Transport::Uds ? true : true;
case Method::DownloadProvideAuth: return t == Transport::Uds ? true : false;
case Method::DownloadRefreshUrl: return t == Transport::Uds ? true : true;
case Method::DownloadRemove: return t == Transport::Uds ? true : false;
case Method::DownloadResume: return t == Transport::Uds ? true : true;
@@ -7069,6 +7145,7 @@ std::int32_t deadline_ms(Method m) noexcept {
case Method::DownloadList: return 5000;
case Method::DownloadPause: return 5000;
case Method::DownloadProbe: return 30000;
case Method::DownloadProvideAuth: return 5000;
case Method::DownloadRefreshUrl: return 30000;
case Method::DownloadRemove: return 10000;
case Method::DownloadResume: return 5000;
@@ -7168,8 +7245,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_capture_getRules(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7180,8 +7256,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_capture_offer(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7192,8 +7267,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_category_list(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7204,8 +7278,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_category_remove(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7216,8 +7289,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_category_upsert(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7228,8 +7300,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_download_add(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7240,8 +7311,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_download_addBatch(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7252,8 +7322,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_download_cancel(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7264,8 +7333,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_download_get(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7276,8 +7344,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_download_list(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7288,8 +7355,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_download_pause(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7300,8 +7366,18 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_download_probe(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
case Method::DownloadProvideAuth: {
auto p = parse<DownloadProvideAuthParams>(params, "params");
if (!p)
return make_error(id, ErrorCode::InvalidParams, p.error().message,
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_download_provideAuth(*p);
if (!r)
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7312,8 +7388,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_download_refreshUrl(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7324,8 +7399,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_download_remove(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7336,8 +7410,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_download_resume(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7348,8 +7421,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_download_start(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7360,8 +7432,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_download_update(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7372,8 +7443,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_grabber_harvest(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7384,8 +7454,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_grabber_start(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7396,8 +7465,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_grabber_status(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7408,8 +7476,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_limiter_get(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7420,8 +7487,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_limiter_set(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7432,8 +7498,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_media_addVariant(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7444,8 +7509,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_media_listVariants(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7456,8 +7520,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_queue_list(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7468,8 +7531,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_queue_reorder(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7480,8 +7542,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_queue_start(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7492,8 +7553,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_queue_stop(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7504,8 +7564,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_queue_upsert(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7516,8 +7575,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_rules_list(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7528,8 +7586,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_rules_upsert(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7540,8 +7597,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_schedule_get(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7552,8 +7608,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_schedule_set(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7564,8 +7619,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_session_hello(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7576,8 +7630,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_session_pair(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7588,8 +7641,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_session_subscribe(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7600,8 +7652,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_settings_get(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
@@ -7612,8 +7663,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
nlohmann::json{{"path", p.error().path}});
auto r = handler.on_settings_set(*p);
if (!r)
return make_error(id, ErrorCode::InternalError, r.error().message,
nlohmann::json{{"path", r.error().path}});
return make_error(id, r.error().code, r.error().message, r.error().data);
nlohmann::json out = *r;
return make_result(id, std::move(out));
}
+110 -47
View File
@@ -3,7 +3,7 @@
//
// Source: contracts/schema/**
// Generator: contracts/codegen/gen_cpp.py
// Contract: v1.1.0
// Contract: v1.4.0
//
// Hand-editing this file is a merge blocker. Fix the schema and regenerate:
// python3 contracts/codegen/gen_cpp.py
@@ -27,7 +27,7 @@
// docs/adr/0009-generated-protocol-library.md.
namespace velox::proto {
inline constexpr std::string_view kProtocolVersion = "1.1.0";
inline constexpr std::string_view kProtocolVersion = "1.4.0";
/// Why a payload could not be turned into a typed value. `path` is a JSON Pointer
/// into the offending document, so a conformance failure names the exact field.
@@ -533,7 +533,7 @@ struct DownloadSpec {
std::optional<std::string> categoryId{};
/// Required when startMode is 'queue'.
std::optional<std::string> queueId{};
/// The REQUESTED connection count. An upper bound, not a promise: the daemon lowers it to the
/// The REQUESTED connection count. An upper bound, not a promise: the engine lowers it to the
/// per-host cap, and to 1 when the source turns out not to be resumable. What is actually in
/// use comes back as TaskSummary.segments. null means use connection.maxSegmentsPerDownload.
std::optional<std::int64_t> segments{};
@@ -750,9 +750,13 @@ struct Settings {
std::optional<std::int64_t> connection_maxActiveSegments{};
};
/// Why a task is in the failed or retry_wait state. Distinct from the JSON-RPC Error, which
/// describes a failed call rather than a failed download — the two live in different code
/// spaces on purpose, and `code` here is a TaskErrorCode string, never a JSON-RPC integer.
/// Why a task is in the failed, retry_wait, or (when the daemon paused it on its own initiative
/// rather than the user) paused state. Distinct from the JSON-RPC Error, which describes a
/// failed call rather than a failed download — the two live in different code spaces on
/// purpose, and `code` here is a TaskErrorCode string, never a JSON-RPC integer. A pause the
/// user or the scheduler requested carries no error: this field only explains a paused state
/// the daemon entered unilaterally (auth_required, server_file_changed, disk_full and the
/// like), never a deliberate one.
struct TaskError {
TaskErrorCode code{};
/// Human-readable, safe to show a user. Never carries a credential, a token or a full local
@@ -808,6 +812,9 @@ struct TaskSummary {
std::string createdAt{};
std::optional<std::string> lastTryAt{};
std::optional<std::string> completedAt{};
/// Set when state is failed or retry_wait, and also when state is paused and the daemon entered
/// that state on its own initiative rather than at a user's or scheduler's request. null on
/// every other state, including a deliberate pause.
std::optional<TaskError> error{};
};
@@ -828,7 +835,7 @@ struct TaskDetail {
/// use.
std::optional<std::int64_t> bufferBytes{};
/// The write buffer actually in use per live segment, right now. May be well below bufferBytes:
/// the daemon reduces every live segment's buffer to fit connection.maxTotalBufferBytes across
/// the engine reduces every live segment's buffer to fit connection.maxTotalBufferBytes across
/// connection.maxActiveSegments concurrently-transferring segments, and reports the reduced
/// value here so the GUI can show '16 MiB (using 4 MiB)'. null before the task has started its
/// first segment.
@@ -1004,6 +1011,20 @@ struct DownloadProbeResult {
std::optional<bool> requiresAuth{};
};
struct DownloadProvideAuthParams {
std::string taskId{};
std::string username{};
std::string password{};
/// true persists the credential in the Secret Service, keyed by host and realm, for future
/// downloads from the same site. false or null uses it for this task's retry only. Never
/// affects SQLite or the daemon's logs either way.
std::optional<bool> save{};
};
struct DownloadProvideAuthResult {
bool ok{};
};
struct DownloadRefreshUrlParams {
std::string taskId{};
std::string url{};
@@ -1380,6 +1401,12 @@ struct TaskStateEvent {
TaskState state{};
std::optional<TaskState> previousState{};
std::optional<TaskSummary> summary{};
/// Set when the new state is failed or retry_wait, and also when it is paused and the daemon
/// entered that state on its own initiative — auth_required, server_file_changed, disk_full and
/// the like — rather than because of a user action, a schedule window closing, or an
/// admission-control decision. null on every other transition, including every
/// deliberately-requested pause. A client must not assume a paused task has no error just
/// because it usually doesn't; check this field rather than the state name alone.
std::optional<TaskError> error{};
};
@@ -1454,6 +1481,8 @@ void to_json(nlohmann::json& j, const DownloadListResult& v);
void to_json(nlohmann::json& j, const DownloadPauseParams& v);
void to_json(nlohmann::json& j, const DownloadProbeParams& v);
void to_json(nlohmann::json& j, const DownloadProbeResult& v);
void to_json(nlohmann::json& j, const DownloadProvideAuthParams& v);
void to_json(nlohmann::json& j, const DownloadProvideAuthResult& v);
void to_json(nlohmann::json& j, const DownloadRefreshUrlParams& v);
void to_json(nlohmann::json& j, const DownloadRefreshUrlResult& v);
void to_json(nlohmann::json& j, const DownloadRemoveParams& v);
@@ -1599,6 +1628,8 @@ template <> Result<DownloadListResult> parse<DownloadListResult>(const nlohmann:
template <> Result<DownloadPauseParams> parse<DownloadPauseParams>(const nlohmann::json& j, std::string_view path);
template <> Result<DownloadProbeParams> parse<DownloadProbeParams>(const nlohmann::json& j, std::string_view path);
template <> Result<DownloadProbeResult> parse<DownloadProbeResult>(const nlohmann::json& j, std::string_view path);
template <> Result<DownloadProvideAuthParams> parse<DownloadProvideAuthParams>(const nlohmann::json& j, std::string_view path);
template <> Result<DownloadProvideAuthResult> parse<DownloadProvideAuthResult>(const nlohmann::json& j, std::string_view path);
template <> Result<DownloadRefreshUrlParams> parse<DownloadRefreshUrlParams>(const nlohmann::json& j, std::string_view path);
template <> Result<DownloadRefreshUrlResult> parse<DownloadRefreshUrlResult>(const nlohmann::json& j, std::string_view path);
template <> Result<DownloadRemoveParams> parse<DownloadRemoveParams>(const nlohmann::json& j, std::string_view path);
@@ -1686,6 +1717,7 @@ enum class Method {
DownloadList, // download.list
DownloadPause, // download.pause
DownloadProbe, // download.probe
DownloadProvideAuth, // download.provideAuth
DownloadRefreshUrl, // download.refreshUrl
DownloadRemove, // download.remove
DownloadResume, // download.resume
@@ -1714,7 +1746,7 @@ enum class Method {
SettingsSet, // settings.set
};
inline constexpr std::size_t kMethodCount = 38;
inline constexpr std::size_t kMethodCount = 39;
std::string_view to_string(Method m) noexcept;
std::optional<Method> method_from_string(std::string_view s) noexcept;
@@ -1753,9 +1785,30 @@ nlohmann::json make_error(const nlohmann::json& id, ErrorCode code, std::string_
nlohmann::json make_result(const nlohmann::json& id, nlohmann::json result);
nlohmann::json make_notification(Event e, nlohmann::json params);
/// A handler's own failure — as opposed to ParseError, which is the wire failing
/// to become typed params. Carries any contract error code, a message, and a
/// free-form `data` object that goes straight into the JSON-RPC error's `data`
/// field: `{"taskId": ...}` for TaskNotFound, `{"path": ...}` for InvalidPath,
/// `{"httpStatus": ...}` for ProbeFailed. `code` defaults to InternalError so a
/// handler that sets only a message still produces a valid error response.
///
/// -32001/-32002/-32003 are the server layer's to raise around dispatch(), not a
/// handler's: they are decided before or without reference to method params.
struct HandlerError {
ErrorCode code{ErrorCode::InternalError};
std::string message;
// `= nullptr`, not `{nullptr}`: brace-init of nlohmann::json from nullptr
// yields the array [null], not JSON null. make_error() drops a null data.
nlohmann::json data = nullptr;
};
template <class T>
using HandlerResult = std::expected<T, HandlerError>;
/// One virtual per method. The daemon implements this; `dispatch` below does the
/// envelope handling, the transport check and the parameter parsing, so a handler
/// only ever sees a validated, typed params struct.
/// only ever sees a validated, typed params struct. Return `std::unexpected(
/// HandlerError{...})` to answer with a specific error code and data.
class Dispatcher {
public:
virtual ~Dispatcher() = default;
@@ -1763,191 +1816,201 @@ public:
/// The daemon's capture policy, so the extension's shouldCapture decision cannot drift from the
/// daemon's. Fetched on connect and whenever event.settings.changed names a capture.* key. If
/// this call fails the extension keeps its last known rules and stays fail-open.
virtual Result<CaptureRules> on_capture_getRules(const CaptureGetRulesParams& params) = 0;
virtual HandlerResult<CaptureRules> on_capture_getRules(const CaptureGetRulesParams& params) = 0;
/// Firefox offers an intercepted response to the daemon. The daemon MUST reply within 750 ms;
/// the extension abandons the offer and lets Firefox download normally on timeout. This
/// deadline is the whole reason capture fails open, and it is conformance-tested: a daemon that
/// is slow, down, or erroring must never cost the user a download.
virtual Result<CaptureOfferResult> on_capture_offer(const CaptureOfferParams& params) = 0;
virtual HandlerResult<CaptureOfferResult> on_capture_offer(const CaptureOfferParams& params) = 0;
/// Every category with its folder and extension list. The extension calls this to populate its
/// default-category picker, which is why it is not privileged; it is read-only and exposes only
/// paths the user already configured.
virtual Result<CategoryListResult> on_category_list(const CategoryListParams& params) = 0;
virtual HandlerResult<CategoryListResult> on_category_list(const CategoryListParams& params) = 0;
/// Delete a user-created category. Built-in categories are refused with -32602. Tasks filed
/// under it are reassigned to reassignTo, or to the default category when that is null; no task
/// is ever orphaned.
virtual Result<CategoryRemoveResult> on_category_remove(const CategoryRemoveParams& params) = 0;
virtual HandlerResult<CategoryRemoveResult> on_category_remove(const CategoryRemoveParams& params) = 0;
/// Create or replace a category. Omit categoryId to create; supply it to replace. Changing
/// saveDir does not move existing files — the GUI asks separately and issues download.update
/// per task, so a re-point is never a surprise mass file move.
virtual Result<CategoryUpsertResult> on_category_upsert(const CategoryUpsertParams& params) = 0;
virtual HandlerResult<CategoryUpsertResult> on_category_upsert(const CategoryUpsertParams& params) = 0;
/// Create one task. saveDir is canonicalized and checked against saveTo.allowedRoots before
/// anything is written; a path that escapes them is refused with -32011 and no file is created.
virtual Result<DownloadAddResult> on_download_add(const DownloadSpec& params) = 0;
virtual HandlerResult<DownloadAddResult> on_download_add(const DownloadSpec& params) = 0;
/// Create many tasks in one call: the clipboard blob, the wildcard expander, and the
/// extension's 'Download all links'. Partial success is normal and is reported per item rather
/// than failing the whole batch.
virtual Result<DownloadAddBatchResult> on_download_addBatch(const DownloadAddBatchParams& params) = 0;
virtual HandlerResult<DownloadAddBatchResult> on_download_addBatch(const DownloadAddBatchParams& params) = 0;
/// Stop the given tasks and mark them cancelled. The .veloxpart file is kept so the user can
/// still resume from the list; download.remove is what deletes bytes.
virtual Result<BulkTaskResult> on_download_cancel(const DownloadCancelParams& params) = 0;
virtual HandlerResult<BulkTaskResult> on_download_cancel(const DownloadCancelParams& params) = 0;
/// Full detail for one task, including per-segment state. Backs the progress dialog. Poll it no
/// faster than the progress dialog repaints; the table must use events instead.
virtual Result<TaskDetail> on_download_get(const DownloadGetParams& params) = 0;
virtual HandlerResult<TaskDetail> on_download_get(const DownloadGetParams& params) = 0;
/// The main table. Filtering, sorting and paging all happen in the daemon so the GUI never
/// materializes 100k rows to show 40. Called once on connect; after that the table is
/// maintained from events, never re-fetched on a progress tick.
virtual Result<DownloadListResult> on_download_list(const DownloadListParams& params) = 0;
virtual HandlerResult<DownloadListResult> on_download_list(const DownloadListParams& params) = 0;
/// Suspend transfers and flush every segment's progress to the .veloxpart.meta file, so a pause
/// is indistinguishable from a crash as far as resume is concerned. Never loses bytes already
/// written.
virtual Result<BulkTaskResult> on_download_pause(const DownloadPauseParams& params) = 0;
virtual HandlerResult<BulkTaskResult> on_download_pause(const DownloadPauseParams& params) = 0;
/// Ask what is at a URL without creating a task. Populates the File Info dialog. Runs a HEAD,
/// falling back to a ranged GET when HEAD is refused, which is also how resumability is
/// established. Never blocks the RPC loop; the dialog opens immediately and fills in when this
/// lands.
virtual Result<DownloadProbeResult> on_download_probe(const DownloadProbeParams& params) = 0;
virtual HandlerResult<DownloadProbeResult> on_download_probe(const DownloadProbeParams& params) = 0;
/// Answer an event.auth.required challenge. The task sits in retry_wait until this arrives; on
/// success the daemon retries with the credentials attached and the task resumes on its own —
/// this method does not itself start the transfer. Privileged and Unix-socket-only: a
/// credential-bearing method must never be reachable from the browser, which is exactly the
/// boundary event.auth.required's own description draws ('never back through this event, never
/// into a log') — this is the other half of that promise. Credentials are handed to the Secret
/// Service, never to SQLite and never logged; save only tells the daemon whether to persist
/// them there for next time, or use them for this attempt alone.
virtual HandlerResult<DownloadProvideAuthResult> on_download_provideAuth(const DownloadProvideAuthParams& params) = 0;
/// IDM's 'Refresh Download Address'. Point an existing task at a freshly-issued URL when a
/// signed link has expired, keeping every byte already on disk. The daemon re-probes and
/// compares size and validator: if they still match, the transfer resumes from where it
/// stopped; if they do not, it says so rather than silently restarting.
virtual Result<DownloadRefreshUrlResult> on_download_refreshUrl(const DownloadRefreshUrlParams& params) = 0;
virtual HandlerResult<DownloadRefreshUrlResult> on_download_refreshUrl(const DownloadRefreshUrlParams& params) = 0;
/// Drop tasks from the list, optionally deleting the bytes on disk. Privileged: this is the
/// only method that destroys user data, and the extension is never allowed to reach it. The
/// daemon deletes the .veloxpart and .veloxpart.meta pair, and the finished file only when
/// deleteFile is true.
virtual Result<DownloadRemoveResult> on_download_remove(const DownloadRemoveParams& params) = 0;
virtual HandlerResult<DownloadRemoveResult> on_download_remove(const DownloadRemoveParams& params) = 0;
/// Continue paused tasks. Resumption is revalidated with If-Range against the stored ETag or
/// Last-Modified; a 200 where 206 was expected means the file changed on the server, and the
/// task moves to failed with a clear error rather than corrupting the part file.
virtual Result<BulkTaskResult> on_download_resume(const DownloadResumeParams& params) = 0;
virtual HandlerResult<BulkTaskResult> on_download_resume(const DownloadResumeParams& params) = 0;
/// Begin or restart the given tasks. A task in 'queued' jumps its queue; a task already
/// downloading is a no-op reported as changed false.
virtual Result<BulkTaskResult> on_download_start(const DownloadStartParams& params) = 0;
virtual HandlerResult<BulkTaskResult> on_download_start(const DownloadStartParams& params) = 0;
/// Change a task's mutable fields. Moving saveDir or filename moves the file on disk in the
/// same operation, which is what makes dragging a row onto a category work as one RPC.
/// Privileged: it can name a destination path.
virtual Result<TaskSummary> on_download_update(const DownloadUpdateParams& params) = 0;
virtual HandlerResult<TaskSummary> on_download_update(const DownloadUpdateParams& params) = 0;
/// Turn selected crawl results into tasks. This is the only grabber call that creates
/// downloads, and it names exactly the files the user ticked — a crawl never starts a download
/// on its own.
virtual Result<GrabberHarvestResult> on_grabber_harvest(const GrabberHarvestParams& params) = 0;
virtual HandlerResult<GrabberHarvestResult> on_grabber_harvest(const GrabberHarvestParams& params) = 0;
/// Start a depth-limited crawl. Nothing is downloaded by this call: it only walks pages and
/// collects candidate links, which the wizard then shows for selection. Privileged because an
/// unbounded crawl is a resource commitment the browser must not be able to make on the user's
/// behalf.
virtual Result<GrabberStartResult> on_grabber_start(const GrabberStartParams& params) = 0;
virtual HandlerResult<GrabberStartResult> on_grabber_start(const GrabberStartParams& params) = 0;
/// Poll one crawl. Also delivered as event.grabber.progress; the poll exists so the wizard can
/// be reopened on a job it did not start and still catch up.
virtual Result<GrabberStatusResult> on_grabber_status(const GrabberStatusParams& params) = 0;
virtual HandlerResult<GrabberStatusResult> on_grabber_status(const GrabberStatusParams& params) = 0;
/// Current global speed limit. Privileged: changing or reading the limiter belongs to the GUI
/// and CLI; the extension shows throughput from event.speed.global instead.
virtual Result<Limiter> on_limiter_get(const LimiterGetParams& params) = 0;
virtual HandlerResult<Limiter> on_limiter_get(const LimiterGetParams& params) = 0;
/// Set the global token-bucket limit. With applyToRunning true the change re-tunes transfers
/// already in flight instead of taking effect only on the next task — the Speed Limiter
/// window's 'apply now' button.
virtual Result<Limiter> on_limiter_set(const Limiter& params) = 0;
virtual HandlerResult<Limiter> on_limiter_set(const Limiter& params) = 0;
/// Turn one enumerated variant into a task. The daemon fetches the segments in parallel and
/// muxes them with ffmpeg; the result is an ordinary task that appears in the list like any
/// other download. Refused with -32602 when the variant is DRM-protected.
virtual Result<MediaAddVariantResult> on_media_addVariant(const MediaAddVariantParams& params) = 0;
virtual HandlerResult<MediaAddVariantResult> on_media_addVariant(const MediaAddVariantParams& params) = 0;
/// Parse an HLS or DASH manifest in the daemon and enumerate its renditions. The extension
/// never parses a manifest — that logic lives in one language, in one place. Variants with drm
/// true are reported so the UI can grey them out; DRM-protected streams are refused, not
/// attempted.
virtual Result<MediaListVariantsResult> on_media_listVariants(const MediaListVariantsParams& params) = 0;
virtual HandlerResult<MediaListVariantsResult> on_media_listVariants(const MediaListVariantsParams& params) = 0;
/// Every queue with its run state and ordering. Not privileged: the extension's 'Add to Queue'
/// picker needs it.
virtual Result<QueueListResult> on_queue_list(const QueueListParams& params) = 0;
virtual HandlerResult<QueueListResult> on_queue_list(const QueueListParams& params) = 0;
/// Rewrite a queue's run order. taskIds must be a permutation of the queue's current
/// membership; anything else is -32602 rather than a partial reorder, so a stale drag from an
/// out-of-date view cannot quietly reshuffle the queue.
virtual Result<QueueReorderResult> on_queue_reorder(const QueueReorderParams& params) = 0;
virtual HandlerResult<QueueReorderResult> on_queue_reorder(const QueueReorderParams& params) = 0;
/// Start a queue running. The scheduler then admits up to maxConcurrent tasks from it, in
/// order, and keeps that many running until the queue drains or is stopped.
virtual Result<QueueStartResult> on_queue_start(const QueueStartParams& params) = 0;
virtual HandlerResult<QueueStartResult> on_queue_start(const QueueStartParams& params) = 0;
/// Stop admitting new tasks from a queue. Tasks already running are paused when pauseRunning is
/// true, and otherwise allowed to finish — the difference between 'stop the queue' and 'stop
/// everything', which IDM conflates and users trip over.
virtual Result<QueueStopResult> on_queue_stop(const QueueStopParams& params) = 0;
virtual HandlerResult<QueueStopResult> on_queue_stop(const QueueStopParams& params) = 0;
/// Create or replace a queue, including its schedule and concurrency cap. Omit queueId to
/// create. taskIds in the payload is ignored — membership changes through download.update and
/// queue.reorder so that two clients editing at once cannot silently drop a task.
virtual Result<QueueUpsertResult> on_queue_upsert(const QueueUpsertParams& params) = 0;
virtual HandlerResult<QueueUpsertResult> on_queue_upsert(const QueueUpsertParams& params) = 0;
/// The rules engine's table, in priority order. Privileged: these are the daemon's routing
/// policy. The extension gets its own narrowed view through capture.getRules instead.
virtual Result<RulesListResult> on_rules_list(const RulesListParams& params) = 0;
virtual HandlerResult<RulesListResult> on_rules_list(const RulesListParams& params) = 0;
/// Create, replace, or delete rules in one atomic write. 'upsert' carries the rules to store
/// and 'remove' the ruleIds to drop; applying both at once means a reprioritisation never
/// leaves the table in a half-valid state.
virtual Result<RulesUpsertResult> on_rules_upsert(const RulesUpsertParams& params) = 0;
virtual HandlerResult<RulesUpsertResult> on_rules_upsert(const RulesUpsertParams& params) = 0;
/// The schedule for one queue, or every schedule when queueId is null. Backs the Scheduler
/// window.
virtual Result<ScheduleGetResult> on_schedule_get(const ScheduleGetParams& params) = 0;
virtual HandlerResult<ScheduleGetResult> on_schedule_get(const ScheduleGetParams& params) = 0;
/// Set or clear a queue's schedule. A null schedule clears it and leaves the queue under manual
/// control. Times are local wall-clock and are re-evaluated on a DST change rather than being
/// resolved to absolute instants at set time.
virtual Result<ScheduleSetResult> on_schedule_set(const ScheduleSetParams& params) = 0;
virtual HandlerResult<ScheduleSetResult> on_schedule_set(const ScheduleSetParams& params) = 0;
/// First call on every connection, on every transport. The daemon compares protocolVersion
/// majors and refuses a mismatch with -32001 so a stale GUI or extension fails loudly on
/// connect instead of subtly at the tenth field. On the WebSocket transport a valid token is
/// required unless the client is about to call session.pair.
virtual Result<SessionHelloResult> on_session_hello(const SessionHelloParams& params) = 0;
virtual HandlerResult<SessionHelloResult> on_session_hello(const SessionHelloParams& params) = 0;
/// WebSocket transport only. Triggers a GUI or desktop-notification prompt showing a four-digit
/// code; the user must approve before a token is issued. Failed attempts are rate-limited to
/// 5/min followed by a 60 s lockout (-32014) so a token cannot be brute-forced by another local
/// process. The daemon stores only a hash of the token.
virtual Result<SessionPairResult> on_session_pair(const SessionPairParams& params) = 0;
virtual HandlerResult<SessionPairResult> on_session_pair(const SessionPairParams& params) = 0;
/// Choose which notifications this connection receives. Subscribing replaces the previous
/// selection rather than adding to it, so a client can narrow its firehose without
/// reconnecting. Nothing is delivered until this is called.
virtual Result<SessionSubscribeResult> on_session_subscribe(const SessionSubscribeParams& params) = 0;
virtual HandlerResult<SessionSubscribeResult> on_session_subscribe(const SessionSubscribeParams& params) = 0;
/// Read settings. keys null means everything. Privileged: the settings bag names local
/// filesystem paths and the allowed write roots, which the extension has no business
/// enumerating — it gets capture.getRules instead.
virtual Result<SettingsGetResult> on_settings_get(const SettingsGetParams& params) = 0;
virtual HandlerResult<SettingsGetResult> on_settings_get(const SettingsGetParams& params) = 0;
/// Write settings. Only the keys present in values change. Rejected with -32602 if a key is
/// unknown or a value fails the Settings schema, and with -32011 if a directory key names a
/// path that cannot be written. Emits event.settings.changed with exactly the keys that took
/// effect.
virtual Result<SettingsSetResult> on_settings_set(const SettingsSetParams& params) = 0;
virtual HandlerResult<SettingsSetResult> on_settings_set(const SettingsSetParams& params) = 0;
};
+48 -3
View File
@@ -1,8 +1,12 @@
# `libveloxcore` — public API
**Status: M1 in progress.** Only `util/` is landed. The download-facing API
(`DownloadSpec`, `DownloadTask`, probe, typed callbacks) arrives with later stages and
is reviewed by DAEMON before M2 (AGENT-CORE DoD).
**Status: M1 in progress.** `util/`, `net/` (http_client, probe, url, content_disposition),
`io/` (sparse_file, write_buffer), `meta/veloxpart`, `segment/` (segmenter, budget),
`rate/` (token_bucket), `task/`+`engine.hpp` (the download engine itself — `vdm::Engine`,
`DownloadSpec`/`DownloadHandle`/`DownloadCallbacks`), and `rules/` (filename sanitization,
collision policy, rule-table matching) are landed. `media/` is M4, not started — see
[`core/docs/engine-api-m1.md`](../../docs/engine-api-m1.md) for the engine API's own
DAEMON-review history.
Layering (CLAUDE.md §3): this library knows nothing about JSON, SQL, Qt, or RPC. Input is
a spec value; output is bytes on disk plus typed callbacks. DAEMON projects engine state
@@ -71,3 +75,44 @@ Sink interface — core does no I/O itself. `LogSink` abstract base; DAEMON inst
via `set_log_sink()`, default discards. `CallbackSink` adapter (with a min-level filter).
`VDM_LOG_{TRACE,DEBUG,INFO,WARN,ERROR}(category, fmt, args...)``std::format` syntax,
only formatted when a sink is installed and wants the level.
---
## `rules/` — filename sanitization, collision policy, rule-table matching
Pure functions only: no I/O, no filesystem access, no notion of the wire `Rule` type or
its JSON/SQL representation. DAEMON owns the rule table (storage, `rules.upsert`, the
generated `Rule` type) and decodes it into the plain structs below before calling in.
### `vdm/rules/filename.hpp`
`sanitize_filename(raw, max_bytes = 255)` — turns a raw candidate (from
`net::parse_content_disposition` or `net::url_filename`, neither of which is
filesystem-safe by design — see their own headers) into one safe to create on ext4, APFS,
and NTFS alike: strips separators/control bytes, folds NTFS-illegal characters to `_`,
neutralizes reserved Windows device names (`CON`, `COM1`, ...), and clamps length on a
UTF-8 boundary. Total: never empty, never throws. **Not** the path-traversal security
boundary — that's DAEMON's `fs/safepath`, which runs after this and is the one that
matters adversarially.
### `vdm/rules/collision.hpp`
`resolve_collision(desired, exists, policy, max_attempts = 1000)` — given an existence
predicate (DAEMON supplies a real one; tests supply an in-memory set), finds the next free
name Explorer/Finder-style (`"name (1).ext"`, `"name (2).ext"`, ...) under
`CollisionPolicy::rename`, or returns `desired` unchanged under `::overwrite`. Never
fabricates a guaranteed-unique name past `max_attempts` — returns the last candidate tried
and leaves "still colliding" for the caller to treat as a real error.
### `vdm/rules/match.hpp`
`match_rules(rules, input) -> optional<RuleAction>` — the evaluation half of
`contracts/schema/types/Rule.schema.json`: tries rules in ascending `priority` order
(ties keep table order), skips disabled rows, returns the first whose every *present*
match clause (`extensions`, `mime_types`, `host_pattern`, `url_pattern`,
`min_size_bytes`/`max_size_bytes`) is satisfied — an absent clause is not a constraint,
and a size clause never matches speculatively when `MatchInput::size_bytes` is still
unknown (pre-probe). `std::nullopt` means no rule matched; the caller's own default
category applies. `glob_match(pattern, text)` — the `*`/`?` matcher `host_pattern` and
`url_pattern` both use, case-insensitive, bounded work even on a pathological
all-`*` pattern (iterative, not recursive).
+82
View File
@@ -0,0 +1,82 @@
// vdm/engine.hpp — the download engine's single entry point. REVIEW SKETCH (stage 7
// pre-work); bodies land in stage 8. See core/docs/engine-api-m1.md.
//
// The engine owns the HTTP client, the probe pool, the segment budget, and the disk I/O.
// Its input is a DownloadSpec; its output is bytes at save_path plus typed callbacks. No
// JSON, no SQL, no Qt, no RPC — DAEMON projects the callbacks onto the wire contract.
//
// This header compiles standalone.
#ifndef VDM_ENGINE_HPP
#define VDM_ENGINE_HPP
#include <cstdint>
#include <memory>
#include "vdm/rate/token_bucket.hpp"
#include "vdm/segment/budget.hpp"
#include "vdm/task/download.hpp"
namespace vdm {
class Engine {
public:
struct Config {
// Defaults used when a DownloadSpec leaves the field unset. Live-adjustable via
// the setters below (they take effect on the next segment (re)assignment, not by
// resizing an in-flight buffer).
std::uint32_t default_segments = 8; // connection.maxSegmentsPerDownload
std::uint64_t default_buffer_bytes = 1u << 20; // connection.bufferBytes (1 MiB)
std::uint64_t min_segment_bytes = 1u << 20; // never split below this
std::uint64_t max_total_buffer_bytes = 128ull << 20; // connection.maxTotalBufferBytes
std::uint32_t max_active_segments = 32; // connection.maxActiveSegments
std::uint32_t probe_pool_size = 4; // ADR 0011 §5, outside the budget
long default_max_retries = 10; // per segment
std::uint32_t http_workers = 0; // 0 => hardware-derived (<=4)
};
Engine(); // default Config
explicit Engine(Config cfg);
~Engine(); // cancels every running task and joins before returning
Engine(const Engine &) = delete;
Engine &operator=(const Engine &) = delete;
// Start a download. Returns immediately with a handle; the task begins in `probing`
// (or `connecting` when spec.probe_hint is supplied). Every failure — bad URL, DNS,
// an unwritable save_path — is delivered through callbacks.on_finished, never thrown.
[[nodiscard]] task::DownloadHandle start(task::DownloadSpec spec,
task::DownloadCallbacks callbacks);
// The global segment allocator. DAEMON's scheduler drives admission through this
// (set_max_active_segments / set_host_segment_cap / set_task_order) and reads
// occupancy from it (budget() / segments_active() / starved_tasks() /
// on_budget_changed). See ADR 0011.
[[nodiscard]] segment::SegmentBudget &segment_budget() noexcept;
// The hierarchical speed limiter (docs/04 §6): global -> per-queue -> per-task token
// buckets. `limiter.set {globalBps, enabled}` -> rate_limiter().set_global_limit();
// per-queue / per-task limits and the task<->queue attachment come from DAEMON too.
// The engine paces every segment read through it.
[[nodiscard]] rate::RateLimiter &rate_limiter() noexcept;
// Live settings (connection.* changes from settings.set). Each affects future work.
void set_default_segments(std::uint32_t n);
void set_default_buffer_bytes(std::uint64_t bytes);
void set_max_total_buffer_bytes(std::uint64_t bytes);
void set_probe_pool_size(std::uint32_t n);
// A standalone probe for the File Info dialog, on the same pool as spec-less probes
// (never charged against the segment budget). capture.offer's 750 ms deadline is
// DAEMON's to enforce — it should answer `ignore` and probe after, never block on
// this.
void probe(net::ProbeRequest req, std::function<void(Result<net::ProbeResult>)> done);
private:
struct Impl;
std::unique_ptr<Impl> impl_;
};
} // namespace vdm
#endif // VDM_ENGINE_HPP
+54
View File
@@ -0,0 +1,54 @@
// vdm/ids.hpp — opaque engine-internal identifiers.
//
// TaskId is CORE's handle for a download. DAEMON owns the wire UUID and keeps a
// UUID <-> TaskId map; CORE never sees the UUID (layering rule — no wire types in the
// engine). Assigned by CORE when DAEMON registers a task.
//
// This header compiles standalone.
#ifndef VDM_IDS_HPP
#define VDM_IDS_HPP
#include <chrono>
#include <compare>
#include <cstddef>
#include <cstdint>
#include <functional>
namespace vdm {
struct TaskId {
std::uint64_t value = 0;
[[nodiscard]] constexpr bool valid() const noexcept { return value != 0; }
friend constexpr auto operator<=>(const TaskId &, const TaskId &) = default;
};
// A scheduler queue. A task may belong to one — for the per-queue rate limit and
// per-queue concurrency. A task with no queue rate-limits against the global bucket only.
struct QueueId {
std::uint64_t value = 0;
[[nodiscard]] constexpr bool valid() const noexcept { return value != 0; }
friend constexpr auto operator<=>(const QueueId &, const QueueId &) = default;
};
using SteadyTime = std::chrono::steady_clock::time_point;
} // namespace vdm
template <>
struct std::hash<vdm::TaskId> {
std::size_t operator()(vdm::TaskId id) const noexcept {
return std::hash<std::uint64_t>{}(id.value);
}
};
template <>
struct std::hash<vdm::QueueId> {
std::size_t operator()(vdm::QueueId id) const noexcept {
return std::hash<std::uint64_t>{}(id.value);
}
};
#endif // VDM_IDS_HPP
+80
View File
@@ -0,0 +1,80 @@
// vdm/io/sparse_file.hpp — the one output file, written at absolute offsets.
//
// docs/04 §4: one file, opened once, O_WRONLY; posix_fallocate the full size up front;
// each segment pwrite()s at its own offset so there is no reassembly pass; fadvise
// DONTNEED on written ranges; fdatasync on a timer, never per write.
//
// This header compiles standalone.
#ifndef VDM_IO_SPARSE_FILE_HPP
#define VDM_IO_SPARSE_FILE_HPP
#include <cstdint>
#include <string>
#include <string_view>
#include "vdm/util/bytes.hpp"
#include "vdm/util/result.hpp"
namespace vdm::io {
class SparseFile {
public:
struct OpenOptions {
std::uint64_t total_size = 0; // full final size; 0 = unknown (chunked transfer)
bool preallocate = true; // posix_fallocate; falls back to ftruncate
bool truncate_existing = false; // start fresh (true) vs. resume into an existing
// part file (false)
};
SparseFile() = default;
~SparseFile();
SparseFile(SparseFile &&) noexcept;
SparseFile &operator=(SparseFile &&) noexcept;
SparseFile(const SparseFile &) = delete;
SparseFile &operator=(const SparseFile &) = delete;
// Open `path` for writing at absolute offsets, creating it if needed. With
// `preallocate` and a known `total_size`, posix_fallocate the whole file (contiguous
// extents, no ENOSPC surprise at 99%). EOPNOTSUPP/ENOSYS (tmpfs, some network FS)
// falls back to ftruncate — sparse, no extent reservation — and is reported by
// preallocated().
[[nodiscard]] Result<void> open(std::string_view path, const OpenOptions &opts);
[[nodiscard]] Result<void> open(std::string_view path); // default OpenOptions
[[nodiscard]] bool is_open() const noexcept { return fd_ >= 0; }
[[nodiscard]] bool preallocated() const noexcept { return preallocated_; }
[[nodiscard]] std::string_view path() const noexcept { return path_; }
// pwrite the whole span at `offset`, looping over short writes and retrying EINTR.
// Safe to call concurrently with other write_at()/sync() on the same object as long
// as the byte ranges do not overlap — POSIX guarantees each pwrite is atomic for a
// regular file, so no lock is taken on the hot path.
[[nodiscard]] Result<void> write_at(std::uint64_t offset, ConstByteSpan data);
// fdatasync. Call on a timer (default 5 s) and on pause — never per write (docs/04 §4).
[[nodiscard]] Result<void> sync();
// POSIX_FADV_DONTNEED on [offset, offset+len): drop already-written pages from the
// page cache so a 40 GB ISO does not evict the user's working set. Most effective
// after sync(). Best-effort — failures are ignored.
void advise_dontneed(std::uint64_t offset, std::uint64_t len) noexcept;
// ftruncate to `size`: give a chunked download its real size once known, or trim a
// preallocated tail that a steal/mirror never filled.
[[nodiscard]] Result<void> resize(std::uint64_t size);
[[nodiscard]] Result<void> close();
private:
void reset() noexcept;
int fd_ = -1;
bool preallocated_ = false;
std::string path_;
};
} // namespace vdm::io
#endif // VDM_IO_SPARSE_FILE_HPP
+69
View File
@@ -0,0 +1,69 @@
// vdm/io/write_buffer.hpp — per-segment accumulate-and-flush buffer.
//
// docs/04 §4: curl's write callback appends; the buffer is flushed with a single pwrite
// when full or when the segment ends. §8: NO ALLOCATION in the write-callback hot path —
// the buffer is preallocated at construction and append() only memcpys.
//
// Single-threaded. The owning download_task (stage 8) adds the disk-writer-thread handoff
// (double buffering) on top; this primitive takes no lock.
//
// This header compiles standalone.
#ifndef VDM_IO_WRITE_BUFFER_HPP
#define VDM_IO_WRITE_BUFFER_HPP
#include <cstddef>
#include <cstdint>
#include <functional>
#include <vector>
#include "vdm/util/bytes.hpp"
#include "vdm/util/result.hpp"
namespace vdm::io {
class WriteBuffer {
public:
// Writes `span` durably at absolute file offset `offset`. Must not allocate
// (SparseFile::write_at doesn't). Return an error to abort the segment; WriteBuffer
// propagates it and keeps the un-flushed bytes so the caller can decide.
using FlushFn = std::function<Result<void>(std::uint64_t offset, ConstByteSpan span)>;
// `capacity` is the effective per-segment buffer_bytes (already clamped by the
// segmenter against max_total_buffer_bytes). Must be > 0.
WriteBuffer(std::uint64_t start_offset, std::size_t capacity, FlushFn flush);
WriteBuffer(WriteBuffer &&) noexcept = default;
WriteBuffer &operator=(WriteBuffer &&) noexcept = default;
WriteBuffer(const WriteBuffer &) = delete;
WriteBuffer &operator=(const WriteBuffer &) = delete;
// Append body bytes. Flushes automatically each time the buffer fills. A chunk at
// least `capacity` bytes long, arriving when the buffer is empty, is written straight
// through (one extra flush call, still no allocation and no memcpy).
[[nodiscard]] Result<void> append(ConstByteSpan span);
// Flush whatever is buffered right now. Call at segment end and on pause. A no-op
// (success) when nothing is pending.
[[nodiscard]] Result<void> flush();
[[nodiscard]] std::size_t capacity() const noexcept { return buf_.size(); }
[[nodiscard]] std::size_t pending() const noexcept { return len_; }
// File offset the next flush will write at (start + all bytes already flushed).
[[nodiscard]] std::uint64_t next_offset() const noexcept { return base_; }
// Total bytes handed to append() over this buffer's life.
[[nodiscard]] std::uint64_t total_appended() const noexcept { return appended_; }
private:
Result<void> flush_pending();
std::vector<std::byte> buf_;
std::size_t len_ = 0; // bytes currently in buf_
std::uint64_t base_ = 0; // file offset of buf_[0]
std::uint64_t appended_ = 0;
FlushFn flush_;
};
} // namespace vdm::io
#endif // VDM_IO_WRITE_BUFFER_HPP
+109
View File
@@ -0,0 +1,109 @@
// vdm/meta/veloxpart.hpp — the `<name>.veloxpart.meta` resume sidecar (docs/04 §5).
//
// Written next to the part file so a download survives a daemon crash, a reboot, and a
// database loss. Little-endian, versioned, CRC-32 over the whole record, fdatasync'd at
// segment boundaries.
//
// The reader is written first and fuzzed (AGENT-CORE §5): this file lives in a
// world-writable-ish download directory, so parse_veloxpart() is total on hostile input —
// every malformation is a Result error (meta_corrupt / meta_version_unsupported), never a
// crash, an over-read, or an unbounded allocation.
//
// On-disk layout (all integers little-endian):
//
// magic "VDMP" 4 bytes
// version u16 (this build writes/reads kVersion)
// flags u16 (bit0: sha256_state present)
// total_size u64 (0 = unknown / chunked)
// downloaded u64 (sum of segment.completed; a fast read)
// url_count u32
// url[0] = original, url[1] = effective, url[2..] = mirrors, each length-prefixed
// etag length-prefixed UTF-8
// last_modified length-prefixed UTF-8
// content_type length-prefixed UTF-8
// segment_count u32
// per segment: start u64, end u64 (INCLUSIVE), completed u64
// [flags bit0] sha256_state_len u32, then that many opaque bytes
// crc32 u32 (over every byte before this field)
//
// length-prefixed = u32 length, then that many bytes.
//
// This header compiles standalone.
#ifndef VDM_META_VELOXPART_HPP
#define VDM_META_VELOXPART_HPP
#include <cstdint>
#include <string>
#include <string_view>
#include <vector>
#include "vdm/util/bytes.hpp"
#include "vdm/util/result.hpp"
namespace vdm::meta {
inline constexpr std::uint16_t kVersion = 1;
inline constexpr std::uint16_t kFlagHasShaState = 0x0001;
// Hard caps the reader enforces so a hostile count/length can't drive allocation or work.
inline constexpr std::uint32_t kMaxUrls = 64;
inline constexpr std::uint32_t kMaxSegments = 1024; // contract ceiling is 32; headroom
inline constexpr std::uint32_t kMaxStringLen = 16 * 1024;
inline constexpr std::uint32_t kMaxShaStateLen = 4 * 1024;
inline constexpr std::size_t kMaxImageBytes = 256 * 1024; // a real sidecar is < 4 KiB
struct SegmentRecord {
std::uint64_t start = 0;
std::uint64_t end = 0; // INCLUSIVE, per contract Segment.endByte / ADR 0010
std::uint64_t completed = 0;
[[nodiscard]] std::uint64_t length() const noexcept {
return end >= start ? end - start + 1 : 0;
}
bool operator==(const SegmentRecord &) const = default;
};
struct VeloxPart {
std::uint16_t version = kVersion;
std::uint16_t flags = 0;
std::uint64_t total_size = 0;
std::uint64_t downloaded = 0;
std::vector<std::string> urls; // [0]=original, [1]=effective, [2..]=mirrors
std::string etag;
std::string last_modified;
std::string content_type;
std::vector<SegmentRecord> segments;
std::vector<std::byte> sha256_state;
[[nodiscard]] std::string_view original_url() const {
return urls.empty() ? std::string_view{} : std::string_view(urls[0]);
}
[[nodiscard]] std::string_view effective_url() const {
return urls.size() < 2 ? original_url() : std::string_view(urls[1]);
}
bool operator==(const VeloxPart &) const = default;
};
// Parse a sidecar image. Every failure is a Result error, never a throw or a crash:
// meta_corrupt — bad magic, truncation, a count/length past a cap or past
// the buffer, trailing bytes, or a CRC mismatch
// meta_version_unsupported — magic OK, CRC OK, but version > kVersion
[[nodiscard]] Result<VeloxPart> parse_veloxpart(ConstByteSpan image);
// Serialize. Deterministic: the same VeloxPart always produces the same bytes, so an
// unchanged sidecar is not rewritten. The CRC-32 is appended.
[[nodiscard]] std::vector<std::byte> serialize_veloxpart(const VeloxPart &vp);
// File helpers — the sidecar path is `<part file>.veloxpart.meta`.
[[nodiscard]] Result<VeloxPart> read_veloxpart_file(std::string_view path);
// Writes atomically (temp + rename) and, when `fsync`, fdatasync's the file and its
// directory before returning — call at every segment-boundary update (docs/04 §5).
[[nodiscard]] Result<void> write_veloxpart_file(std::string_view path, const VeloxPart &vp,
bool fsync = true);
} // namespace vdm::meta
#endif // VDM_META_VELOXPART_HPP
@@ -0,0 +1,43 @@
// vdm/net/content_disposition.hpp — parse a Content-Disposition header into a filename.
//
// This is a classic mojibake source (AGENT-CORE §3): RFC 6266 `filename`, RFC 5987
// `filename*` ext-values, RFC 2047 encoded-words in the legacy quoted form, and raw
// Latin-1 bytes all show up in the wild. The parser is total — hostile input yields a
// best-effort or empty result, never a throw or a crash — and has its own test table
// (content_disposition_test.cpp) and a fuzz target (tools/fuzz).
//
// This header compiles standalone.
#ifndef VDM_NET_CONTENT_DISPOSITION_HPP
#define VDM_NET_CONTENT_DISPOSITION_HPP
#include <string>
#include <string_view>
namespace vdm::net {
struct ContentDisposition {
enum class Type { none, inline_, attachment, form_data, other };
Type type = Type::none;
// Best-effort UTF-8 filename: path components stripped, control bytes (incl. NUL) and
// edge whitespace removed, or empty when the header carries none. NOT fully sanitized
// for the filesystem — that is rules/ (stage 9). `..`, reserved names, and other
// printable-but-unsafe content may still be present.
std::string filename;
// The filename came from an RFC 5987 `filename*` ext-value (preferred over a plain
// `filename` per RFC 6266 §4.3 when both are present).
bool filename_from_ext = false;
[[nodiscard]] bool is_attachment() const noexcept { return type == Type::attachment; }
[[nodiscard]] bool has_filename() const noexcept { return !filename.empty(); }
};
// Parse the value of a Content-Disposition header (everything after the colon).
[[nodiscard]] ContentDisposition parse_content_disposition(std::string_view header_value);
} // namespace vdm::net
#endif // VDM_NET_CONTENT_DISPOSITION_HPP
+91
View File
@@ -0,0 +1,91 @@
// vdm/net/probe.hpp — "what is at this URL?" without downloading it.
//
// Feeds the File Info dialog (docs/04 §2). HEAD first; a ranged GET `bytes=0-0` follows to
// PROVE resumability (a 206 with a matching Content-Range) rather than trust
// `Accept-Ranges`, which servers lie about (docs/06 R4). The ranged GET is also the
// fallback when HEAD is refused (403/405/501).
//
// Runs on its own small worker pool, sized outside the segment budget (ADR 0011 §5) so a
// burst of probes can't starve transfers and capture.offer's 750 ms path never waits on
// one.
//
// This header compiles standalone.
#ifndef VDM_NET_PROBE_HPP
#define VDM_NET_PROBE_HPP
#include <cstdint>
#include <functional>
#include <memory>
#include <optional>
#include <string>
#include <string_view>
#include <vector>
#include "vdm/net/content_disposition.hpp"
#include "vdm/net/http_types.hpp"
#include "vdm/util/result.hpp"
namespace vdm::net {
struct ProbeRequest {
std::string url;
std::vector<HeaderField> headers; // browser headers, verbatim
std::vector<Cookie> cookies;
std::string user_agent;
std::string referrer;
ProxyConfig proxy;
AuthConfig auth; // credentials for a re-probe after a 401 (leave scheme == none otherwise)
long connect_timeout_ms = 15000;
long overall_timeout_ms = 25000; // download.probe deadline is 30 s
};
struct ProbeResult {
std::string effective_url;
std::vector<std::string> redirect_chain; // requested URL first, effective_url last
long http_status = 0;
std::optional<std::uint64_t> total_size; // full-resource size, if known
std::string mime; // Content-Type value (params kept)
std::string etag;
std::string last_modified;
bool accept_ranges = false; // server advertised Accept-Ranges: bytes
bool resumable = false; // PROVEN: ranged GET -> 206 + matching Content-Range,
// and a validator (ETag or Last-Modified) is present
bool requires_auth = false; // a 401/407 was seen
// Decoded, path-stripped; NOT filesystem-sanitized (rules/ owns that, stage 9).
std::string filename_from_disposition;
std::string filename_from_url;
ContentDisposition::Type disposition_type = ContentDisposition::Type::none;
};
// Resolution order (docs/04 §2.5): explicit user name -> Content-Disposition -> URL path
// segment -> "download.bin". Only a light path/control strip here; rules/ does the
// authoritative sanitize, byte cap, and collision handling.
[[nodiscard]] std::string suggest_filename(const ProbeResult &r,
std::string_view explicit_name = {});
class Prober {
public:
// max_concurrent bounds outstanding probe transfers; the rest queue.
explicit Prober(unsigned max_concurrent = 4);
~Prober();
Prober(const Prober &) = delete;
Prober &operator=(const Prober &) = delete;
// Async. `done` runs on an internal worker thread, exactly once. A 401/407 is a
// SUCCESS result with requires_auth = true (the GUI collects credentials), not an
// error; transport failures and hard HTTP errors (404/410/5xx) are errors.
void probe(ProbeRequest req, std::function<void(Result<ProbeResult>)> done);
private:
struct Impl;
std::unique_ptr<Impl> impl_;
};
} // namespace vdm::net
#endif // VDM_NET_PROBE_HPP
+40
View File
@@ -0,0 +1,40 @@
// vdm/net/url.hpp — a small, total URL splitter.
//
// Not a full RFC 3986 parser (libcurl does the real fetching); just enough to pull a
// filename out of a path and to sanity-check a scheme. Total on hostile input — it has a
// fuzz target (tools/fuzz) — never throws, never asserts.
//
// This header compiles standalone.
#ifndef VDM_NET_URL_HPP
#define VDM_NET_URL_HPP
#include <cstdint>
#include <optional>
#include <string>
#include <string_view>
namespace vdm::net {
struct SplitUrl {
std::string scheme; // lowercased, without "://"
std::string userinfo; // before '@', if any
std::string host; // lowercased; bracketed IPv6 keeps its brackets stripped
std::optional<std::uint16_t> port;
std::string path; // includes the leading '/', or empty
std::string query; // without the '?'
std::string fragment; // without the '#'
bool valid = false; // scheme + host present and scheme is http/https
[[nodiscard]] bool is_http() const noexcept { return scheme == "http" || scheme == "https"; }
};
[[nodiscard]] SplitUrl split_url(std::string_view url);
// The last non-empty path segment, percent-decoded, path components stripped. Empty when
// the path has no usable segment (ends in '/', is empty, or is only "/").
[[nodiscard]] std::string url_filename(std::string_view url);
} // namespace vdm::net
#endif // VDM_NET_URL_HPP
+186
View File
@@ -0,0 +1,186 @@
// vdm/rate/token_bucket.hpp — a lazily-refilled token bucket, and the global -> queue ->
// task limiter hierarchy built on it (docs/04 §6).
//
// A segment worker calls RateLimiter::acquire(task, n) after receiving n body bytes. If
// every applicable level (task, its queue, global) has n tokens, it consumes n from each
// and returns 0. Otherwise it consumes nothing and returns how long to wait before
// retrying — the worker returns CURL_WRITEFUNC_PAUSE and schedules a curl_easy_pause
// resume after that delay (the "precision" layer on top of CURLOPT_MAX_RECV_SPEED_LARGE).
//
// This header compiles standalone.
#ifndef VDM_RATE_TOKEN_BUCKET_HPP
#define VDM_RATE_TOKEN_BUCKET_HPP
#include <algorithm>
#include <chrono>
#include <cstdint>
#include <mutex>
#include <optional>
#include <unordered_map>
#include "vdm/ids.hpp"
namespace vdm::rate {
// rate_bps == 0 means unlimited: consume() always succeeds and never waits.
class TokenBucket {
public:
TokenBucket() = default;
// `burst` caps how many tokens accumulate while idle; 0 => 1 second's worth.
explicit TokenBucket(std::uint64_t rate_bps, std::uint64_t burst = 0) {
set_rate(rate_bps, burst);
}
void set_rate(std::uint64_t rate_bps, std::uint64_t burst = 0) {
std::lock_guard lk(mu_);
const bool was_unlimited = rate_ == 0;
rate_ = rate_bps;
cap_ = burst ? burst : rate_bps; // 1 s of burst by default
// A freshly-limited bucket starts full: you may transfer at burst speed
// immediately, then it throttles (classic token bucket / IDM behaviour). Lowering
// an existing limit only clamps down — it never hands out a fresh burst.
if (rate_bps > 0 && was_unlimited)
tokens_ = cap_;
else if (tokens_ > cap_)
tokens_ = cap_;
last_ = clock::now();
}
[[nodiscard]] std::uint64_t rate() const {
std::lock_guard lk(mu_);
return rate_;
}
// Consume `n` if available; otherwise consume nothing. Returns the wait until `n`
// tokens *would* be available (0 when it consumed).
[[nodiscard]] std::chrono::nanoseconds consume(std::uint64_t n) {
std::lock_guard lk(mu_);
if (rate_ == 0)
return {};
refill_locked();
if (tokens_ >= n) {
tokens_ -= n;
return {};
}
const std::uint64_t deficit = n - tokens_;
// ns to earn `deficit` tokens at rate_ bytes/s
return std::chrono::nanoseconds{
static_cast<std::int64_t>((deficit * 1'000'000'000ull + rate_ - 1) / rate_)};
}
// Two-phase for the hierarchy: check every level, then commit on all or none.
[[nodiscard]] std::chrono::nanoseconds peek(std::uint64_t n) {
std::lock_guard lk(mu_);
if (rate_ == 0)
return {};
refill_locked();
if (tokens_ >= n)
return {};
const std::uint64_t deficit = n - tokens_;
return std::chrono::nanoseconds{
static_cast<std::int64_t>((deficit * 1'000'000'000ull + rate_ - 1) / rate_)};
}
void commit(std::uint64_t n) {
std::lock_guard lk(mu_);
if (rate_ == 0)
return;
tokens_ = tokens_ >= n ? tokens_ - n : 0;
}
private:
using clock = std::chrono::steady_clock;
void refill_locked() {
auto now = clock::now();
auto dt = std::chrono::duration_cast<std::chrono::nanoseconds>(now - last_).count();
if (dt <= 0)
return;
last_ = now;
// added = rate_ * dt / 1e9, guarding overflow for very long idle gaps
long double added =
static_cast<long double>(rate_) * static_cast<long double>(dt) / 1'000'000'000.0L;
std::uint64_t add =
added >= static_cast<long double>(cap_) ? cap_ : static_cast<std::uint64_t>(added);
tokens_ = tokens_ + add > cap_ ? cap_ : tokens_ + add;
}
mutable std::mutex mu_;
std::uint64_t rate_ = 0;
std::uint64_t cap_ = 0;
std::uint64_t tokens_ = 0;
clock::time_point last_ = clock::now();
};
// The hierarchy. All limits default to 0 (unlimited). A task with no queue is limited by
// task + global only.
class RateLimiter {
public:
void set_global_limit(std::uint64_t bps) { global_.set_rate(bps); }
[[nodiscard]] std::uint64_t global_limit() const { return global_.rate(); }
void set_queue_limit(QueueId q, std::uint64_t bps) {
std::lock_guard lk(mu_);
queues_[q].set_rate(bps);
}
void set_task_limit(TaskId t, std::uint64_t bps) {
std::lock_guard lk(mu_);
tasks_[t].set_rate(bps);
}
void attach_task(TaskId t, std::optional<QueueId> q) {
std::lock_guard lk(mu_);
tasks_.try_emplace(t);
if (q) {
task_queue_[t] = *q;
queues_.try_emplace(*q);
} else {
task_queue_.erase(t);
}
}
void detach_task(TaskId t) {
std::lock_guard lk(mu_);
tasks_.erase(t);
task_queue_.erase(t);
}
// Consume `n` bytes against task, queue and global. 0 => consumed everywhere. > 0 =>
// consumed nowhere; wait that long and retry. Held under mu_ for its whole duration
// so a concurrent detach_task() can't invalidate the bucket it is using.
[[nodiscard]] std::chrono::nanoseconds acquire(TaskId t, std::uint64_t n) {
std::lock_guard lk(mu_);
TokenBucket *tb = nullptr;
TokenBucket *qb = nullptr;
if (auto it = tasks_.find(t); it != tasks_.end())
tb = &it->second;
if (auto qit = task_queue_.find(t); qit != task_queue_.end())
if (auto q = queues_.find(qit->second); q != queues_.end())
qb = &q->second;
// peek all, then commit all or none — no level "leaks" tokens on a partial miss.
std::chrono::nanoseconds wait{};
if (tb)
wait = std::max(wait, tb->peek(n));
if (qb)
wait = std::max(wait, qb->peek(n));
wait = std::max(wait, global_.peek(n));
if (wait.count() > 0)
return wait;
if (tb)
tb->commit(n);
if (qb)
qb->commit(n);
global_.commit(n);
return {};
}
private:
mutable std::mutex mu_; // guards the maps AND serialises acquire()
TokenBucket global_;
std::unordered_map<QueueId, TokenBucket> queues_;
std::unordered_map<TaskId, TokenBucket> tasks_;
std::unordered_map<TaskId, QueueId> task_queue_;
};
} // namespace vdm::rate
#endif // VDM_RATE_TOKEN_BUCKET_HPP
View File
+47
View File
@@ -0,0 +1,47 @@
// vdm/rules/collision.hpp — when a chosen filename is already taken in the destination
// directory, decide what to try next.
//
// Pure: takes an existence predicate rather than touching a filesystem itself, so it never
// races what it's deciding about and stays testable without one. DAEMON (which owns the
// actual directory listing / stat calls, downstream of its own fs/safepath gate) supplies
// that predicate; a test supplies an in-memory set.
//
// This header compiles standalone.
#ifndef VDM_RULES_COLLISION_HPP
#define VDM_RULES_COLLISION_HPP
#include <functional>
#include <string>
#include <string_view>
namespace vdm::rules {
enum class CollisionPolicy {
rename, // try "name (1).ext", "name (2).ext", ... until one is free
overwrite, // return `desired` unchanged — caller intends to replace what's there
};
// Under `CollisionPolicy::rename`: calls `exists(candidate)` first with `desired` itself,
// then with "<stem> (1)<ext>", "<stem> (2)<ext>", ... (Explorer/Finder-style, splitting
// `desired` on its last '.' the same way `sanitize_filename`'s truncation does), returning
// the first candidate for which it returns false. `exists` is never called with anything
// but a single leaf name, never a path.
//
// `max_attempts` bounds a pathological `exists` that always returns true (this function
// always returns — it is not fallible): once reached, the last candidate tried is returned
// as-is, still possibly colliding. That is deliberately not papered over with a
// fabricated-unique name (a timestamp suffix, say) — silently handing back a name nobody
// asked for is exactly the kind of thing that turns into a mystery file days later; a
// caller that hits the bound should treat it as a real error, not swallow it here.
//
// Under `CollisionPolicy::overwrite`, `exists` and `max_attempts` are unused — `desired`
// is returned unchanged.
[[nodiscard]] std::string resolve_collision(std::string_view desired,
const std::function<bool(std::string_view)> &exists,
CollisionPolicy policy = CollisionPolicy::rename,
int max_attempts = 1000);
} // namespace vdm::rules
#endif // VDM_RULES_COLLISION_HPP
+49
View File
@@ -0,0 +1,49 @@
// vdm/rules/filename.hpp — turn a raw, untrusted filename candidate into one safe to
// create on a real filesystem, cross-platform.
//
// This is NOT the path-traversal security boundary — that's DAEMON's fs/safepath (the
// process's one canonicalize-and-verify-against-allowed-roots gate; see its own header
// comment). This runs earlier and is cooperative, not adversarial-proof on its own: turn
// whatever `net::parse_content_disposition` or `net::url_filename` handed back (see their
// headers — neither fully sanitizes for the filesystem, by design; this is where that
// finishes) into a *reasonable* candidate so an ordinary download doesn't needlessly
// collide with a reserved device name, get silently mangled by NTFS, or get truncated
// mid-extension by safepath's own leaf check.
//
// Total on hostile input: never throws, never asserts, never returns empty (falls back to
// a generic name). No JSON, no SQL, no Qt, no RPC, no filesystem access (CLAUDE.md §3) —
// pure string transformation.
//
// This header compiles standalone.
#ifndef VDM_RULES_FILENAME_HPP
#define VDM_RULES_FILENAME_HPP
#include <cstddef>
#include <string>
#include <string_view>
namespace vdm::rules {
// Sanitizes `raw` into a single path component safe to create on ext4, APFS, and NTFS
// alike:
// - strips path separators ('/' always; '\' too — Windows treats it as one) and collapses
// any run of '.' that would otherwise still read as a traversal attempt ("..", "...")
// down to a single '.', so a stripped-separator name can't reconstitute one
// - strips C0 control bytes (incl. NUL) and DEL (0x7F)
// - replaces the other NTFS-illegal characters (`< > : " | ? *`) with '_', so a name
// that's fine on Linux doesn't silently fail to sync/export to a Windows-formatted
// drive or SMB share
// - strips trailing '.' and ' ' (both are NTFS traps: silently dropped by the Win32 API,
// so "name." and "name" would otherwise collide invisibly on export)
// - a reserved Windows device name (CON, PRN, AUX, NUL, COM19, LPT19), matched
// case-insensitively against the part before the first '.' (or the whole name if
// there's no '.'), gets a trailing '_' so it stops shadowing a device
// - clamps to `max_bytes` (default 255, the common ext4/APFS/NTFS component limit),
// cutting on a UTF-8 boundary and preferring to keep a short trailing extension intact
// - empty, or entirely stripped down to nothing, falls back to "download"
[[nodiscard]] std::string sanitize_filename(std::string_view raw, std::size_t max_bytes = 255);
} // namespace vdm::rules
#endif // VDM_RULES_FILENAME_HPP
+91
View File
@@ -0,0 +1,91 @@
// vdm/rules/match.hpp — the pure evaluation half of the rules engine
// (contracts/schema/types/Rule.schema.json): given a rule table and what's known about one
// candidate download, decide which rule fires and what it says to do.
//
// DAEMON owns the rule table itself — storage, `rules.upsert`/`rules.list`, the wire
// `Rule` type generated from the contract. It decodes its own JSON/SQL representation into
// the plain structs below and calls in; core never sees JSON, SQL, or the generated
// protocol types (CLAUDE.md §3) — these structs mirror the contract's shape in CORE's own
// vocabulary, the same relationship `util/error.hpp`'s `Error` has to the wire error codes.
//
// Pure and total: no I/O, no throw, no crash on any input (an empty table, an empty
// pattern, a rule with every match clause absent).
//
// This header compiles standalone.
#ifndef VDM_RULES_MATCH_HPP
#define VDM_RULES_MATCH_HPP
#include <cstdint>
#include <optional>
#include <string>
#include <string_view>
#include <vector>
namespace vdm::rules {
enum class StartMode { now, later, queue }; // mirrors contracts' StartMode
enum class CaptureVerdict { take, ignore }; // mirrors RuleAction.capture
// All present clauses must match; an absent clause is not a constraint
// (contracts/schema/types/Rule.schema.json's own wording for RuleMatch).
struct RuleMatch {
std::optional<std::vector<std::string>> extensions; // no leading '.'; matched
// case-insensitively
std::optional<std::vector<std::string>> mime_types; // matched case-insensitively
std::optional<std::string> host_pattern; // glob_match against the host
std::optional<std::string> url_pattern; // glob_match against the whole URL
std::optional<std::uint64_t> min_size_bytes; // inclusive
std::optional<std::uint64_t> max_size_bytes; // inclusive
};
// What to do with a matching download.
struct RuleAction {
std::optional<std::string> category_id;
std::optional<std::string> save_dir;
std::optional<std::string> queue_id;
std::optional<std::uint32_t> segments;
std::optional<StartMode> start_mode;
std::optional<CaptureVerdict> capture;
};
// One row of the rules engine.
struct Rule {
std::string rule_id;
bool enabled = true;
std::int64_t priority = 0; // lower runs first
RuleMatch match;
RuleAction action;
};
// What's known about one candidate download, to match rules against. A field being empty
// (not `std::nullopt` — these are plain strings, not optionals) means "unknown, matches no
// non-empty clause that needs it" — e.g. `size_bytes` is absent pre-probe, so any rule with
// a size clause simply doesn't match yet; the caller is expected to re-run matching once
// the probe fills it in, same as DownloadSpec itself gets refined post-probe.
struct MatchInput {
std::string extension; // lowercased, no leading '.'; empty if none
std::string mime_type; // lowercased; empty if unknown
std::string host; // lowercased effective-URL host
std::string url; // the whole effective URL
std::optional<std::uint64_t> size_bytes;
};
// Rules are tried in ascending `priority` order (ties broken by table order), skipping
// disabled rows; the first whose every present match clause is satisfied wins.
// `std::nullopt` means no rule matched — the caller falls back to its own default category
// (this function has no notion of a default; that policy lives with the caller).
[[nodiscard]] std::optional<RuleAction> match_rules(const std::vector<Rule> &rules,
const MatchInput &input);
// Case-insensitive glob: '*' matches any run of characters including none, '?' matches
// exactly one character. No character classes, no escaping — rule patterns are meant to
// stay simple (contracts/schema/types/Rule.schema.json's own hostPattern/urlPattern
// description gives only "*.example.com" as the example). Exposed on its own because
// hostPattern and urlPattern are both just this against different text, and because it has
// its own test table worth keeping separate from match_rules's.
[[nodiscard]] bool glob_match(std::string_view pattern, std::string_view text) noexcept;
} // namespace vdm::rules
#endif // VDM_RULES_MATCH_HPP
+146
View File
@@ -0,0 +1,146 @@
// vdm/segment/budget.hpp — the global segment allocator (ADR 0011).
//
// One instance per engine. It owns exactly one ceiling — `maxActiveSegments`, in segment
// units — and the min-1-before-seconds fairness rule (ADR 0011 §3). DAEMON's scheduler
// counts tasks and never touches this except through the read-outs and setters below;
// CORE's download_task (stage 8) drives the task-facing half.
//
// Fairness (two-pass, recomputed on every edge): a guarantee pass gives every task that
// wants a slot and holds zero exactly one, in DAEMON's priority order; then a growth pass
// round-robins the remainder up to each task's effective cap. A task's target can drop
// below what it holds (a lower-priority task shedding for a higher-priority arrival, or a
// live `set_max_active_segments` cut) — the task then *yields*: it releases a slot at its
// next segment boundary, never mid-segment (ADR 0011 A1). A finishing worker whose target
// still covers it *steals* instead (slot-neutral). That steal-vs-yield choice lives in
// stage 8, driven by comparing this budget's target to the task's live worker count.
//
// This header compiles standalone.
#ifndef VDM_SEGMENT_BUDGET_HPP
#define VDM_SEGMENT_BUDGET_HPP
#include <chrono>
#include <condition_variable>
#include <cstdint>
#include <functional>
#include <mutex>
#include <optional>
#include <span>
#include <string>
#include <thread>
#include <unordered_map>
#include <vector>
#include "vdm/ids.hpp"
namespace vdm::segment {
class SegmentBudget {
public:
struct EngineBudget {
std::uint32_t total = 0; // == maxActiveSegments
std::uint32_t active = 0; // slots held across all tasks
std::uint32_t tasks_starved = 0; // running tasks holding zero slots (ADR 0011 §3.6)
bool operator==(const EngineBudget &) const = default;
};
struct Options {
std::uint32_t max_active_segments = 32;
std::chrono::milliseconds notify_period{250}; // <=4 Hz, per event.task.progress
};
SegmentBudget(); // default Options
explicit SegmentBudget(Options opts);
~SegmentBudget();
SegmentBudget(const SegmentBudget &) = delete;
SegmentBudget &operator=(const SegmentBudget &) = delete;
// ---- task-facing (download_task, stage 8) -------------------------------------------
struct TaskParams {
std::string host; // key for the per-host segment cap
std::uint32_t per_task_cap = 1; // min(spec.segments ?? maxSegmentsPerDownload, 32)
bool resumable = false; // false => effective cap forced to 1
};
// Called with the new absolute slot target for the task. Runs on a budget thread (or
// the caller's, for the edge case) — must not block and must not re-enter the budget
// beyond confirm_slot()/release_slot(). The task starts or yields workers to match.
using SlotTargetFn = std::function<void(std::uint32_t target)>;
void register_task(TaskId id, const TaskParams &params, SlotTargetFn on_target);
void deregister_task(TaskId id); // pause / complete / cancel
// How many slots the task could use right now (0 .. per_task_cap): incomplete
// segments it has range for. 0 while retry_wait / assembling / verifying / paused —
// which is exactly why those states are never counted as starvation.
void set_want(TaskId id, std::uint32_t want);
// A worker actually started on a granted slot. false => the target was cut in the
// race and the worker must not start.
[[nodiscard]] bool confirm_slot(TaskId id);
// A held slot is free: segment complete with no steal, failed, paused, or yielded.
void release_slot(TaskId id);
// ---- DAEMON-facing (sched/) -------------------------------------------------------
void set_max_active_segments(std::uint32_t n); // drain-not-kill (ADR 0011 §2)
void set_host_segment_cap(std::string host, std::uint32_t cap); // 0 clears
void set_task_order(std::span<const TaskId> priority_order); // pushed on change
[[nodiscard]] EngineBudget budget() const;
[[nodiscard]] std::uint32_t segments_active(TaskId id) const;
[[nodiscard]] std::vector<TaskId> starved_tasks() const;
[[nodiscard]] std::optional<SteadyTime> starved_since(TaskId id) const;
void on_budget_changed(std::function<void(EngineBudget)> cb);
private:
struct Task {
std::string host;
std::uint32_t per_task_cap = 1;
bool resumable = false;
std::uint32_t want = 0;
std::uint32_t held = 0;
std::uint32_t target = 0; // last published
SlotTargetFn on_target;
std::optional<SteadyTime> starved_since;
// confirm_slot() was denied by the engine-wide cap (not by this task's own
// target) and hasn't been retried since. Cleared on the task's next successful
// confirm_slot(), however that retry was triggered. See release_slot()'s comment.
bool waiting_for_slot = false;
};
// A unit of deferred work: callbacks are copied out here so the public entry points
// can invoke them AFTER dropping mu_.
struct Plan {
std::vector<std::pair<SlotTargetFn, std::uint32_t>> targets;
std::optional<std::pair<std::function<void(EngineBudget)>, EngineBudget>> notify_now;
};
[[nodiscard]] std::vector<TaskId> priority_order_locked() const;
Plan reallocate_locked();
// Appends a retry hint for the highest-priority task with waiting_for_slot set (other
// than `exclude`, the task whose release just freed this slot -- see release_slot()'s
// comment) to `plan`, if one exists.
void wake_one_waiter_locked(Plan &plan, TaskId exclude) const;
static void run(Plan &p);
[[nodiscard]] std::uint32_t effective_cap_locked(const Task &t) const;
[[nodiscard]] EngineBudget snapshot_locked() const;
void notifier_loop(std::stop_token st);
mutable std::mutex mu_;
std::unordered_map<TaskId, Task> tasks_;
std::vector<TaskId> order_;
std::unordered_map<std::string, std::uint32_t> host_caps_;
std::uint32_t max_active_;
std::uint32_t active_ = 0;
std::function<void(EngineBudget)> on_changed_;
EngineBudget last_notified_;
std::uint32_t last_starved_ = 0;
bool dirty_ = false;
std::chrono::milliseconds notify_period_;
std::condition_variable notify_cv_;
std::jthread notifier_;
};
} // namespace vdm::segment
#endif // VDM_SEGMENT_BUDGET_HPP
+178
View File
@@ -0,0 +1,178 @@
// vdm/segment/segmenter.hpp — per-download range management and dynamic segment stealing.
//
// docs/04 §3. Owns the split of [0, total_size) for one download: the initial layout, a
// split when a slot is granted (`assign_slot`), a *steal* when a worker finishes and may
// keep its slot (`on_complete` — take the second half of the largest remaining range),
// and a re-split of an orphaned range when a segment fails 3x on the same host with a
// mirror available (`on_failed` -> requeue).
//
// Thread model: one mutex — the segmenter's — *is* "the task lock" (docs/04 §3: the steal
// is "atomic under the task lock"). Every method takes it. `advance()` and the per-worker
// accessors are called once per buffer flush (a few Hz per segment), not from the curl
// write callback, so a lock there is free; the no-lock/no-alloc rule is about that
// callback and its ring buffer. Segment fields stay std::atomic so the store type is
// trivially relocatable and reads never tear. The store is a std::deque so a steal's
// push_back never moves an existing record.
//
// This header compiles standalone.
#ifndef VDM_SEGMENT_SEGMENTER_HPP
#define VDM_SEGMENT_SEGMENTER_HPP
#include <atomic>
#include <cstdint>
#include <deque>
#include <mutex>
#include <optional>
#include <vector>
namespace vdm::segment {
inline constexpr std::uint64_t kDefaultMinSegmentBytes = 1u << 20; // 1 MiB (docs/04 §3)
inline constexpr std::uint32_t kDefaultSegments = 8;
inline constexpr std::uint32_t kMaxSegments = 32;
enum class SegState : std::uint8_t {
idle, // range assigned, no worker connected yet
connecting,
downloading,
stalled, // low-speed; still holds its slot
complete,
failed, // gave up (orphaned; range requeued or lost to a steal)
};
// A flat, copyable view of one segment record. Ranges are absolute byte offsets,
// **inclusive** on both ends (contract Segment.endByte / ADR 0010).
struct SegmentView {
std::uint32_t index = 0;
std::uint64_t start = 0;
std::uint64_t end = 0;
std::uint64_t completed = 0;
SegState state = SegState::idle;
std::uint32_t consecutive_failures = 0;
[[nodiscard]] std::uint64_t length() const noexcept {
return end >= start ? end - start + 1 : 0;
}
[[nodiscard]] std::uint64_t remaining() const noexcept {
return length() - (completed < length() ? completed : length());
}
bool operator==(const SegmentView &) const = default;
};
// One resumed range, as read back from .veloxpart.meta (kept independent of meta/ so this
// header stands alone).
struct ResumedRange {
std::uint64_t start = 0;
std::uint64_t end = 0;
std::uint64_t completed = 0;
};
enum class FailAction {
retry, // same range, backoff (owned by the task/state machine)
requeue, // 3x connection failure + a mirror exists: orphan the remaining range and
// re-split it; the segment's slot is released
};
class Segmenter {
public:
// total_size 0 => unknown (chunked): forces a single segment. resumable == false also
// forces a single segment (docs/04 §3: "Non-resumable servers -> exactly 1 segment").
Segmenter(std::uint64_t total_size, std::uint32_t requested_segments, bool resumable,
std::uint64_t min_segment_bytes = kDefaultMinSegmentBytes);
// Resume: rebuild from a persisted segment table. Ranges must tile [0, total_size)
// with no gap or overlap; a malformed table falls back to a single segment.
Segmenter(std::uint64_t total_size, std::uint32_t requested_segments,
const std::vector<ResumedRange> &resumed, bool resumable,
std::uint64_t min_segment_bytes = kDefaultMinSegmentBytes);
Segmenter(const Segmenter &) = delete;
Segmenter &operator=(const Segmenter &) = delete;
// The count this download would use with an unlimited budget. 1 when non-resumable /
// unknown size; otherwise min(requested, floor(total / min_segment_bytes), 32).
[[nodiscard]] std::uint32_t target_segment_count() const noexcept { return target_count_; }
[[nodiscard]] std::uint64_t total_size() const noexcept { return total_size_; }
// Give a worker something to download. Called when the budget grants a slot. Prefers
// an orphaned range from a requeue; otherwise splits the largest remaining range in
// half and hands back the tail. Returns nullopt when the target count is already met
// or nothing splits to >= min_segment_bytes. `state` of the returned segment is
// `connecting`.
[[nodiscard]] std::optional<std::uint32_t> assign_slot();
// A worker finished its range. If `may_steal` is false the slot is being yielded —
// returns nullopt and the caller releases the slot to the budget. If true and a
// remaining range splits to >= min_segment_bytes, steals its second half: returns a
// NEW segment index for the same worker to continue on (slot-neutral). Otherwise
// nullopt (nothing to steal -> release).
[[nodiscard]] std::optional<std::uint32_t> on_complete(std::uint32_t idx, bool may_steal);
// A worker's segment errored. `connection_error` distinguishes a transport failure
// (reset/timeout/refused) from an HTTP/content one. Returns requeue only on the 3rd
// consecutive connection error when `has_mirror`; on requeue the remaining range is
// orphaned for assign_slot() to re-split and the segment is marked failed.
FailAction on_failed(std::uint32_t idx, bool connection_error, bool has_mirror);
// A successful (re)connection resets the consecutive-failure counter.
void note_connected(std::uint32_t idx);
// Progress from the write path. Lock-free. `bytes` is the absolute completed count
// within the segment; clamped to the segment length.
void advance(std::uint32_t idx, std::uint64_t bytes) noexcept;
// Per-segment fields for a worker. A worker reads `segment_end` before every write so
// a concurrent steal that shrank its range stops it cleanly. `segment_start` never
// changes for a given index.
[[nodiscard]] std::uint64_t segment_start(std::uint32_t idx) const noexcept;
[[nodiscard]] std::uint64_t segment_end(std::uint32_t idx) const noexcept;
[[nodiscard]] std::uint64_t segment_completed(std::uint32_t idx) const noexcept;
[[nodiscard]] SegState segment_state(std::uint32_t idx) const noexcept;
void set_segment_state(std::uint32_t idx, SegState s) noexcept;
// Hand a segment back to the pool without touching its `completed`: it becomes an
// unassigned idle range that the next assign_slot() picks up (used on pause).
void release_segment(std::uint32_t idx) noexcept;
// Sum of bytes done across every segment (active + already complete). Locks.
[[nodiscard]] std::uint64_t downloaded() const;
[[nodiscard]] bool all_complete() const;
// Every segment record, for TaskDetail projection and the .veloxpart.meta writer.
[[nodiscard]] std::vector<SegmentView> snapshot() const;
private:
struct Seg {
std::uint32_t index;
std::uint64_t start;
std::atomic<std::uint64_t> end;
std::atomic<std::uint64_t> completed{0};
std::atomic<SegState> state{SegState::idle};
std::uint32_t consecutive_failures = 0;
bool assigned = false; // a worker holds this segment right now
Seg(std::uint32_t i, std::uint64_t s, std::uint64_t e) : index(i), start(s), end(e) {}
};
void compute_target(std::uint32_t requested);
std::uint32_t add_seg_locked(std::uint64_t start, std::uint64_t end, std::uint64_t completed,
SegState state, bool assigned);
std::uint32_t split_largest_remaining_locked(); // returns new index, or UINT32_MAX
[[nodiscard]] std::uint64_t remaining_of_locked(const Seg &s) const noexcept;
[[nodiscard]] std::uint32_t assigned_count_locked() const noexcept;
mutable std::mutex mu_;
std::deque<Seg> segs_;
std::vector<ResumedRange> orphans_; // requeued ranges awaiting re-split
std::uint64_t total_size_;
std::uint64_t min_seg_;
bool resumable_;
std::uint32_t target_count_ = 1;
std::uint32_t next_index_ = 0;
};
} // namespace vdm::segment
#endif // VDM_SEGMENT_SEGMENTER_HPP
+223
View File
@@ -0,0 +1,223 @@
// vdm/task/download.hpp — the public download API: what DAEMON hands the engine and how
// the engine reports back. REVIEW SKETCH (stage 7 pre-work) — value types are final
// enough to build against; Engine/DownloadHandle bodies land in stage 8.
//
// See core/docs/engine-api-m1.md for the threading, lifetime, and pause/resume/cancel
// contract that goes with these signatures.
//
// This header compiles standalone.
#ifndef VDM_TASK_DOWNLOAD_HPP
#define VDM_TASK_DOWNLOAD_HPP
#include <chrono>
#include <cstdint>
#include <functional>
#include <memory>
#include <optional>
#include <string>
#include <vector>
#include "vdm/ids.hpp"
#include "vdm/net/http_types.hpp"
#include "vdm/net/probe.hpp"
#include "vdm/segment/segmenter.hpp"
#include "vdm/util/error.hpp"
#include "vdm/util/result.hpp"
namespace vdm {
class Engine; // owns and fills DownloadHandle (see vdm/engine.hpp)
} // namespace vdm
namespace vdm::task {
// The task control block. Opaque: defined only in the engine's translation unit. A handle
// holds a shared_ptr to one; the engine keeps its own copy so the task outlives a caller
// that drops its handle.
struct DownloadTaskState;
// --- input ---------------------------------------------------------------------------
struct Checksum {
enum class Algo { md5, sha1, sha256, sha512 }; // matches the wire Checksum set
Algo algo = Algo::sha256;
std::string hex; // lower-case, no separators
};
// Everything the engine needs to run ONE download. DAEMON has already run the rules
// engine, canonicalised the path, checked it against the allowed roots, resolved the
// filename, and created the parent directory — `save_path` is absolute and final and its
// directory exists. `<save_path>.veloxpart` and `<save_path>.veloxpart.meta` live beside
// it during the transfer; on success the part file is renamed in place. If the directory
// is missing at open time the task fails with Error::path_rejected.
struct DownloadSpec {
std::string url;
std::vector<std::string> mirrors; // alternative URLs for the same bytes
std::vector<net::HeaderField> headers; // the browser's, verbatim
std::vector<net::Cookie> cookies;
std::string referrer;
std::string user_agent;
std::string save_path; // absolute; the engine never canonicalises or root-checks
std::optional<std::uint32_t> segments; // requested 1..32; nullopt => engine default
std::optional<std::uint64_t> buffer_bytes; // requested per segment; nullopt => default
net::ProxyConfig proxy;
net::AuthConfig auth; // credentials known up front (e.g. from the Secret Service);
// leave scheme == none to be prompted on a 401/407
std::optional<Checksum> checksum; // verified during `verifying`; mismatch => failed
// DAEMON usually probed already for the File Info dialog. Pass it to skip a second
// probe; the engine still revalidates on resume. nullopt => the engine probes.
std::optional<net::ProbeResult> probe_hint;
bool allow_resume = true; // if a valid .veloxpart.meta sits beside save_path, resume
// from it; false starts fresh and overwrites
std::optional<long> max_retries; // per-segment; nullopt => engine default (10)
};
// --- lifecycle (the CORE-owned subset of the wire TaskState; ADR 0013 §1) -------------
enum class EngineState {
probing,
connecting,
downloading,
paused, // shared with DAEMON; entered by either side, idempotently
retry_wait, // the engine's own backoff timer
assembling, // no-op rename in M1; a real mux step for HLS/DASH (M4)
verifying, // checksum
complete, // terminal
failed, // terminal
cancelled, // terminal; always DAEMON- or user-initiated
};
[[nodiscard]] constexpr bool is_terminal(EngineState s) noexcept {
return s == EngineState::complete || s == EngineState::failed || s == EngineState::cancelled;
}
// --- progress ---------------------------------------------------------------------------
struct SegmentProgress {
std::uint32_t index = 0;
std::uint64_t start = 0;
std::uint64_t end = 0; // inclusive
std::uint64_t completed = 0;
std::uint64_t speed_bps = 0;
segment::SegState state = segment::SegState::idle;
};
struct Progress {
std::uint64_t downloaded = 0;
std::optional<std::uint64_t> total; // absent for a chunked source until it ends
std::uint64_t speed_bps = 0; // aggregate over the last window
std::optional<std::uint32_t> eta_seconds;
std::uint32_t effective_segments = 0; // slots the budget granted (held)
std::uint64_t effective_buffer_bytes = 0; // per segment, after the maxTotal clamp
std::vector<SegmentProgress> segments;
};
// --- interaction callbacks ----------------------------------------------------------
// A 401/407. The task has already auto-paused (state -> paused, error == auth_required).
// DAEMON collects credentials and calls handle.provide_auth().
struct AuthChallenge {
std::string host;
std::string realm;
enum class Scheme { basic, digest, ntlm, negotiate, unknown };
Scheme scheme = Scheme::unknown;
};
// The server's copy changed under us (a 200 where a 206 was expected, or an If-Range /
// ETag mismatch on resume — docs/04 §5), or the range metadata went stale (416). The
// task has auto-paused. DAEMON asks the user and calls handle.decide().
struct DecisionRequest {
enum class Kind { server_file_changed, range_metadata_stale };
Kind kind = Kind::server_file_changed;
std::string detail; // human-readable, for the dialog body
};
enum class Decision {
restart, // discard the partial file, download again from scratch
keep_partial, // trust what is on disk and continue (the user's risk)
abort, // give up: the task goes to `failed`
};
struct DownloadOutcome {
std::string final_path;
std::uint64_t bytes = 0;
std::optional<std::string> sha256_hex; // present when a checksum was requested/derived
std::chrono::milliseconds elapsed{0};
};
// All callbacks are optional. See core/docs/engine-api-m1.md for the rules; in short:
// they arrive on an engine thread, are serialised per task, must not block, and must not
// re-enter THIS task's handle synchronously.
struct DownloadCallbacks {
// Coalesced to <= 4 Hz per task (matches the wire event.task.progress cadence).
std::function<void(const Progress &)> on_progress;
// Every lifecycle transition, including the auto-pauses above (to == paused with a
// populated ErrorInfo) and terminals.
std::function<void(EngineState from, EngineState to, const std::optional<ErrorInfo> &)>
on_state;
std::function<void(const AuthChallenge &)> on_auth_required;
std::function<void(const DecisionRequest &)> on_decision_needed;
// Fired exactly once, last. Success carries the outcome; failure carries the mapped
// ErrorInfo. After it returns the engine makes no further callbacks for this task and
// the handle's control methods become no-ops.
std::function<void(Result<DownloadOutcome>)> on_finished;
};
// --- the handle -------------------------------------------------------------------------
// Copyable (shared state). Every method is safe to call from any thread; each posts to
// the engine and returns immediately. Dropping the last handle does NOT cancel the task —
// call cancel() for that. Bodies land in stage 8.
class DownloadHandle {
public:
DownloadHandle() = default;
// The engine builds handles; `DownloadTaskState` is incomplete everywhere else, so
// this is effectively engine-only without a friend declaration.
explicit DownloadHandle(std::shared_ptr<DownloadTaskState> s) : state_(std::move(s)) {}
[[nodiscard]] TaskId id() const noexcept;
[[nodiscard]] bool valid() const noexcept { return static_cast<bool>(state_); }
// Idempotent. pause() on an already-paused or terminal task is a no-op (no error);
// likewise resume() on a task that is not paused. The resulting state is observed via
// on_state / this->state(), never a return value (ADR 0013 §2).
void pause();
void resume();
// Idempotent, terminal. discard_partial also removes the .veloxpart[.meta] files.
// Always fires on_state(_, cancelled, nullopt) then on_finished(Err{Error::canceled}),
// in that order. `download.cancel` == cancel(false); `download.remove` == cancel(true)
// (plus DAEMON's own row/file cleanup).
void cancel(bool discard_partial = false);
// Only act while the task is awaiting the matching input (auto-paused for auth /
// decision); otherwise a no-op. `remember` asks DAEMON to persist to the Secret
// Service — the engine never stores a credential.
void provide_auth(std::string username, std::string password, bool remember);
void decide(Decision d);
// IDM's "Refresh Download Address": swap the URL (e.g. a fresh signed URL) on a live
// or paused task without losing progress. Empty `headers` keeps the current ones.
void refresh_url(std::string url, std::vector<net::HeaderField> headers = {});
// Synchronous snapshots — cheap, lock-guarded, safe any time.
[[nodiscard]] EngineState state() const;
[[nodiscard]] Progress progress() const;
private:
std::shared_ptr<DownloadTaskState> state_;
};
} // namespace vdm::task
#endif // VDM_TASK_DOWNLOAD_HPP
+48
View File
@@ -0,0 +1,48 @@
// vdm/util/crc32.hpp — CRC-32 (IEEE 802.3 / zlib polynomial), header-only.
//
// Used to integrity-check the .veloxpart.meta resume sidecar (docs/04 §5). Standard
// reflected CRC-32 with 0xEDB88320, init/xorout 0xFFFFFFFF — byte-compatible with
// zlib's crc32() and `cksum -o3` — so the value is reproducible outside this codebase.
//
// This header compiles standalone.
#ifndef VDM_UTIL_CRC32_HPP
#define VDM_UTIL_CRC32_HPP
#include <array>
#include <cstddef>
#include <cstdint>
#include "vdm/util/bytes.hpp"
namespace vdm {
namespace detail {
inline constexpr std::array<std::uint32_t, 256> make_crc32_table() {
std::array<std::uint32_t, 256> t{};
for (std::uint32_t i = 0; i < 256; ++i) {
std::uint32_t c = i;
for (int k = 0; k < 8; ++k)
c = (c & 1u) ? (0xEDB88320u ^ (c >> 1)) : (c >> 1);
t[i] = c;
}
return t;
}
inline constexpr std::array<std::uint32_t, 256> kCrc32Table = make_crc32_table();
} // namespace detail
// Incremental: pass the previous result back as `seed` to continue over split buffers.
[[nodiscard]] inline std::uint32_t crc32_update(std::uint32_t seed, ConstByteSpan data) noexcept {
std::uint32_t c = seed ^ 0xFFFFFFFFu;
for (std::byte b : data)
c = detail::kCrc32Table[(c ^ std::to_integer<std::uint8_t>(b)) & 0xFFu] ^ (c >> 8);
return c ^ 0xFFFFFFFFu;
}
[[nodiscard]] inline std::uint32_t crc32(ConstByteSpan data) noexcept {
return crc32_update(0u, data);
}
} // namespace vdm
#endif // VDM_UTIL_CRC32_HPP
+169
View File
@@ -0,0 +1,169 @@
// vdm/engine.cpp
#include "vdm/engine.hpp"
#include <atomic>
#include <condition_variable>
#include <cstdint>
#include <map>
#include <mutex>
#include <thread>
#include <unordered_map>
#include "task/download_task.hpp"
namespace vdm {
struct Engine::Impl : task::TaskHost {
explicit Impl(Config c)
: cfg_(c),
http_(net::HttpClient::Options{.workers = c.http_workers}),
prober_(c.probe_pool_size ? c.probe_pool_size : 4),
budget_(segment::SegmentBudget::Options{.max_active_segments = c.max_active_segments}) {
timer_ = std::jthread([this](std::stop_token st) { timer_loop(st); });
}
~Impl() override {
// Quiesce every task first so no callback fires during or after teardown: mark it
// retired and cancel its transfers. Then stop the timer thread (a fn already
// running holds a shared_ptr and finishes, but seg_finished early-returns on
// `retired`). Then drop the registry; http_/prober_/budget_ destruct after.
{
std::lock_guard lk(reg_mu_);
for (auto &[id, t] : tasks_)
task::quiesce_task(t);
}
timer_.request_stop();
timer_cv_.notify_all();
if (timer_.joinable())
timer_.join();
{
std::lock_guard lk(reg_mu_);
tasks_.clear();
}
}
// --- TaskHost -------------------------------------------------------------------
net::HttpClient &http() override { return http_; }
segment::SegmentBudget &budget() override { return budget_; }
rate::RateLimiter &limiter() override { return limiter_; }
const Config &config() override { return cfg_; }
task::TimerId schedule(SteadyTime at, std::function<void()> fn) override {
task::TimerId id;
{
std::lock_guard lk(timer_mu_);
id = ++next_timer_;
timers_.emplace(at, Entry{id, std::move(fn)});
}
timer_cv_.notify_all();
return id;
}
void cancel_timer(task::TimerId id) override {
std::lock_guard lk(timer_mu_);
for (auto it = timers_.begin(); it != timers_.end(); ++it)
if (it->second.id == id) {
timers_.erase(it);
return;
}
}
void probe(net::ProbeRequest req, std::function<void(Result<net::ProbeResult>)> done) override {
prober_.probe(std::move(req), std::move(done));
}
void task_retired(TaskId id) override {
std::lock_guard lk(reg_mu_);
tasks_.erase(id);
}
// --- engine surface ----------------------------------------------------------------
task::DownloadHandle start(task::DownloadSpec spec, task::DownloadCallbacks cbs) {
TaskId id{++next_id_};
auto st = task::create_task(*this, id, std::move(spec), std::move(cbs));
{
std::lock_guard lk(reg_mu_);
tasks_[id] = st;
}
return task::DownloadHandle(std::move(st));
}
Config cfg_;
net::HttpClient http_;
net::Prober prober_;
segment::SegmentBudget budget_;
rate::RateLimiter limiter_;
std::atomic<std::uint64_t> next_id_{0};
std::mutex reg_mu_;
std::unordered_map<TaskId, std::shared_ptr<task::DownloadTaskState>> tasks_;
struct Entry {
task::TimerId id;
std::function<void()> fn;
};
std::mutex timer_mu_;
std::condition_variable timer_cv_;
std::multimap<SteadyTime, Entry> timers_;
std::atomic<std::uint64_t> next_timer_{0};
std::jthread timer_;
void timer_loop(std::stop_token st) {
std::unique_lock lk(timer_mu_);
while (!st.stop_requested()) {
if (timers_.empty()) {
timer_cv_.wait_for(lk, std::chrono::seconds(1));
continue;
}
auto next_at = timers_.begin()->first;
if (timer_cv_.wait_until(lk, next_at, [&] {
return st.stop_requested() ||
(!timers_.empty() && timers_.begin()->first < next_at);
})) {
continue; // stop, or an earlier timer landed — re-evaluate
}
// fire everything due
std::vector<std::function<void()>> due;
auto now = std::chrono::steady_clock::now();
for (auto it = timers_.begin(); it != timers_.end() && it->first <= now;)
due.push_back(std::move(it->second.fn)), it = timers_.erase(it);
lk.unlock();
for (auto &fn : due)
fn();
lk.lock();
}
}
};
// --- Engine ---------------------------------------------------------------------------
Engine::Engine() : Engine(Config{}) {}
Engine::Engine(Config cfg) : impl_(std::make_unique<Impl>(cfg)) {}
Engine::~Engine() = default;
task::DownloadHandle Engine::start(task::DownloadSpec spec, task::DownloadCallbacks cbs) {
return impl_->start(std::move(spec), std::move(cbs));
}
segment::SegmentBudget &Engine::segment_budget() noexcept {
return impl_->budget_;
}
rate::RateLimiter &Engine::rate_limiter() noexcept {
return impl_->limiter_;
}
void Engine::set_default_segments(std::uint32_t n) {
impl_->cfg_.default_segments = n ? n : 1;
}
void Engine::set_default_buffer_bytes(std::uint64_t b) {
impl_->cfg_.default_buffer_bytes = b;
}
void Engine::set_max_total_buffer_bytes(std::uint64_t b) {
impl_->cfg_.max_total_buffer_bytes = b;
}
void Engine::set_probe_pool_size(std::uint32_t) { /* prober pool is fixed at construction in M1 */ }
void Engine::probe(net::ProbeRequest req, std::function<void(Result<net::ProbeResult>)> done) {
impl_->prober_.probe(std::move(req), std::move(done));
}
} // namespace vdm
+186
View File
@@ -0,0 +1,186 @@
// vdm/io/sparse_file.cpp
#include "vdm/io/sparse_file.hpp"
#include <fcntl.h>
#include <unistd.h>
#include <cerrno>
#include <cstring>
#include <utility>
namespace vdm::io {
namespace {
Error errno_to_error(int e) noexcept {
switch (e) {
case ENOSPC:
case EDQUOT:
return Error::disk_full;
case EACCES:
case EPERM:
case EROFS:
return Error::permission_denied;
case ENOENT:
case ENOTDIR:
case EISDIR:
case ENAMETOOLONG:
case ELOOP:
return Error::path_rejected;
default:
return Error::io_error;
}
}
ErrorInfo sys_error(std::string_view what, int e) {
return ErrorInfo(errno_to_error(e), std::string(what) + ": " + std::strerror(e));
}
} // namespace
SparseFile::~SparseFile() {
if (fd_ >= 0)
::close(fd_);
}
SparseFile::SparseFile(SparseFile &&o) noexcept
: fd_(std::exchange(o.fd_, -1)),
preallocated_(std::exchange(o.preallocated_, false)),
path_(std::move(o.path_)) {}
SparseFile &SparseFile::operator=(SparseFile &&o) noexcept {
if (this != &o) {
if (fd_ >= 0)
::close(fd_);
fd_ = std::exchange(o.fd_, -1);
preallocated_ = std::exchange(o.preallocated_, false);
path_ = std::move(o.path_);
}
return *this;
}
void SparseFile::reset() noexcept {
fd_ = -1;
preallocated_ = false;
path_.clear();
}
Result<void> SparseFile::open(std::string_view path) {
return open(path, OpenOptions{});
}
Result<void> SparseFile::open(std::string_view path, const OpenOptions &opts) {
if (fd_ >= 0)
return ErrorInfo(Error::internal, "SparseFile already open");
std::string p(path);
// O_NOFOLLOW: the final component of a download target must never be a symlink, on
// create or on resume. DAEMON canonicalises the path and checks it against the allowed
// roots before start(), but a symlink swapped in afterwards would redirect our writes
// outside those roots (daemon/docs/safepath-adversarial.md leans on this open closing
// that TOCTOU window). A symlinked leaf fails here with ELOOP -> Error::path_rejected.
int flags = O_WRONLY | O_CREAT | O_CLOEXEC | O_NOFOLLOW;
if (opts.truncate_existing)
flags |= O_TRUNC;
int fd = ::open(p.c_str(), flags, 0644);
if (fd < 0)
return sys_error("open " + p, errno);
bool prealloc = false;
if (opts.total_size > 0) {
if (opts.preallocate) {
// posix_fallocate returns the error number directly and does not set errno.
int rc = ::posix_fallocate(fd, 0, static_cast<off_t>(opts.total_size));
if (rc == 0) {
prealloc = true;
} else if (rc == EOPNOTSUPP || rc == ENOSYS || rc == EINVAL) {
if (::ftruncate(fd, static_cast<off_t>(opts.total_size)) != 0) {
int e = errno;
::close(fd);
return sys_error("ftruncate " + p, e);
}
} else {
::close(fd);
return sys_error("posix_fallocate " + p, rc);
}
} else if (!opts.truncate_existing) {
// Resuming: make sure the file is at least total_size so pwrite offsets land.
if (::ftruncate(fd, static_cast<off_t>(opts.total_size)) != 0) {
int e = errno;
::close(fd);
return sys_error("ftruncate " + p, e);
}
}
}
fd_ = fd;
preallocated_ = prealloc;
path_ = std::move(p);
return ok();
}
Result<void> SparseFile::write_at(std::uint64_t offset, ConstByteSpan data) {
if (fd_ < 0)
return ErrorInfo(Error::internal, "write_at on a closed SparseFile");
const std::byte *p = data.data();
std::size_t remaining = data.size();
off_t pos = static_cast<off_t>(offset);
while (remaining > 0) {
ssize_t n = ::pwrite(fd_, p, remaining, pos);
if (n < 0) {
if (errno == EINTR)
continue;
return sys_error("pwrite", errno);
}
if (n == 0)
return ErrorInfo(Error::io_error, "pwrite returned 0");
p += n;
pos += n;
remaining -= static_cast<std::size_t>(n);
}
return ok();
}
Result<void> SparseFile::sync() {
if (fd_ < 0)
return ErrorInfo(Error::internal, "sync on a closed SparseFile");
while (::fdatasync(fd_) != 0) {
if (errno == EINTR)
continue;
return sys_error("fdatasync", errno);
}
return ok();
}
void SparseFile::advise_dontneed(std::uint64_t offset, std::uint64_t len) noexcept {
if (fd_ < 0 || len == 0)
return;
::posix_fadvise(fd_, static_cast<off_t>(offset), static_cast<off_t>(len), POSIX_FADV_DONTNEED);
}
Result<void> SparseFile::resize(std::uint64_t size) {
if (fd_ < 0)
return ErrorInfo(Error::internal, "resize on a closed SparseFile");
while (::ftruncate(fd_, static_cast<off_t>(size)) != 0) {
if (errno == EINTR)
continue;
return sys_error("ftruncate", errno);
}
return ok();
}
Result<void> SparseFile::close() {
if (fd_ < 0)
return ok();
int fd = std::exchange(fd_, -1);
int rc = ::close(fd);
reset();
if (rc != 0)
return sys_error("close", errno);
return ok();
}
} // namespace vdm::io
+57
View File
@@ -0,0 +1,57 @@
// vdm/io/write_buffer.cpp
#include "vdm/io/write_buffer.hpp"
#include <algorithm>
#include <cassert>
#include <cstring>
#include <utility>
namespace vdm::io {
WriteBuffer::WriteBuffer(std::uint64_t start_offset, std::size_t capacity, FlushFn flush)
: buf_(capacity), base_(start_offset), flush_(std::move(flush)) {
assert(capacity > 0 && "WriteBuffer capacity must be > 0");
}
Result<void> WriteBuffer::flush_pending() {
if (len_ == 0)
return ok();
VDM_TRY(flush_(base_, ConstByteSpan(buf_.data(), len_)));
base_ += len_;
len_ = 0;
return ok();
}
Result<void> WriteBuffer::append(ConstByteSpan span) {
// On an error return, next_offset() still reflects exactly what is durable; buffered
// (non-durable) bytes and any unconsumed tail of `span` are the caller's to abandon —
// the segment aborts and resumes/restarts from next_offset().
while (!span.empty()) {
// Buffer empty and the incoming chunk fills at least a whole buffer: write it
// straight through — no memcpy, no allocation, just an extra flush call.
if (len_ == 0 && span.size() >= buf_.size()) {
VDM_TRY(flush_(base_, span));
base_ += span.size();
appended_ += span.size();
return ok();
}
const std::size_t room = buf_.size() - len_;
const std::size_t n = std::min(span.size(), room);
std::memcpy(buf_.data() + len_, span.data(), n);
len_ += n;
appended_ += n;
span = span.subspan(n);
if (len_ == buf_.size())
VDM_TRY(flush_pending());
}
return ok();
}
Result<void> WriteBuffer::flush() {
return flush_pending();
}
} // namespace vdm::io
+311
View File
@@ -0,0 +1,311 @@
// vdm/meta/veloxpart.cpp
//
// Reader first (AGENT-CORE §5). parse_veloxpart() is the attacker-facing surface; it is
// total on any byte string.
#include "vdm/meta/veloxpart.hpp"
#include <fcntl.h>
#include <unistd.h>
#include <cerrno>
#include <cstring>
#include <string>
#include <utility>
#include "vdm/util/crc32.hpp"
namespace vdm::meta {
namespace {
// magic(4)+ver(2)+flags(2)+total(8)+downloaded(8)+url_count(4)
// +etag_len(4)+lm_len(4)+ct_len(4)+seg_count(4)+crc(4)
constexpr std::size_t kMinImageBytes = 52;
constexpr char kMagic[4] = {'V', 'D', 'M', 'P'};
Error errno_to_error(int e) noexcept {
switch (e) {
case ENOSPC:
case EDQUOT:
return Error::disk_full;
case EACCES:
case EPERM:
case EROFS:
return Error::permission_denied;
case ENOENT:
case ENOTDIR:
case EISDIR:
case ENAMETOOLONG:
case ELOOP:
return Error::path_rejected;
default:
return Error::io_error;
}
}
ErrorInfo sys_error(std::string_view what, int e) {
return ErrorInfo(errno_to_error(e), std::string(what) + ": " + std::strerror(e));
}
ErrorInfo corrupt(std::string_view where) {
return ErrorInfo(Error::meta_corrupt, std::string(".veloxpart.meta: ") + std::string(where));
}
} // namespace
// ---------------------------------------------------------------------------------------
// Reader
Result<VeloxPart> parse_veloxpart(ConstByteSpan image) {
if (image.size() > kMaxImageBytes)
return corrupt("image exceeds cap");
if (image.size() < kMinImageBytes)
return corrupt("image shorter than the header");
// CRC over everything but the trailing u32 — reject before interpreting any field.
const ConstByteSpan body = image.first(image.size() - 4);
const std::uint32_t want = load_le<std::uint32_t>(image.subspan(image.size() - 4));
if (crc32(body) != want)
return corrupt("crc32 mismatch");
ByteReader r(body);
if (as_chars(r.raw(4)) != std::string_view(kMagic, 4))
return corrupt("bad magic");
VeloxPart vp;
vp.version = r.u16();
vp.flags = r.u16();
if (vp.version > kVersion)
return ErrorInfo(Error::meta_version_unsupported,
".veloxpart.meta: version " + std::to_string(vp.version) +
" > supported " + std::to_string(kVersion));
vp.total_size = r.u64();
vp.downloaded = r.u64();
const std::uint32_t url_count = r.u32();
if (url_count > kMaxUrls)
return corrupt("url_count past cap");
if (static_cast<std::uint64_t>(url_count) * 4 > r.remaining())
return corrupt("url_count");
vp.urls.reserve(url_count);
for (std::uint32_t i = 0; i < url_count; ++i) {
std::string_view s = r.lp_string();
if (r.overran() || s.size() > kMaxStringLen)
return corrupt("url");
vp.urls.emplace_back(s);
}
auto read_str = [&](std::string &dst, std::string_view what) -> Result<void> {
std::string_view s = r.lp_string();
if (r.overran() || s.size() > kMaxStringLen)
return corrupt(what);
dst.assign(s);
return ok();
};
VDM_TRY(read_str(vp.etag, "etag"));
VDM_TRY(read_str(vp.last_modified, "last_modified"));
VDM_TRY(read_str(vp.content_type, "content_type"));
const std::uint32_t seg_count = r.u32();
if (seg_count > kMaxSegments)
return corrupt("segment_count past cap");
if (static_cast<std::uint64_t>(seg_count) * 24 > r.remaining())
return corrupt("segment_count");
vp.segments.reserve(seg_count);
for (std::uint32_t i = 0; i < seg_count; ++i) {
SegmentRecord s;
s.start = r.u64();
s.end = r.u64();
s.completed = r.u64();
if (r.overran())
return corrupt("segment record");
if (s.end >= s.start && s.completed > s.end - s.start + 1)
return corrupt("segment.completed exceeds its range");
vp.segments.push_back(s);
}
if (vp.flags & kFlagHasShaState) {
const std::uint32_t n = r.u32();
if (r.overran() || n > kMaxShaStateLen)
return corrupt("sha256_state length");
ConstByteSpan blob = r.raw(n);
if (r.overran())
return corrupt("sha256_state body");
vp.sha256_state.assign(blob.begin(), blob.end());
}
if (r.overran())
return corrupt("truncated");
if (r.remaining() != 0)
return corrupt("trailing bytes after the record");
return vp;
}
// ---------------------------------------------------------------------------------------
// Writer
std::vector<std::byte> serialize_veloxpart(const VeloxPart &vp) {
std::vector<std::byte> out;
std::size_t est = kMinImageBytes + vp.segments.size() * 24 + vp.sha256_state.size() + 64;
for (const auto &u : vp.urls)
est += 4 + u.size();
est += vp.etag.size() + vp.last_modified.size() + vp.content_type.size();
out.reserve(est);
auto put_bytes = [&](const void *p, std::size_t n) {
const auto *b = static_cast<const std::byte *>(p);
out.insert(out.end(), b, b + n);
};
auto put_u16 = [&](std::uint16_t v) {
std::byte t[2];
store_le<std::uint16_t>(t, v);
put_bytes(t, 2);
};
auto put_u32 = [&](std::uint32_t v) {
std::byte t[4];
store_le<std::uint32_t>(t, v);
put_bytes(t, 4);
};
auto put_u64 = [&](std::uint64_t v) {
std::byte t[8];
store_le<std::uint64_t>(t, v);
put_bytes(t, 8);
};
auto put_str = [&](std::string_view s) {
put_u32(static_cast<std::uint32_t>(s.size()));
put_bytes(s.data(), s.size());
};
// Normalise the sha-state flag to match the payload so a round-trip is exact.
std::uint16_t flags = vp.flags;
if (vp.sha256_state.empty())
flags &= static_cast<std::uint16_t>(~kFlagHasShaState);
else
flags |= kFlagHasShaState;
put_bytes(kMagic, 4);
put_u16(vp.version);
put_u16(flags);
put_u64(vp.total_size);
put_u64(vp.downloaded);
put_u32(static_cast<std::uint32_t>(vp.urls.size()));
for (const auto &u : vp.urls)
put_str(u);
put_str(vp.etag);
put_str(vp.last_modified);
put_str(vp.content_type);
put_u32(static_cast<std::uint32_t>(vp.segments.size()));
for (const auto &s : vp.segments) {
put_u64(s.start);
put_u64(s.end);
put_u64(s.completed);
}
if (!vp.sha256_state.empty()) {
put_u32(static_cast<std::uint32_t>(vp.sha256_state.size()));
put_bytes(vp.sha256_state.data(), vp.sha256_state.size());
}
put_u32(crc32(ConstByteSpan(out.data(), out.size())));
return out;
}
// ---------------------------------------------------------------------------------------
// File helpers
Result<VeloxPart> read_veloxpart_file(std::string_view path) {
std::string p(path);
int fd = ::open(p.c_str(), O_RDONLY | O_CLOEXEC);
if (fd < 0)
return sys_error("open " + p, errno);
std::vector<std::byte> buf;
buf.resize(kMaxImageBytes + 1);
std::size_t total = 0;
for (;;) {
ssize_t n = ::read(fd, buf.data() + total, buf.size() - total);
if (n < 0) {
if (errno == EINTR)
continue;
int e = errno;
::close(fd);
return sys_error("read " + p, e);
}
if (n == 0)
break;
total += static_cast<std::size_t>(n);
if (total > kMaxImageBytes) {
::close(fd);
return corrupt("sidecar file exceeds cap");
}
}
::close(fd);
buf.resize(total);
return parse_veloxpart(ConstByteSpan(buf.data(), buf.size()));
}
Result<void> write_veloxpart_file(std::string_view path, const VeloxPart &vp, bool fsync) {
std::string p(path);
std::string tmp = p + ".tmp";
std::vector<std::byte> image = serialize_veloxpart(vp);
int fd = ::open(tmp.c_str(), O_WRONLY | O_CREAT | O_TRUNC | O_CLOEXEC, 0644);
if (fd < 0)
return sys_error("open " + tmp, errno);
const std::byte *pd = image.data();
std::size_t remaining = image.size();
while (remaining > 0) {
ssize_t n = ::write(fd, pd, remaining);
if (n < 0) {
if (errno == EINTR)
continue;
int e = errno;
::close(fd);
::unlink(tmp.c_str());
return sys_error("write " + tmp, e);
}
pd += n;
remaining -= static_cast<std::size_t>(n);
}
if (fsync) {
while (::fdatasync(fd) != 0) {
if (errno == EINTR)
continue;
int e = errno;
::close(fd);
::unlink(tmp.c_str());
return sys_error("fdatasync " + tmp, e);
}
}
if (::close(fd) != 0) {
int e = errno;
::unlink(tmp.c_str());
return sys_error("close " + tmp, e);
}
if (::rename(tmp.c_str(), p.c_str()) != 0) {
int e = errno;
::unlink(tmp.c_str());
return sys_error("rename " + tmp + " -> " + p, e);
}
if (fsync) {
// fsync the directory so the rename itself is durable.
std::string dir = p.substr(0, p.find_last_of('/'));
if (dir.empty() || dir == p)
dir = ".";
int dfd = ::open(dir.c_str(), O_RDONLY | O_DIRECTORY | O_CLOEXEC);
if (dfd >= 0) {
while (::fsync(dfd) != 0 && errno == EINTR) {
}
::close(dfd);
}
}
return ok();
}
} // namespace vdm::meta
+292
View File
@@ -0,0 +1,292 @@
// vdm/net/content_disposition.cpp
#include "vdm/net/content_disposition.hpp"
#include <algorithm>
#include <string>
#include <utility>
#include <vector>
#include "net/text_codec.hpp"
namespace vdm::net {
namespace {
using detail::decode_rfc2047;
using detail::latin1_to_utf8;
using detail::percent_decode;
using detail::to_utf8_best_effort;
std::string_view trim(std::string_view s) {
while (!s.empty() && (s.front() == ' ' || s.front() == '\t'))
s.remove_prefix(1);
while (!s.empty() &&
(s.back() == ' ' || s.back() == '\t' || s.back() == '\r' || s.back() == '\n'))
s.remove_suffix(1);
return s;
}
std::string ascii_lower(std::string_view s) {
std::string r(s);
for (char &c : r)
if (c >= 'A' && c <= 'Z')
c = static_cast<char>(c - 'A' + 'a');
return r;
}
// Split "type; a=b; c*=d; e=\"f;g\"" into the type and a param list, honouring quoted
// strings (a ';' inside quotes is not a separator). For quoted values the value stored is
// the *raw* inner text (quotes removed, escapes NOT yet resolved) with `quoted = true`;
// callers resolve escapes and strip path components together (order matters — see
// unquote_strip). Keys are lowercased.
struct Param {
std::string key;
std::string value;
bool quoted = false;
};
struct Params {
std::string type;
std::vector<Param> kv;
[[nodiscard]] const Param *find(std::string_view key) const {
for (const auto &p : kv)
if (p.key == key)
return &p;
return nullptr;
}
};
Params tokenize(std::string_view h) {
Params out;
std::size_t i = 0;
const std::size_t n = h.size();
auto read_segment = [&]() -> std::string_view {
std::size_t start = i;
bool in_q = false;
for (; i < n; ++i) {
char c = h[i];
if (c == '"') {
in_q = !in_q;
} else if (c == '\\' && in_q && i + 1 < n) {
++i; // skip escaped char
} else if (c == ';' && !in_q) {
break;
}
}
std::string_view seg = h.substr(start, i - start);
if (i < n)
++i; // consume ';'
return seg;
};
out.type = ascii_lower(trim(read_segment()));
while (i < n) {
std::string_view seg = trim(read_segment());
if (seg.empty())
continue;
auto eq = seg.find('=');
if (eq == std::string_view::npos) {
out.kv.emplace_back(ascii_lower(seg), std::string{});
continue;
}
std::string key = ascii_lower(trim(seg.substr(0, eq)));
std::string_view rawval = trim(seg.substr(eq + 1));
Param param;
param.key = std::move(key);
if (rawval.size() >= 2 && rawval.front() == '"') {
std::string_view inner = rawval.substr(1);
auto close = inner.rfind('"');
if (close != std::string_view::npos)
inner = inner.substr(0, close);
param.value.assign(inner); // raw, escapes unresolved
param.quoted = true;
} else {
param.value.assign(rawval);
}
out.kv.push_back(std::move(param));
}
return out;
}
// Drop C0 control bytes and DEL, then trim edge whitespace. NUL and control characters
// are never a legitimate part of a filename and are a classic truncation/spoofing vector,
// so the decode layer strips them even though rules/ (stage 9) owns the authoritative
// sanitize. `..` and other "unsafe but printable" content is left for rules/.
std::string sanitize_leaf(std::string s) {
std::string out;
out.reserve(s.size());
for (unsigned char c : s)
if (c >= 0x20 && c != 0x7F)
out.push_back(static_cast<char>(c));
std::string_view v = trim(out);
return std::string(v);
}
std::string strip_path(std::string s) {
auto slash = s.find_last_of("/\\");
if (slash != std::string::npos)
s.erase(0, slash + 1);
return s;
}
// Resolve ONLY the `\"` escape (needed so a quote can appear mid-name). Every other
// backslash is kept literal and later treated as a path separator by strip_path — real
// Windows paths in the wild use `\` unescaped, and path-traversal defence matters more
// than supporting the vanishingly rare filename with a literal backslash.
std::string unescape_dquote(std::string_view raw) {
std::string out;
out.reserve(raw.size());
for (std::size_t i = 0; i < raw.size(); ++i) {
if (raw[i] == '\\' && i + 1 < raw.size() && raw[i + 1] == '"') {
out.push_back('"');
++i;
} else {
out.push_back(raw[i]);
}
}
return out;
}
// Decode an RFC 5987 ext-value: charset'lang'pct-encoded-octets
std::string decode_ext_value(std::string_view v) {
auto q1 = v.find('\'');
if (q1 == std::string_view::npos)
return percent_decode(v); // malformed: best effort
auto q2 = v.find('\'', q1 + 1);
if (q2 == std::string_view::npos)
return percent_decode(v.substr(q1 + 1));
std::string_view charset = v.substr(0, q1);
std::string_view enc = v.substr(q2 + 1);
std::string bytes = percent_decode(enc);
std::string cs = ascii_lower(charset);
if (cs == "iso-8859-1" || cs == "latin1")
return latin1_to_utf8(bytes);
return to_utf8_best_effort(bytes); // utf-8 or unknown -> best effort
}
// Reassemble RFC 2231 continuations: name*0*, name*1, name*2* ... in order.
std::string join_continuations(const Params &p, std::string_view base, bool &is_ext) {
std::vector<std::pair<int, std::string>> parts;
is_ext = false;
for (const auto &param : p.kv) {
const std::string &k = param.key;
const std::string &val = param.value;
if (k.size() <= base.size() + 1 || k.compare(0, base.size(), base) != 0)
continue;
if (k[base.size()] != '*')
continue;
std::string_view rest(k);
rest.remove_prefix(base.size() + 1); // after "base*"
bool star = false;
if (!rest.empty() && rest.back() == '*') {
star = true;
rest.remove_suffix(1);
}
int idx = 0;
for (char c : rest) {
if (c < '0' || c > '9') {
idx = -1;
break;
}
idx = idx * 10 + (c - '0');
}
if (idx < 0)
continue;
if (star)
is_ext = true;
parts.emplace_back(idx, val);
}
if (parts.empty())
return {};
std::sort(parts.begin(), parts.end(),
[](const auto &a, const auto &b) { return a.first < b.first; });
// Piece 0 (if star-form) carries charset'lang' prefix; later pieces are raw
// percent-encoded. Concatenate the percent-encoded text then decode once.
std::string charset_prefix;
std::string enc;
bool first = true;
for (auto &[idx, val] : parts) {
if (first && is_ext) {
auto q1 = val.find('\'');
auto q2 = (q1 == std::string::npos) ? std::string::npos : val.find('\'', q1 + 1);
if (q2 != std::string::npos) {
charset_prefix = val.substr(0, q2 + 1);
enc += val.substr(q2 + 1);
} else {
enc += val;
}
} else {
enc += val;
}
first = false;
}
if (is_ext)
return decode_ext_value(charset_prefix + enc);
return to_utf8_best_effort(percent_decode(enc));
}
ContentDisposition::Type classify(std::string_view t) {
if (t == "inline")
return ContentDisposition::Type::inline_;
if (t == "attachment")
return ContentDisposition::Type::attachment;
if (t == "form-data")
return ContentDisposition::Type::form_data;
if (t.empty())
return ContentDisposition::Type::none;
return ContentDisposition::Type::other;
}
} // namespace
ContentDisposition parse_content_disposition(std::string_view header_value) {
ContentDisposition cd;
header_value = trim(header_value);
if (header_value.empty())
return cd;
Params p = tokenize(header_value);
cd.type = classify(p.type);
// RFC 6266 §4.3: prefer filename* over filename.
std::string ext_name;
bool ext_is_ext = false;
if (const Param *fstar = p.find("filename*")) {
ext_name = decode_ext_value(fstar->value);
ext_is_ext = true;
} else {
std::string joined = join_continuations(p, "filename", ext_is_ext);
if (!joined.empty())
ext_name = std::move(joined);
}
std::string plain_name;
if (const Param *f = p.find("filename")) {
// Resolve `\"`, then decode legacy encoded-words if present — BEFORE stripping
// path components, since a base64 payload can legitimately contain '/'.
std::string raw = f->quoted ? unescape_dquote(f->value) : f->value;
bool had_ew = false;
std::string decoded = decode_rfc2047(raw, &had_ew);
plain_name = had_ew ? std::move(decoded) : to_utf8_best_effort(raw);
}
if (!ext_name.empty()) {
cd.filename = strip_path(std::move(ext_name));
cd.filename_from_ext = ext_is_ext;
} else if (!plain_name.empty()) {
cd.filename = strip_path(std::move(plain_name));
cd.filename_from_ext = false;
}
// Drop control bytes (incl. NUL) and edge whitespace the decoders may have produced.
cd.filename = sanitize_leaf(std::move(cd.filename));
return cd;
}
} // namespace vdm::net
+20 -4
View File
@@ -195,6 +195,13 @@ struct HttpClient::Impl {
std::string_view line(buf, total);
if (line.starts_with("HTTP/")) {
// A new status line after we already delivered a 401/407 means libcurl's
// CURLAUTH_ANY handshake just resent with credentials: let the head of this
// second response be delivered too, so callers see the real (2xx/4xx) status
// rather than the challenge. Redirects never reach here delivered — their head
// is suppressed below — so this only fires for the auth resend.
if (st->head_delivered && (st->line_status == 401 || st->line_status == 407))
st->head_delivered = false;
st->line_status = status_from_line(line);
st->head.headers.clear(); // keep only the final response's headers
return total;
@@ -283,11 +290,20 @@ struct HttpClient::Impl {
}
void drain_commands(Worker &w) {
// Called every worker-loop iteration (run(), below) -- once per curl_multi_poll
// wake, so once per socket-readiness event on the transfer hot path -- but commands
// (add/pause/resume/cancel) are rare next to that. Check empty under the lock
// *before* touching `local`: libstdc++'s std::deque allocates its map array on
// default construction even with nothing pushed to it, so constructing one every
// iteration just to usually swap nothing into it was an allocation on every poll
// wake, not just on an actual command -- exactly what the curl-write-callback path
// must never do (AGENT-CORE.md; caught by tools/bench's alloc-check).
std::unique_lock lk(w.mu);
if (w.queue.empty())
return;
std::deque<Command> local;
{
std::lock_guard lk(w.mu);
local.swap(w.queue);
}
local.swap(w.queue);
lk.unlock();
for (auto &cmd : local) {
auto &st = cmd.state;
switch (cmd.kind) {
+360
View File
@@ -0,0 +1,360 @@
// vdm/net/probe.cpp
#include "vdm/net/probe.hpp"
#include <charconv>
#include <deque>
#include <mutex>
#include "net/curl_error.hpp"
#include "vdm/net/http_client.hpp"
#include "vdm/net/url.hpp"
namespace vdm::net {
namespace {
std::string_view trim(std::string_view s) {
while (!s.empty() && (s.front() == ' ' || s.front() == '\t'))
s.remove_prefix(1);
while (!s.empty() && (s.back() == ' ' || s.back() == '\t'))
s.remove_suffix(1);
return s;
}
bool iequals(std::string_view a, std::string_view b) {
return HeaderList::iequals(a, b);
}
// "bytes 0-0/12345" -> 12345 ; "bytes 0-0/*" or malformed -> nullopt
std::optional<std::uint64_t> total_from_content_range(std::string_view v) {
auto slash = v.find('/');
if (slash == std::string_view::npos)
return std::nullopt;
std::string_view tail = trim(v.substr(slash + 1));
if (tail.empty() || tail == "*")
return std::nullopt;
std::uint64_t n = 0;
auto [p, ec] = std::from_chars(tail.data(), tail.data() + tail.size(), n);
(void)p;
if (ec != std::errc{})
return std::nullopt;
return n;
}
std::string light_sanitize(std::string_view in) {
std::string out;
out.reserve(in.size());
for (unsigned char c : in) {
if (c == '/' || c == '\\' || c == 0) {
out.push_back('_');
} else if (c >= 0x20 || (c & 0x80)) { // keep printable ASCII + all UTF-8 bytes
out.push_back(static_cast<char>(c));
}
}
while (!out.empty() && (out.back() == '.' || out.back() == ' '))
out.pop_back();
if (out == "." || out == "..")
out.clear();
return out;
}
} // namespace
std::string suggest_filename(const ProbeResult &r, std::string_view explicit_name) {
std::string cand;
if (!explicit_name.empty())
cand = light_sanitize(explicit_name);
if (cand.empty() && !r.filename_from_disposition.empty())
cand = light_sanitize(r.filename_from_disposition);
if (cand.empty() && !r.filename_from_url.empty())
cand = light_sanitize(r.filename_from_url);
if (cand.empty())
cand = "download.bin";
return cand;
}
// --- Prober::Impl ---------------------------------------------------------------------
struct Prober::Impl {
struct Job {
ProbeRequest req;
std::function<void(Result<ProbeResult>)> done;
};
struct P {
Impl *self = nullptr;
Job job;
ProbeResult result;
bool had_head_ok = false;
bool delivered = false;
};
explicit Impl(unsigned max_concurrent)
: max_(max_concurrent ? max_concurrent : 1), client_(HttpClient::Options{.workers = 2}) {}
unsigned max_;
HttpClient client_;
std::mutex mu_;
unsigned inflight_ = 0;
std::deque<Job> pending_;
void submit(Job j) {
{
std::lock_guard lk(mu_);
if (inflight_ >= max_) {
pending_.push_back(std::move(j));
return;
}
++inflight_;
}
start(std::move(j));
}
void finish_one() {
Job next;
bool have_next = false;
{
std::lock_guard lk(mu_);
--inflight_;
if (!pending_.empty()) {
next = std::move(pending_.front());
pending_.pop_front();
++inflight_;
have_next = true;
}
}
if (have_next)
start(std::move(next));
}
Request base_request(const ProbeRequest &pr) {
Request r;
r.url = pr.url;
r.headers = pr.headers;
r.cookies = pr.cookies;
r.user_agent = pr.user_agent;
r.referrer = pr.referrer;
r.proxy = pr.proxy;
r.auth = pr.auth;
r.follow_redirects = true;
r.accept_encoding = false;
r.connect_timeout_ms = pr.connect_timeout_ms;
r.overall_timeout_ms = pr.overall_timeout_ms;
r.low_speed_bytes_per_sec = 0; // probes are tiny; no stall detector
r.low_speed_secs = 0;
return r;
}
void start(Job j) {
auto p = std::make_shared<P>();
p->self = this;
p->job = std::move(j);
p->result.effective_url = p->job.req.url;
Request req = base_request(p->job.req);
req.method = Method::head;
TransferCallbacks cbs;
cbs.on_head = [p](const ResponseHead &h) {
absorb_head(p->result, h);
return head_action(p, h);
};
cbs.on_data = [](ConstByteSpan) { return DataAction::abort; };
cbs.on_finished = [p](Result<TransferStats> r) { on_head_done(p, std::move(r)); };
// The worker keeps Transfer::State alive; the callbacks keep `p` alive. Storing
// the Transfer in `p` would make a p -> Transfer -> State -> cbs -> p cycle.
client_.start(std::move(req), std::move(cbs));
}
// We want no body from a probe, so the head callback normally aborts after headers.
// The exception: a 401/407 when we were handed credentials — libcurl's CURLAUTH_ANY
// has to see that response before it resends with Authorization, so let this one
// through (a HEAD has no body; the ranged GET's is a single byte). The final status
// then lands on the next header block.
static DataAction head_action(const std::shared_ptr<P> &p, const ResponseHead &h) {
if ((h.status == 401 || h.status == 407) && p->job.req.auth.scheme != AuthScheme::none)
return DataAction::proceed;
return DataAction::abort;
}
static void absorb_head(ProbeResult &res, const ResponseHead &h) {
if (h.status)
res.http_status = h.status;
if (!h.effective_url.empty())
res.effective_url = h.effective_url;
refresh_common(res, h);
if (h.content_length && !res.total_size)
res.total_size = h.content_length;
}
static void refresh_common(ProbeResult &res, const ResponseHead &h) {
if (auto v = h.headers.get("Content-Type"); v && res.mime.empty()) {
std::string_view mv = *v;
mv = mv.substr(0, mv.find(';'));
res.mime.assign(trim(mv));
}
if (auto v = h.headers.get("ETag"); v && res.etag.empty())
res.etag.assign(*v);
if (auto v = h.headers.get("Last-Modified"); v && res.last_modified.empty())
res.last_modified.assign(*v);
if (auto v = h.headers.get("Accept-Ranges")) {
if (iequals(trim(*v), "bytes"))
res.accept_ranges = true;
}
if (auto v = h.headers.get("Content-Disposition");
v && res.filename_from_disposition.empty()) {
ContentDisposition cd = parse_content_disposition(*v);
res.disposition_type = cd.type;
if (cd.has_filename())
res.filename_from_disposition = cd.filename;
}
}
static bool has_validator(const ProbeResult &r) {
return !r.etag.empty() || !r.last_modified.empty();
}
static void on_head_done(std::shared_ptr<P> p, Result<TransferStats> r) {
const long status = p->result.http_status;
if (status == 0) { // never got headers -> transport failure
deliver(p, r.has_value() ? Result<ProbeResult>(ErrorInfo(Error::probe_failed))
: Result<ProbeResult>(std::move(r).error()));
return;
}
p->had_head_ok = (status >= 200 && status < 300);
if (status == 401 || status == 407) {
p->result.requires_auth = true;
finalize_and_deliver(p);
return;
}
if (status == 403 || status == 405 || status == 501) {
start_range_get(p); // HEAD refused; the ranged GET is now the primary probe
return;
}
if (status >= 400) {
deliver(p, ErrorInfo(detail::error_from_curl(CURLE_OK, status), "probe HEAD",
static_cast<int>(status)));
return;
}
// 2xx. Prove resumability with a ranged GET unless the server said "none".
if (auto ar = p->result.accept_ranges; !ar) {
// Accept-Ranges absent or not "bytes": still try one ranged GET — servers
// that support ranges without advertising are common (docs/06 R4).
}
start_range_get(p);
}
static void start_range_get(const std::shared_ptr<P> &p) {
Request req = p->self->base_request(p->job.req);
req.method = Method::get;
req.range = ByteRange{0, 0};
TransferCallbacks cbs;
cbs.on_head = [p](const ResponseHead &h) {
absorb_range_head(p->result, h);
return head_action(p, h); // abort after headers, except a 401/407 with creds
};
cbs.on_data = [](ConstByteSpan) { return DataAction::abort; };
cbs.on_finished = [p](Result<TransferStats> r) { on_range_done(p, std::move(r)); };
p->self->client_.start(std::move(req), std::move(cbs));
}
static void absorb_range_head(ProbeResult &res, const ResponseHead &h) {
if (h.status)
res.http_status = h.status;
if (!h.effective_url.empty())
res.effective_url = h.effective_url;
refresh_common(res, h);
if (h.status == 206) {
res.accept_ranges = true; // proven, not just advertised
if (auto cr = h.headers.get("Content-Range")) {
if (auto total = total_from_content_range(*cr))
res.total_size = total;
}
} else if (h.status == 200) {
if (h.content_length)
res.total_size = h.content_length;
}
}
static void on_range_done(std::shared_ptr<P> p, Result<TransferStats> r) {
const long status = p->result.http_status;
if (status == 0) { // range GET died at transport level
if (p->had_head_ok) {
p->result.resumable = false;
finalize_and_deliver(p);
} else {
deliver(p, r.has_value() ? Result<ProbeResult>(ErrorInfo(Error::probe_failed))
: Result<ProbeResult>(std::move(r).error()));
}
return;
}
if (status == 401 || status == 407) {
p->result.requires_auth = true;
finalize_and_deliver(p);
return;
}
if (status == 206) {
p->result.resumable = has_validator(p->result);
finalize_and_deliver(p);
return;
}
if (status == 200 || status == 416) {
p->result.resumable = false;
finalize_and_deliver(p);
return;
}
if (status >= 400) {
if (p->had_head_ok) {
p->result.resumable = false;
finalize_and_deliver(p);
} else {
deliver(p, ErrorInfo(detail::error_from_curl(CURLE_OK, status), "probe ranged GET",
static_cast<int>(status)));
}
return;
}
p->result.resumable = false;
finalize_and_deliver(p);
}
static void finalize_and_deliver(const std::shared_ptr<P> &p) {
ProbeResult &res = p->result;
if (res.effective_url.empty())
res.effective_url = p->job.req.url;
res.filename_from_url = url_filename(res.effective_url);
if (res.filename_from_url.empty() && res.effective_url != p->job.req.url)
res.filename_from_url = url_filename(p->job.req.url);
res.redirect_chain.clear();
res.redirect_chain.push_back(p->job.req.url);
if (res.effective_url != p->job.req.url)
res.redirect_chain.push_back(res.effective_url);
deliver(p, ProbeResult(res));
}
static void deliver(const std::shared_ptr<P> &p, Result<ProbeResult> out) {
if (p->delivered)
return;
p->delivered = true;
Impl *self = p->self;
p->job.done(std::move(out));
self->finish_one();
}
};
// --- Prober -----------------------------------------------------------------------
Prober::Prober(unsigned max_concurrent) : impl_(std::make_unique<Impl>(max_concurrent)) {}
Prober::~Prober() = default;
void Prober::probe(ProbeRequest req, std::function<void(Result<ProbeResult>)> done) {
impl_->submit(Impl::Job{std::move(req), std::move(done)});
}
} // namespace vdm::net
+262
View File
@@ -0,0 +1,262 @@
// vdm/net/text_codec.cpp
#include "net/text_codec.hpp"
#include <array>
#include <cctype>
namespace vdm::net::detail {
namespace {
int hex_val(char c) noexcept {
if (c >= '0' && c <= '9')
return c - '0';
if (c >= 'a' && c <= 'f')
return c - 'a' + 10;
if (c >= 'A' && c <= 'F')
return c - 'A' + 10;
return -1;
}
void append_utf8(std::string &out, std::uint32_t cp) {
if (cp <= 0x7F) {
out.push_back(static_cast<char>(cp));
} else if (cp <= 0x7FF) {
out.push_back(static_cast<char>(0xC0 | (cp >> 6)));
out.push_back(static_cast<char>(0x80 | (cp & 0x3F)));
} else if (cp <= 0xFFFF) {
out.push_back(static_cast<char>(0xE0 | (cp >> 12)));
out.push_back(static_cast<char>(0x80 | ((cp >> 6) & 0x3F)));
out.push_back(static_cast<char>(0x80 | (cp & 0x3F)));
} else {
out.push_back(static_cast<char>(0xF0 | (cp >> 18)));
out.push_back(static_cast<char>(0x80 | ((cp >> 12) & 0x3F)));
out.push_back(static_cast<char>(0x80 | ((cp >> 6) & 0x3F)));
out.push_back(static_cast<char>(0x80 | (cp & 0x3F)));
}
}
bool charset_is(std::string_view cs, std::string_view want) {
if (cs.size() != want.size())
return false;
for (std::size_t i = 0; i < cs.size(); ++i) {
char a = cs[i], b = want[i];
if (a >= 'A' && a <= 'Z')
a = static_cast<char>(a - 'A' + 'a');
if (b >= 'A' && b <= 'Z')
b = static_cast<char>(b - 'A' + 'a');
if (a != b)
return false;
}
return true;
}
bool is_utf8(std::string_view cs) {
return charset_is(cs, "utf-8") || charset_is(cs, "utf8");
}
bool is_latin1(std::string_view cs) {
return charset_is(cs, "iso-8859-1") || charset_is(cs, "latin1") ||
charset_is(cs, "iso8859-1") || charset_is(cs, "windows-1252");
}
} // namespace
std::string percent_decode(std::string_view in, bool plus_as_space) {
std::string out;
out.reserve(in.size());
for (std::size_t i = 0; i < in.size(); ++i) {
char c = in[i];
if (c == '%' && i + 2 < in.size()) {
int hi = hex_val(in[i + 1]);
int lo = hex_val(in[i + 2]);
if (hi >= 0 && lo >= 0) {
out.push_back(static_cast<char>((hi << 4) | lo));
i += 2;
continue;
}
}
if (c == '+' && plus_as_space) {
out.push_back(' ');
continue;
}
out.push_back(c);
}
return out;
}
bool is_valid_utf8(std::string_view s) noexcept {
std::size_t i = 0;
const std::size_t n = s.size();
auto cont = [&](std::size_t k) {
return k < n && (static_cast<unsigned char>(s[k]) & 0xC0) == 0x80;
};
while (i < n) {
unsigned char c = static_cast<unsigned char>(s[i]);
if (c < 0x80) {
++i;
} else if ((c & 0xE0) == 0xC0) {
if (!cont(i + 1))
return false;
std::uint32_t cp = (c & 0x1F) << 6 | (static_cast<unsigned char>(s[i + 1]) & 0x3F);
if (cp < 0x80)
return false; // overlong
i += 2;
} else if ((c & 0xF0) == 0xE0) {
if (!cont(i + 1) || !cont(i + 2))
return false;
std::uint32_t cp = (c & 0x0F) << 12 |
(static_cast<unsigned char>(s[i + 1]) & 0x3F) << 6 |
(static_cast<unsigned char>(s[i + 2]) & 0x3F);
if (cp < 0x800 || (cp >= 0xD800 && cp <= 0xDFFF))
return false;
i += 3;
} else if ((c & 0xF8) == 0xF0) {
if (!cont(i + 1) || !cont(i + 2) || !cont(i + 3))
return false;
std::uint32_t cp = (c & 0x07) << 18 |
(static_cast<unsigned char>(s[i + 1]) & 0x3F) << 12 |
(static_cast<unsigned char>(s[i + 2]) & 0x3F) << 6 |
(static_cast<unsigned char>(s[i + 3]) & 0x3F);
if (cp < 0x10000 || cp > 0x10FFFF)
return false;
i += 4;
} else {
return false;
}
}
return true;
}
std::string latin1_to_utf8(std::string_view s) {
std::string out;
out.reserve(s.size() + s.size() / 2);
for (char ch : s)
append_utf8(out, static_cast<unsigned char>(ch));
return out;
}
std::string to_utf8_best_effort(std::string_view s) {
return is_valid_utf8(s) ? std::string(s) : latin1_to_utf8(s);
}
std::string base64_decode(std::string_view in) {
auto val = [](char c) -> int {
if (c >= 'A' && c <= 'Z')
return c - 'A';
if (c >= 'a' && c <= 'z')
return c - 'a' + 26;
if (c >= '0' && c <= '9')
return c - '0' + 52;
if (c == '+')
return 62;
if (c == '/')
return 63;
return -1;
};
std::string out;
out.reserve(in.size() / 4 * 3 + 3);
std::uint32_t acc = 0;
int bits = 0;
for (char c : in) {
if (c == '=' || c == '\r' || c == '\n' || c == ' ' || c == '\t')
continue;
int v = val(c);
if (v < 0)
continue; // skip stray bytes
acc = (acc << 6) | static_cast<std::uint32_t>(v);
bits += 6;
if (bits >= 8) {
bits -= 8;
out.push_back(static_cast<char>((acc >> bits) & 0xFF));
}
}
return out;
}
std::string decode_rfc2047(std::string_view in, bool *had_encoded_word) {
if (had_encoded_word)
*had_encoded_word = false;
std::string out;
out.reserve(in.size());
std::size_t i = 0;
const std::size_t n = in.size();
while (i < n) {
auto start = in.find("=?", i);
if (start == std::string_view::npos) {
out.append(in.substr(i));
break;
}
out.append(in.substr(i, start - i));
// =?charset?enc?text?=
auto q1 = in.find('?', start + 2);
if (q1 == std::string_view::npos) {
out.append(in.substr(start));
break;
}
auto q2 = in.find('?', q1 + 1);
if (q2 == std::string_view::npos || q2 != q1 + 2) {
out.append("=?");
i = start + 2;
continue;
}
auto end = in.find("?=", q2 + 1);
if (end == std::string_view::npos) {
out.append(in.substr(start));
break;
}
std::string_view charset = in.substr(start + 2, q1 - (start + 2));
char enc = in[q1 + 1];
std::string_view text = in.substr(q2 + 1, end - (q2 + 1));
std::string bytes;
if (enc == 'B' || enc == 'b') {
bytes = base64_decode(text);
} else if (enc == 'Q' || enc == 'q') {
for (std::size_t k = 0; k < text.size(); ++k) {
char c = text[k];
if (c == '_') {
bytes.push_back(' ');
} else if (c == '=' && k + 2 < text.size()) {
int hi = hex_val(text[k + 1]);
int lo = hex_val(text[k + 2]);
if (hi >= 0 && lo >= 0) {
bytes.push_back(static_cast<char>((hi << 4) | lo));
k += 2;
} else {
bytes.push_back(c);
}
} else {
bytes.push_back(c);
}
}
} else {
out.append(in.substr(start, end + 2 - start)); // unknown encoding: verbatim
i = end + 2;
continue;
}
if (is_utf8(charset))
out.append(to_utf8_best_effort(bytes));
else if (is_latin1(charset))
out.append(latin1_to_utf8(bytes));
else
out.append(to_utf8_best_effort(bytes));
if (had_encoded_word)
*had_encoded_word = true;
i = end + 2;
// RFC 2047: whitespace between adjacent encoded words is elided.
std::size_t j = i;
while (j < n && (in[j] == ' ' || in[j] == '\t'))
++j;
if (j < n && in.compare(j, 2, "=?") == 0)
i = j;
}
return out;
}
} // namespace vdm::net::detail
+40
View File
@@ -0,0 +1,40 @@
// vdm/net/text_codec.hpp — internal: small byte/text codecs for header parsing.
//
// Not a public header. Everything here is pure, allocation-bounded, and total (no throw,
// no assert on input): the inputs come off the wire from untrusted servers.
#ifndef VDM_NET_TEXT_CODEC_HPP
#define VDM_NET_TEXT_CODEC_HPP
#include <cstdint>
#include <string>
#include <string_view>
namespace vdm::net::detail {
// Percent-decode ("%XX"). A stray '%' or a non-hex digit after it is emitted literally.
// `plus_as_space` handles application/x-www-form-urlencoded style; off for URL paths.
[[nodiscard]] std::string percent_decode(std::string_view in, bool plus_as_space = false);
// True if `s` is well-formed UTF-8 (no overlong forms, no surrogates, no > U+10FFFF).
[[nodiscard]] bool is_valid_utf8(std::string_view s) noexcept;
// Reinterpret each byte as a Latin-1 (ISO-8859-1) code point and re-encode as UTF-8.
[[nodiscard]] std::string latin1_to_utf8(std::string_view s);
// Decode standard base64 (RFC 4648, '+' '/', optional '=' padding). Whitespace is
// skipped. Invalid trailing bits are dropped. Returns the decoded bytes.
[[nodiscard]] std::string base64_decode(std::string_view in);
// Decode RFC 2047 "encoded-word" runs: =?charset?B?..?= / =?charset?Q?..?=. Text outside
// encoded words is passed through. Only UTF-8 and ISO-8859-1/Latin-1 charsets are
// transcoded; anything else is passed through as-is (best effort). `had_encoded_word`
// reports whether at least one well-formed word was found.
[[nodiscard]] std::string decode_rfc2047(std::string_view in, bool *had_encoded_word = nullptr);
// If `s` is valid UTF-8, return it unchanged; otherwise treat it as Latin-1 and transcode.
[[nodiscard]] std::string to_utf8_best_effort(std::string_view s);
} // namespace vdm::net::detail
#endif // VDM_NET_TEXT_CODEC_HPP
+126
View File
@@ -0,0 +1,126 @@
// vdm/net/url.cpp
#include "vdm/net/url.hpp"
#include <charconv>
#include "net/text_codec.hpp"
namespace vdm::net {
namespace {
std::string ascii_lower(std::string_view s) {
std::string r(s);
for (char &c : r)
if (c >= 'A' && c <= 'Z')
c = static_cast<char>(c - 'A' + 'a');
return r;
}
bool valid_scheme(std::string_view s) {
if (s.empty())
return false;
if (!((s[0] >= 'a' && s[0] <= 'z') || (s[0] >= 'A' && s[0] <= 'Z')))
return false;
for (char c : s) {
bool ok = (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') ||
c == '+' || c == '-' || c == '.';
if (!ok)
return false;
}
return true;
}
} // namespace
SplitUrl split_url(std::string_view url) {
SplitUrl out;
auto scheme_end = url.find("://");
if (scheme_end == std::string_view::npos)
return out;
std::string_view scheme = url.substr(0, scheme_end);
if (!valid_scheme(scheme))
return out;
out.scheme = ascii_lower(scheme);
std::string_view rest = url.substr(scheme_end + 3);
// authority ends at the first '/', '?' or '#'
std::size_t auth_end = rest.size();
for (std::size_t i = 0; i < rest.size(); ++i) {
char c = rest[i];
if (c == '/' || c == '?' || c == '#') {
auth_end = i;
break;
}
}
std::string_view authority = rest.substr(0, auth_end);
std::string_view tail = rest.substr(auth_end);
if (auto at = authority.rfind('@'); at != std::string_view::npos) {
out.userinfo.assign(authority.substr(0, at));
authority = authority.substr(at + 1);
}
std::string_view host = authority;
std::string_view port;
if (!authority.empty() && authority.front() == '[') {
auto close = authority.find(']');
if (close != std::string_view::npos) {
host = authority.substr(1, close - 1); // strip brackets
if (close + 1 < authority.size() && authority[close + 1] == ':')
port = authority.substr(close + 2);
}
} else if (auto colon = authority.rfind(':'); colon != std::string_view::npos) {
host = authority.substr(0, colon);
port = authority.substr(colon + 1);
}
out.host = ascii_lower(host);
if (!port.empty()) {
unsigned v = 0;
auto [p, ec] = std::from_chars(port.data(), port.data() + port.size(), v);
(void)p;
if (ec == std::errc{} && v > 0 && v <= 65535)
out.port = static_cast<std::uint16_t>(v);
}
// tail = path [ '?' query ] [ '#' fragment ]
std::string_view path_and_rest = tail;
if (auto hash = path_and_rest.find('#'); hash != std::string_view::npos) {
out.fragment.assign(path_and_rest.substr(hash + 1));
path_and_rest = path_and_rest.substr(0, hash);
}
if (auto q = path_and_rest.find('?'); q != std::string_view::npos) {
out.query.assign(path_and_rest.substr(q + 1));
path_and_rest = path_and_rest.substr(0, q);
}
out.path.assign(path_and_rest);
out.valid = !out.host.empty() && out.is_http();
return out;
}
std::string url_filename(std::string_view url) {
SplitUrl u = split_url(url);
std::string_view path = u.path;
if (path.empty())
return {};
auto slash = path.find_last_of('/');
std::string_view seg = (slash == std::string_view::npos) ? path : path.substr(slash + 1);
if (seg.empty())
return {};
std::string name = detail::percent_decode(seg);
// Guard against a decoded segment that reintroduces a separator or NUL.
for (char &c : name)
if (c == '/' || c == '\\' || c == '\0')
c = '_';
if (name == "." || name == "..")
return {};
return name;
}
} // namespace vdm::net
View File
+40
View File
@@ -0,0 +1,40 @@
#include "vdm/rules/collision.hpp"
#include <string>
#include <string_view>
namespace vdm::rules {
namespace {
// Split on the last '.', unless it's a leading dot (a dotfile like ".gitignore" has no
// extension by this convention — matches sanitize_filename's own treatment). Returns
// {stem, ext} where `ext` includes the leading '.' when present.
std::pair<std::string_view, std::string_view> split_stem_ext(std::string_view name) {
auto dot = name.rfind('.');
if (dot == std::string_view::npos || dot == 0)
return {name, {}};
return {name.substr(0, dot), name.substr(dot)};
}
} // namespace
std::string resolve_collision(std::string_view desired,
const std::function<bool(std::string_view)> &exists,
CollisionPolicy policy, int max_attempts) {
if (policy == CollisionPolicy::overwrite)
return std::string(desired);
if (!exists || !exists(desired))
return std::string(desired);
auto [stem, ext] = split_stem_ext(desired);
std::string candidate;
for (int n = 1; n <= max_attempts; ++n) {
candidate = std::string(stem) + " (" + std::to_string(n) + ")" + std::string(ext);
if (!exists(candidate))
return candidate;
}
return candidate; // still colliding; see the header comment on why this isn't hidden
}
} // namespace vdm::rules
+127
View File
@@ -0,0 +1,127 @@
#include "vdm/rules/filename.hpp"
#include <algorithm>
#include <array>
#include <cctype>
#include <string>
#include <string_view>
namespace vdm::rules {
namespace {
bool is_control_byte(unsigned char c) { return c < 0x20 || c == 0x7F; }
bool is_ntfs_illegal(char c) {
switch (c) {
case '<':
case '>':
case ':':
case '"':
case '|':
case '?':
case '*':
return true;
default:
return false;
}
}
// CON, PRN, AUX, NUL, COM1-9, LPT1-9 — Win32 device names, matched case-insensitively.
// `stem` is already ASCII-only by the time this runs (everything else has been filtered),
// so a byte-wise toupper is enough; no locale, no UTF-8 concerns.
bool is_reserved_device_name(std::string_view stem) {
static constexpr std::array<std::string_view, 4> kFixed = {"CON", "PRN", "AUX", "NUL"};
std::string upper(stem);
std::transform(upper.begin(), upper.end(), upper.begin(),
[](unsigned char c) { return static_cast<char>(std::toupper(c)); });
for (auto f : kFixed)
if (upper == f)
return true;
if (upper.size() == 4 && (upper.starts_with("COM") || upper.starts_with("LPT")) &&
upper[3] >= '1' && upper[3] <= '9')
return true;
return false;
}
// Backs `pos` up off any UTF-8 continuation bytes (10xxxxxx) so a byte-length cut never
// splits a multi-byte codepoint. `pos` is a candidate cut index into `s`, 0 <= pos <=
// s.size().
std::size_t utf8_safe_cut(std::string_view s, std::size_t pos) {
while (pos > 0 && (static_cast<unsigned char>(s[pos]) & 0xC0) == 0x80)
--pos;
return pos;
}
} // namespace
std::string sanitize_filename(std::string_view raw, std::size_t max_bytes) {
// Pass 1: drop control bytes and path separators outright; fold the other
// NTFS-illegal characters to '_'. Everything else (including non-ASCII UTF-8) passes
// through untouched.
std::string s;
s.reserve(raw.size());
for (char c : raw) {
auto uc = static_cast<unsigned char>(c);
if (is_control_byte(uc) || c == '/' || c == '\\')
continue;
s.push_back(is_ntfs_illegal(c) ? '_' : c);
}
// Pass 2: collapse any run of 2+ '.' down to one, so stripped separators can't
// reconstitute a ".." (or longer) traversal-looking sequence out of what's left.
{
std::string collapsed;
collapsed.reserve(s.size());
for (std::size_t i = 0; i < s.size(); ++i) {
if (s[i] == '.' && i > 0 && collapsed.size() > 0 && collapsed.back() == '.')
continue;
collapsed.push_back(s[i]);
}
s = std::move(collapsed);
}
// Pass 3: strip trailing '.' and ' ' — both are silently dropped by the Win32 API, so
// leaving them lets two different requested names collide invisibly on export.
while (!s.empty() && (s.back() == '.' || s.back() == ' '))
s.pop_back();
if (s.empty())
s = "download";
// Pass 4: reserved device name check, against the part before the first '.' (or the
// whole name if there's none). The '_' goes right after the stem, before any
// extension ("NUL.txt" -> "NUL_.txt"), so the result still looks like the same kind of
// file rather than growing a spurious trailing character after its extension.
{
auto dot = s.find('.');
std::string_view stem = dot != std::string::npos ? std::string_view(s).substr(0, dot)
: std::string_view(s);
if (is_reserved_device_name(stem))
s.insert(stem.size(), "_");
}
// Pass 5: clamp to max_bytes, UTF-8-safe, keeping a short trailing extension intact
// where possible.
if (s.size() > max_bytes) {
std::string_view ext;
if (auto dot = s.rfind('.'); dot != std::string::npos && dot > 0 && s.size() - dot <= 16)
ext = std::string_view(s).substr(dot);
if (ext.size() < max_bytes) {
std::size_t stem_budget = max_bytes - ext.size();
std::size_t cut = utf8_safe_cut(s, stem_budget);
s = s.substr(0, cut) + std::string(ext);
} else {
s = s.substr(0, utf8_safe_cut(s, max_bytes));
}
// Re-strip: truncation can expose a new trailing '.'/' ' (e.g. the byte right
// before the cut was itself a dot that pass 3 had no reason to touch).
while (!s.empty() && (s.back() == '.' || s.back() == ' '))
s.pop_back();
if (s.empty())
s = "download";
}
return s;
}
} // namespace vdm::rules
+98
View File
@@ -0,0 +1,98 @@
#include "vdm/rules/match.hpp"
#include <algorithm>
#include <cctype>
#include <cstddef>
#include <numeric>
namespace vdm::rules {
namespace {
char lower_ascii(char c) { return static_cast<char>(std::tolower(static_cast<unsigned char>(c))); }
bool ieq(std::string_view a, std::string_view b) {
return a.size() == b.size() &&
std::equal(a.begin(), a.end(), b.begin(),
[](char x, char y) { return lower_ascii(x) == lower_ascii(y); });
}
// A rule's `extensions` entries are documented with no leading '.', but be lenient about
// one showing up anyway (a hand-edited rule table, an older client) rather than let a
// clause that never matches silently swallow a whole category.
std::string_view strip_leading_dot(std::string_view s) {
return (!s.empty() && s.front() == '.') ? s.substr(1) : s;
}
bool match_clause(const RuleMatch &m, const MatchInput &in) {
if (m.extensions) {
bool any = std::any_of(m.extensions->begin(), m.extensions->end(), [&](const auto &e) {
return ieq(strip_leading_dot(e), in.extension);
});
if (!any)
return false;
}
if (m.mime_types) {
bool any = std::any_of(m.mime_types->begin(), m.mime_types->end(),
[&](const auto &t) { return ieq(t, in.mime_type); });
if (!any)
return false;
}
if (m.host_pattern && !glob_match(*m.host_pattern, in.host))
return false;
if (m.url_pattern && !glob_match(*m.url_pattern, in.url))
return false;
if (m.min_size_bytes) {
if (!in.size_bytes || *in.size_bytes < *m.min_size_bytes)
return false;
}
if (m.max_size_bytes) {
if (!in.size_bytes || *in.size_bytes > *m.max_size_bytes)
return false;
}
return true;
}
} // namespace
bool glob_match(std::string_view pattern, std::string_view text) noexcept {
// Classic iterative wildcard match (single backtrack point at the most recent '*'), not
// the naive recursive version — bounded work on any input, including a pattern that is
// nothing but repeated '*'s against a long `text`.
std::size_t p = 0, t = 0;
std::size_t star = std::string_view::npos, mark = 0;
while (t < text.size()) {
if (p < pattern.size() && (pattern[p] == '?' || lower_ascii(pattern[p]) == lower_ascii(text[t]))) {
++p;
++t;
} else if (p < pattern.size() && pattern[p] == '*') {
star = p++;
mark = t;
} else if (star != std::string_view::npos) {
p = star + 1;
t = ++mark;
} else {
return false;
}
}
while (p < pattern.size() && pattern[p] == '*') ++p;
return p == pattern.size();
}
std::optional<RuleAction> match_rules(const std::vector<Rule> &rules, const MatchInput &input) {
std::vector<std::size_t> order(rules.size());
std::iota(order.begin(), order.end(), 0);
// Stable by construction: std::stable_sort keeps table order among equal priorities.
std::stable_sort(order.begin(), order.end(), [&](std::size_t a, std::size_t b) {
return rules[a].priority < rules[b].priority;
});
for (auto i : order) {
const Rule &r = rules[i];
if (!r.enabled)
continue;
if (match_clause(r.match, input))
return r.action;
}
return std::nullopt;
}
} // namespace vdm::rules
+343
View File
@@ -0,0 +1,343 @@
// vdm/segment/budget.cpp
#include "vdm/segment/budget.hpp"
#include <algorithm>
namespace vdm::segment {
SegmentBudget::SegmentBudget() : SegmentBudget(Options{}) {}
SegmentBudget::SegmentBudget(Options opts)
: max_active_(opts.max_active_segments ? opts.max_active_segments : 1),
notify_period_(opts.notify_period) {
notifier_ = std::jthread([this](std::stop_token st) { notifier_loop(st); });
}
SegmentBudget::~SegmentBudget() {
notifier_.request_stop();
notify_cv_.notify_all();
}
// --- allocation -----------------------------------------------------------------------
std::uint32_t SegmentBudget::effective_cap_locked(const Task &t) const {
std::uint32_t base = t.resumable ? t.per_task_cap : 1;
base = std::clamp<std::uint32_t>(base, 1, 32);
if (auto it = host_caps_.find(t.host); it != host_caps_.end() && it->second > 0)
base = std::min(base, it->second);
return base;
}
SegmentBudget::EngineBudget SegmentBudget::snapshot_locked() const {
std::uint32_t starved = 0;
for (const auto &[id, t] : tasks_)
if (t.want >= 1 && t.held == 0)
++starved;
return EngineBudget{max_active_, active_, starved};
}
// DAEMON's list first, then any registered task not in it (defensive; "a running task
// absent from the list sorts last"). Shared by reallocate_locked() and
// wake_one_waiter_locked(), which need the same priority ordering.
std::vector<TaskId> SegmentBudget::priority_order_locked() const {
std::vector<TaskId> order;
order.reserve(tasks_.size());
for (TaskId id : order_)
if (tasks_.count(id))
order.push_back(id);
for (const auto &[id, _] : tasks_)
if (std::find(order.begin(), order.end(), id) == order.end())
order.push_back(id);
return order;
}
// The two-pass fairness allocation. Recomputes every task's target from scratch (so a
// live cap cut naturally produces target < held -> yield), diffs against the last
// published target, and collects the callbacks to fire once mu_ is released.
SegmentBudget::Plan SegmentBudget::reallocate_locked() {
std::vector<TaskId> order = priority_order_locked();
std::unordered_map<TaskId, std::uint32_t> target;
target.reserve(order.size());
std::uint32_t pool = max_active_;
auto capped_want = [&](TaskId id) {
const Task &t = tasks_.at(id);
return std::min(t.want, effective_cap_locked(t));
};
// Guarantee pass: one slot each, in priority order, to anyone who wants one.
for (TaskId id : order) {
if (pool == 0)
break;
if (capped_want(id) >= 1) {
target[id] = 1;
--pool;
}
}
// Growth pass: round-robin the remainder, up to each task's effective cap.
while (pool > 0) {
bool granted = false;
for (TaskId id : order) {
if (pool == 0)
break;
std::uint32_t &tv = target[id];
if (tv < capped_want(id)) {
++tv;
--pool;
granted = true;
}
}
if (!granted)
break;
}
const SteadyTime now = std::chrono::steady_clock::now();
Plan plan;
for (auto &[id, t] : tasks_) {
std::uint32_t nt = target.count(id) ? target[id] : 0;
if (nt != t.target) {
t.target = nt;
if (t.on_target)
plan.targets.emplace_back(t.on_target, nt);
}
// starvation timestamp bookkeeping
bool starved_now = t.want >= 1 && t.held == 0;
if (starved_now && !t.starved_since)
t.starved_since = now;
if (!starved_now)
t.starved_since.reset();
}
EngineBudget eb = snapshot_locked();
bool starved_edge = (eb.tasks_starved == 0) != (last_starved_ == 0);
if (eb != last_notified_)
dirty_ = true;
last_starved_ = eb.tasks_starved;
if (starved_edge && on_changed_) {
plan.notify_now = std::make_pair(on_changed_, eb);
last_notified_ = eb;
dirty_ = false;
}
if (dirty_)
notify_cv_.notify_one();
return plan;
}
void SegmentBudget::run(Plan &p) {
for (auto &[fn, n] : p.targets)
if (fn)
fn(n);
if (p.notify_now && p.notify_now->first)
p.notify_now->first(p.notify_now->second);
}
// --- task-facing --------------------------------------------------------------------
void SegmentBudget::register_task(TaskId id, const TaskParams &params, SlotTargetFn on_target) {
Plan plan;
{
std::lock_guard lk(mu_);
Task t;
t.host = params.host;
t.per_task_cap = params.per_task_cap ? params.per_task_cap : 1;
t.resumable = params.resumable;
t.on_target = std::move(on_target);
tasks_[id] = std::move(t);
plan = reallocate_locked();
}
run(plan);
}
void SegmentBudget::deregister_task(TaskId id) {
Plan plan;
{
std::lock_guard lk(mu_);
auto it = tasks_.find(id);
if (it == tasks_.end())
return;
active_ -= it->second.held;
tasks_.erase(it);
plan = reallocate_locked();
wake_one_waiter_locked(plan, id); // a departing task frees real slots too
}
run(plan);
}
void SegmentBudget::set_want(TaskId id, std::uint32_t want) {
Plan plan;
{
std::lock_guard lk(mu_);
auto it = tasks_.find(id);
if (it == tasks_.end())
return;
if (it->second.want == want)
return;
it->second.want = want;
plan = reallocate_locked();
}
run(plan);
}
bool SegmentBudget::confirm_slot(TaskId id) {
std::lock_guard lk(mu_);
auto it = tasks_.find(id);
if (it == tasks_.end())
return false;
Task &t = it->second;
if (t.held >= t.target)
return false; // target was cut in the race
// reallocate_locked()'s pool math bounds sum(target) <= max_active_ *as computed*, but
// that doesn't bound sum(held): a task can be legitimately over its own just-lowered
// target for a while (yield deferred to a segment boundary, ADR 0011 A1), and another
// task's target can correctly rise to claim that capacity before the first task has
// physically released it. active_ is the one number that's always true regardless of
// any task's target bookkeeping, so it's the backstop. Denials here are remembered
// (waiting_for_slot) rather than left for the caller to somehow ask again at the right
// moment -- see release_slot()'s wake_one_waiter_locked() call.
if (active_ >= max_active_) {
t.waiting_for_slot = true;
return false;
}
t.waiting_for_slot = false;
++t.held;
++active_;
if (snapshot_locked() != last_notified_) {
dirty_ = true;
notify_cv_.notify_one();
}
return true;
}
// Appends a retry hint for the highest-priority task with waiting_for_slot set (other
// than `exclude`) to `plan`. Only ever wakes a task confirm_slot() actually turned away --
// not just anyone below its target, which would also fire for tasks that are fairly,
// correctly not entitled to more right now (see reallocate_locked()'s own target math).
void SegmentBudget::wake_one_waiter_locked(Plan &plan, TaskId exclude) const {
for (TaskId id : priority_order_locked()) {
if (id == exclude)
continue;
auto it = tasks_.find(id);
if (it == tasks_.end())
continue;
const Task &t = it->second;
if (t.waiting_for_slot && t.on_target) {
plan.targets.emplace_back(t.on_target, t.target);
return; // exactly one freed slot, exactly one retry hint
}
}
}
void SegmentBudget::release_slot(TaskId id) {
Plan plan;
{
std::lock_guard lk(mu_);
auto it = tasks_.find(id);
if (it == tasks_.end() || it->second.held == 0)
return;
--it->second.held;
--active_;
plan = reallocate_locked();
// reallocate_locked() only fires a callback for a task whose *target* changed.
// The task this freed slot is actually owed to (see confirm_slot()) may have a
// target that was already correct and hasn't moved -- nothing else will ever ask
// it to retry, so the budget has to remember and hand this slot to it directly.
wake_one_waiter_locked(plan, id);
}
run(plan);
}
// --- DAEMON-facing ---------------------------------------------------------------------
void SegmentBudget::set_max_active_segments(std::uint32_t n) {
Plan plan;
{
std::lock_guard lk(mu_);
n = n ? n : 1;
if (n == max_active_)
return;
max_active_ = n;
plan = reallocate_locked();
}
run(plan);
}
void SegmentBudget::set_host_segment_cap(std::string host, std::uint32_t cap) {
Plan plan;
{
std::lock_guard lk(mu_);
if (cap == 0)
host_caps_.erase(host);
else
host_caps_[std::move(host)] = cap;
plan = reallocate_locked();
}
run(plan);
}
void SegmentBudget::set_task_order(std::span<const TaskId> priority_order) {
Plan plan;
{
std::lock_guard lk(mu_);
order_.assign(priority_order.begin(), priority_order.end());
plan = reallocate_locked();
}
run(plan);
}
SegmentBudget::EngineBudget SegmentBudget::budget() const {
std::lock_guard lk(mu_);
return snapshot_locked();
}
std::uint32_t SegmentBudget::segments_active(TaskId id) const {
std::lock_guard lk(mu_);
auto it = tasks_.find(id);
return it == tasks_.end() ? 0 : it->second.held;
}
std::vector<TaskId> SegmentBudget::starved_tasks() const {
std::lock_guard lk(mu_);
std::vector<TaskId> out;
for (const auto &[id, t] : tasks_)
if (t.want >= 1 && t.held == 0)
out.push_back(id);
return out;
}
std::optional<SteadyTime> SegmentBudget::starved_since(TaskId id) const {
std::lock_guard lk(mu_);
auto it = tasks_.find(id);
return it == tasks_.end() ? std::nullopt : it->second.starved_since;
}
void SegmentBudget::on_budget_changed(std::function<void(EngineBudget)> cb) {
std::lock_guard lk(mu_);
on_changed_ = std::move(cb);
}
// --- notifier thread: coalesced <=4 Hz -----------------------------------------------
void SegmentBudget::notifier_loop(std::stop_token st) {
std::unique_lock lk(mu_);
while (!st.stop_requested()) {
notify_cv_.wait_for(lk, notify_period_, [&] { return dirty_ || st.stop_requested(); });
if (st.stop_requested())
break;
if (!dirty_)
continue;
EngineBudget eb = snapshot_locked();
auto cb = on_changed_;
last_notified_ = eb;
last_starved_ = eb.tasks_starved;
dirty_ = false;
lk.unlock();
if (cb)
cb(eb);
lk.lock();
}
}
} // namespace vdm::segment
+321
View File
@@ -0,0 +1,321 @@
// vdm/segment/segmenter.cpp
#include "vdm/segment/segmenter.hpp"
#include <algorithm>
#include <limits>
namespace vdm::segment {
namespace {
constexpr std::uint32_t kNoIndex = std::numeric_limits<std::uint32_t>::max();
constexpr std::uint64_t kU64Max = std::numeric_limits<std::uint64_t>::max();
bool is_live(SegState s) noexcept {
return s == SegState::idle || s == SegState::connecting || s == SegState::downloading ||
s == SegState::stalled;
}
} // namespace
// Seg holds std::atomics, so it is neither copyable nor movable — every insertion is an
// emplace_back that constructs it in place, followed by stores. This helper centralises
// that. Caller holds mu_.
std::uint32_t Segmenter::add_seg_locked(std::uint64_t start, std::uint64_t end,
std::uint64_t completed, SegState state, bool assigned) {
segs_.emplace_back(next_index_++, start, end);
Seg &s = segs_.back();
s.completed.store(completed);
s.state.store(state);
s.assigned = assigned;
return s.index;
}
// --- construction ---------------------------------------------------------------------
Segmenter::Segmenter(std::uint64_t total_size, std::uint32_t requested_segments, bool resumable,
std::uint64_t min_segment_bytes)
: total_size_(total_size),
min_seg_(min_segment_bytes ? min_segment_bytes : 1),
resumable_(resumable) {
compute_target(requested_segments);
}
Segmenter::Segmenter(std::uint64_t total_size, std::uint32_t requested_segments,
const std::vector<ResumedRange> &resumed, bool resumable,
std::uint64_t min_segment_bytes)
: total_size_(total_size),
min_seg_(min_segment_bytes ? min_segment_bytes : 1),
resumable_(resumable) {
compute_target(requested_segments);
// Validate the resumed table tiles [0, total_size) exactly.
bool ok = resumable_ && total_size_ > 0 && !resumed.empty();
if (ok) {
std::vector<ResumedRange> sorted = resumed;
std::sort(sorted.begin(), sorted.end(),
[](const auto &a, const auto &b) { return a.start < b.start; });
std::uint64_t cursor = 0;
for (const auto &r : sorted) {
if (r.start != cursor || r.end < r.start || r.completed > r.end - r.start + 1) {
ok = false;
break;
}
cursor = r.end + 1;
}
if (ok && cursor != total_size_)
ok = false;
if (ok) {
for (const auto &r : sorted) {
bool done = r.completed == r.end - r.start + 1;
add_seg_locked(r.start, r.end, r.completed,
done ? SegState::complete : SegState::idle, false);
}
std::uint32_t incomplete = 0;
for (const auto &s : segs_)
if (s.state.load() != SegState::complete)
++incomplete;
target_count_ = std::clamp<std::uint32_t>(std::max(incomplete, 1u), 1, kMaxSegments);
return;
}
}
// Fall back to a fresh single/target layout (segs created lazily by assign_slot()).
segs_.clear();
}
void Segmenter::compute_target(std::uint32_t requested) {
if (!resumable_ || total_size_ == 0) {
target_count_ = 1;
return;
}
std::uint64_t by_size = total_size_ / min_seg_;
if (by_size == 0)
by_size = 1;
std::uint64_t t = requested == 0 ? kDefaultSegments : requested;
t = std::min<std::uint64_t>(t, by_size);
target_count_ = std::clamp<std::uint32_t>(static_cast<std::uint32_t>(t), 1, kMaxSegments);
}
// --- helpers (mu_ held) --------------------------------------------------------------
std::uint64_t Segmenter::remaining_of_locked(const Seg &s) const noexcept {
std::uint64_t end = s.end.load();
std::uint64_t done = s.start + s.completed.load();
return done > end ? 0 : end - done + 1;
}
std::uint32_t Segmenter::assigned_count_locked() const noexcept {
std::uint32_t n = 0;
for (const auto &s : segs_)
if (s.assigned)
++n;
return n;
}
// Split the largest remaining range; hand back its second half as a new segment.
std::uint32_t Segmenter::split_largest_remaining_locked() {
Seg *victim = nullptr;
std::uint64_t best = 0;
for (auto &s : segs_) {
if (!s.assigned || !is_live(s.state.load()))
continue;
std::uint64_t rem = remaining_of_locked(s);
if (rem > best) {
best = rem;
victim = &s;
}
}
if (!victim || best < 2 * min_seg_)
return kNoIndex;
const std::uint64_t v_end = victim->end.load();
const std::uint64_t half = best / 2; // >= min_seg_ since best >= 2*min
const std::uint64_t mid = v_end - half; // victim keeps [start, mid]
const std::uint64_t cur = victim->start + victim->completed.load();
if (mid < cur || mid - cur + 1 < min_seg_)
return kNoIndex; // victim would be too small
victim->end.store(mid); // the victim's worker reads end before each write and stops here
return add_seg_locked(mid + 1, v_end, 0, SegState::idle, false);
}
// --- public: structural (take the lock) --------------------------------------------
std::optional<std::uint32_t> Segmenter::assign_slot() {
std::lock_guard lk(mu_);
if (!orphans_.empty()) {
ResumedRange o = orphans_.front();
orphans_.erase(orphans_.begin());
return add_seg_locked(o.start, o.end, o.completed, SegState::connecting, true);
}
if (assigned_count_locked() >= target_count_)
return std::nullopt;
if (segs_.empty()) {
std::uint64_t end = total_size_ > 0 ? total_size_ - 1 : kU64Max - 1;
return add_seg_locked(0, end, 0, SegState::connecting, true);
}
// Some resumed segments may be unassigned idle ranges — hand one out before splitting.
for (auto &s : segs_) {
if (!s.assigned && s.state.load() == SegState::idle) {
s.assigned = true;
s.state.store(SegState::connecting);
return s.index;
}
}
std::uint32_t idx = split_largest_remaining_locked();
if (idx == kNoIndex)
return std::nullopt;
segs_[idx].assigned = true;
segs_[idx].state.store(SegState::connecting);
return idx;
}
std::optional<std::uint32_t> Segmenter::on_complete(std::uint32_t idx, bool may_steal) {
std::lock_guard lk(mu_);
if (idx >= segs_.size())
return std::nullopt;
Seg &seg = segs_[idx];
seg.completed.store(seg.end.load() - seg.start + 1);
seg.state.store(SegState::complete);
seg.assigned = false;
if (!may_steal)
return std::nullopt; // yielding the slot
if (!orphans_.empty()) {
ResumedRange o = orphans_.front();
orphans_.erase(orphans_.begin());
return add_seg_locked(o.start, o.end, o.completed, SegState::connecting, true);
}
std::uint32_t new_idx = split_largest_remaining_locked();
if (new_idx == kNoIndex)
return std::nullopt; // nothing to steal -> release the slot
segs_[new_idx].assigned = true;
segs_[new_idx].state.store(SegState::connecting);
return new_idx;
}
FailAction Segmenter::on_failed(std::uint32_t idx, bool connection_error, bool has_mirror) {
std::lock_guard lk(mu_);
if (idx >= segs_.size())
return FailAction::retry;
Seg &seg = segs_[idx];
++seg.consecutive_failures;
if (connection_error && seg.consecutive_failures >= 3 && has_mirror) {
std::uint64_t cur = seg.start + seg.completed.load();
std::uint64_t end = seg.end.load();
if (cur <= end)
orphans_.push_back({cur, end, 0});
seg.state.store(SegState::failed);
seg.assigned = false;
return FailAction::requeue;
}
return FailAction::retry;
}
void Segmenter::note_connected(std::uint32_t idx) {
std::lock_guard lk(mu_);
if (idx < segs_.size())
segs_[idx].consecutive_failures = 0;
}
// --- public: per-worker accessors ----------------------------------------------
//
// These take mu_. They are called from the write path once per buffer flush (a few per
// second per segment), not from the curl write callback — the no-lock/no-alloc rule is
// about that callback and its ring buffer, not about progress bookkeeping. The segment
// fields are still std::atomic so a reader that already holds a stable reference sees a
// torn-free value, and so the deque element type is safe to relocate-free.
void Segmenter::advance(std::uint32_t idx, std::uint64_t bytes) noexcept {
std::lock_guard lk(mu_);
if (idx >= segs_.size())
return;
Seg &s = segs_[idx];
std::uint64_t len = s.end.load() - s.start + 1;
s.completed.store(bytes < len ? bytes : len);
}
std::uint64_t Segmenter::segment_start(std::uint32_t idx) const noexcept {
std::lock_guard lk(mu_);
return idx < segs_.size() ? segs_[idx].start : 0;
}
std::uint64_t Segmenter::segment_end(std::uint32_t idx) const noexcept {
std::lock_guard lk(mu_);
return idx < segs_.size() ? segs_[idx].end.load() : 0;
}
std::uint64_t Segmenter::segment_completed(std::uint32_t idx) const noexcept {
std::lock_guard lk(mu_);
return idx < segs_.size() ? segs_[idx].completed.load() : 0;
}
SegState Segmenter::segment_state(std::uint32_t idx) const noexcept {
std::lock_guard lk(mu_);
return idx < segs_.size() ? segs_[idx].state.load() : SegState::failed;
}
void Segmenter::set_segment_state(std::uint32_t idx, SegState st) noexcept {
std::lock_guard lk(mu_);
if (idx < segs_.size())
segs_[idx].state.store(st);
}
void Segmenter::release_segment(std::uint32_t idx) noexcept {
std::lock_guard lk(mu_);
if (idx >= segs_.size())
return;
segs_[idx].assigned = false;
if (segs_[idx].state.load() != SegState::complete)
segs_[idx].state.store(SegState::idle);
}
// --- public: queries (take the lock) ----------------------------------------------
std::uint64_t Segmenter::downloaded() const {
std::lock_guard lk(mu_);
std::uint64_t sum = 0;
for (const auto &s : segs_)
sum += s.completed.load();
return sum;
}
bool Segmenter::all_complete() const {
std::lock_guard lk(mu_);
if (segs_.empty())
return false;
if (total_size_ == 0)
return segs_.front().state.load() == SegState::complete;
if (!orphans_.empty())
return false;
std::vector<std::pair<std::uint64_t, std::uint64_t>> done; // [start, start+completed)
for (const auto &s : segs_) {
std::uint64_t c = s.completed.load();
if (c > 0)
done.emplace_back(s.start, s.start + c);
}
std::sort(done.begin(), done.end());
std::uint64_t cursor = 0;
for (auto [a, b] : done) {
if (a > cursor)
return false; // gap
if (b > cursor)
cursor = b;
}
return cursor >= total_size_;
}
std::vector<SegmentView> Segmenter::snapshot() const {
std::lock_guard lk(mu_);
std::vector<SegmentView> out;
out.reserve(segs_.size());
for (const auto &s : segs_)
out.push_back(SegmentView{s.index, s.start, s.end.load(), s.completed.load(),
s.state.load(), s.consecutive_failures});
return out;
}
} // namespace vdm::segment
+89
View File
@@ -0,0 +1,89 @@
// vdm/task/digest.cpp
#include "task/digest.hpp"
#include <fcntl.h>
#include <unistd.h>
#include <array>
#include <cerrno>
#include <cstring>
#include <openssl/evp.h>
namespace vdm::task {
namespace {
const EVP_MD *md_for(Checksum::Algo a) {
switch (a) {
case Checksum::Algo::md5:
return EVP_md5();
case Checksum::Algo::sha1:
return EVP_sha1();
case Checksum::Algo::sha256:
return EVP_sha256();
case Checksum::Algo::sha512:
return EVP_sha512();
}
return EVP_sha256();
}
std::string to_hex(const unsigned char *p, unsigned n) {
static const char *h = "0123456789abcdef";
std::string s;
s.reserve(n * 2);
for (unsigned i = 0; i < n; ++i) {
s.push_back(h[p[i] >> 4]);
s.push_back(h[p[i] & 0xF]);
}
return s;
}
} // namespace
Result<std::string> hash_file(std::string_view path, Checksum::Algo algo) {
std::string p(path);
int fd = ::open(p.c_str(), O_RDONLY | O_CLOEXEC);
if (fd < 0)
return ErrorInfo(Error::path_rejected,
std::string("open ") + p + ": " + std::strerror(errno));
EVP_MD_CTX *ctx = EVP_MD_CTX_new();
if (!ctx) {
::close(fd);
return ErrorInfo(Error::internal, "EVP_MD_CTX_new");
}
auto fail = [&](Error e, std::string msg) {
EVP_MD_CTX_free(ctx);
::close(fd);
return Result<std::string>(ErrorInfo(e, std::move(msg)));
};
if (EVP_DigestInit_ex(ctx, md_for(algo), nullptr) != 1)
return fail(Error::internal, "EVP_DigestInit_ex");
std::array<unsigned char, 256 * 1024> buf{};
for (;;) {
ssize_t n = ::read(fd, buf.data(), buf.size());
if (n < 0) {
if (errno == EINTR)
continue;
return fail(Error::io_error, std::string("read: ") + std::strerror(errno));
}
if (n == 0)
break;
if (EVP_DigestUpdate(ctx, buf.data(), static_cast<std::size_t>(n)) != 1)
return fail(Error::internal, "EVP_DigestUpdate");
}
unsigned char out[EVP_MAX_MD_SIZE];
unsigned out_len = 0;
if (EVP_DigestFinal_ex(ctx, out, &out_len) != 1)
return fail(Error::internal, "EVP_DigestFinal_ex");
EVP_MD_CTX_free(ctx);
::close(fd);
return to_hex(out, out_len);
}
} // namespace vdm::task
+20
View File
@@ -0,0 +1,20 @@
// vdm/task/digest.hpp — internal: hash a finished file for checksum verification.
#ifndef VDM_TASK_DIGEST_HPP
#define VDM_TASK_DIGEST_HPP
#include <string>
#include <string_view>
#include "vdm/task/download.hpp"
#include "vdm/util/result.hpp"
namespace vdm::task {
// Stream `path` through the digest and return it lower-case hex. io_error on a read
// failure, path_rejected if the file can't be opened.
[[nodiscard]] Result<std::string> hash_file(std::string_view path, Checksum::Algo algo);
} // namespace vdm::task
#endif // VDM_TASK_DIGEST_HPP
File diff suppressed because it is too large Load Diff
+58
View File
@@ -0,0 +1,58 @@
// vdm/task/download_task.hpp — internal: the task machine behind DownloadHandle, and the
// narrow interface it uses to reach engine-owned resources (so this TU doesn't depend on
// Engine::Impl).
#ifndef VDM_TASK_DOWNLOAD_TASK_HPP
#define VDM_TASK_DOWNLOAD_TASK_HPP
#include <cstdint>
#include <functional>
#include <memory>
#include "vdm/engine.hpp"
#include "vdm/ids.hpp"
#include "vdm/net/http_client.hpp"
#include "vdm/net/probe.hpp"
#include "vdm/rate/token_bucket.hpp"
#include "vdm/segment/budget.hpp"
#include "vdm/task/download.hpp"
namespace vdm::task {
using TimerId = std::uint64_t;
// Implemented by Engine::Impl. Every method is safe to call from any thread.
struct TaskHost {
virtual ~TaskHost() = default;
virtual net::HttpClient &http() = 0;
virtual segment::SegmentBudget &budget() = 0;
virtual rate::RateLimiter &limiter() = 0;
virtual const Engine::Config &config() = 0;
// One-shot timer. `fn` runs on the engine's timer thread. cancel_timer is a no-op if
// it already fired or never existed.
virtual TimerId schedule(SteadyTime at, std::function<void()> fn) = 0;
virtual void cancel_timer(TimerId id) = 0;
// Probe pool, outside the segment budget (ADR 0011 §5).
virtual void probe(net::ProbeRequest req,
std::function<void(Result<net::ProbeResult>)> done) = 0;
// The task reached a terminal state — drop it from the engine's registry.
virtual void task_retired(TaskId id) = 0;
};
// Create a task and begin it (probe or connect). The returned control block is what
// DownloadHandle wraps (DownloadHandle{state}); the engine keeps its own copy so the task
// outlives a caller that drops its handle.
[[nodiscard]] std::shared_ptr<DownloadTaskState> create_task(TaskHost &host, TaskId id,
DownloadSpec spec,
DownloadCallbacks cbs);
// Engine shutdown: stop every transfer and fire no further callbacks. Safe on nullptr.
void quiesce_task(const std::shared_ptr<DownloadTaskState> &s);
} // namespace vdm::task
#endif // VDM_TASK_DOWNLOAD_TASK_HPP
+33 -8
View File
@@ -23,14 +23,39 @@ vdm_add_test(veloxcore_log_test util/log_test.cpp)
# net/ integration tests drive tools/testserver (lane PKG/QA). Skip cleanly if it isn't
# in the tree yet (lanes merge independently).
vdm_add_test(veloxcore_content_disposition_test net/content_disposition_test.cpp)
vdm_add_test(veloxcore_url_test net/url_test.cpp)
vdm_add_test(veloxcore_sparse_file_test io/sparse_file_test.cpp)
vdm_add_test(veloxcore_write_buffer_test io/write_buffer_test.cpp)
vdm_add_test(veloxcore_veloxpart_test meta/veloxpart_test.cpp)
vdm_add_test(veloxcore_segmenter_test segment/segmenter_test.cpp)
vdm_add_test(veloxcore_budget_test segment/budget_test.cpp)
vdm_add_test(veloxcore_engine_api_test task/api_compiles_test.cpp)
vdm_add_test(veloxcore_token_bucket_test rate/token_bucket_test.cpp)
set(_testserver ${CMAKE_SOURCE_DIR}/tools/testserver/testserver.py)
vdm_add_test(veloxcore_http_client_test net/http_client_test.cpp)
target_include_directories(veloxcore_http_client_test PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/net)
vdm_add_test(veloxcore_engine_test task/engine_test.cpp)
target_include_directories(veloxcore_engine_test PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/net ${CMAKE_SOURCE_DIR}/core/src)
if(EXISTS ${_testserver})
target_compile_definitions(veloxcore_http_client_test
PRIVATE VDM_TESTSERVER_PY="${_testserver}")
set_tests_properties(veloxcore_http_client_test PROPERTIES TIMEOUT 120)
else()
message(STATUS "veloxcore: tools/testserver not present; http_client_test will skip "
"its server-backed cases.")
target_compile_definitions(veloxcore_engine_test PRIVATE VDM_TESTSERVER_PY="${_testserver}")
set_tests_properties(veloxcore_engine_test PROPERTIES TIMEOUT 300)
endif()
foreach(net_it http_client probe)
vdm_add_test(veloxcore_${net_it}_test net/${net_it}_test.cpp)
target_include_directories(veloxcore_${net_it}_test
PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/net)
if(EXISTS ${_testserver})
target_compile_definitions(veloxcore_${net_it}_test
PRIVATE VDM_TESTSERVER_PY="${_testserver}")
set_tests_properties(veloxcore_${net_it}_test PROPERTIES TIMEOUT 120)
endif()
endforeach()
if(NOT EXISTS ${_testserver})
message(STATUS "veloxcore: tools/testserver not present; net integration tests will "
"skip their server-backed cases.")
endif()
vdm_add_test(veloxcore_rules_filename_test rules/filename_test.cpp)
vdm_add_test(veloxcore_rules_collision_test rules/collision_test.cpp)
vdm_add_test(veloxcore_rules_match_test rules/match_test.cpp)
+198
View File
@@ -0,0 +1,198 @@
#include "vdm/io/sparse_file.hpp"
#include <fcntl.h>
#include <unistd.h>
#include <array>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <string>
#include <thread>
#include <vector>
#include "vtest.hpp"
using vdm::Error;
using vdm::io::SparseFile;
namespace {
struct TempPath {
std::string path;
TempPath() {
const char *dir = std::getenv("TMPDIR");
path = (dir ? dir : "/tmp");
path += "/vdm_sparse_test_XXXXXX";
int fd = ::mkstemp(path.data());
if (fd >= 0) {
::close(fd);
::unlink(path.c_str()); // we only wanted a unique name
}
}
~TempPath() { ::unlink(path.c_str()); }
};
std::vector<std::byte> read_all(const std::string &path) {
int fd = ::open(path.c_str(), O_RDONLY);
if (fd < 0)
return {};
std::vector<std::byte> out;
std::array<std::byte, 4096> buf{};
for (;;) {
ssize_t n = ::read(fd, buf.data(), buf.size());
if (n <= 0)
break;
out.insert(out.end(), buf.begin(), buf.begin() + n);
}
::close(fd);
return out;
}
std::uint64_t file_size(const std::string &path) {
int fd = ::open(path.c_str(), O_RDONLY);
if (fd < 0)
return 0;
off_t end = ::lseek(fd, 0, SEEK_END);
::close(fd);
return end < 0 ? 0 : static_cast<std::uint64_t>(end);
}
vdm::ConstByteSpan bytes(const char *s) {
return {reinterpret_cast<const std::byte *>(s), std::strlen(s)};
}
} // namespace
VT_TEST(sparse_open_preallocates_full_size) {
TempPath tp;
SparseFile f;
auto r = f.open(tp.path, {.total_size = 1 << 20});
VT_REQUIRE(r.has_value());
VT_CHECK(f.is_open());
VT_CHECK_EQ(file_size(tp.path), 1u << 20);
// /tmp is usually a real fs; if it's tmpfs, preallocated() is false and that's fine.
VT_CHECK(f.close().has_value());
}
VT_TEST(sparse_write_at_absolute_offsets) {
TempPath tp;
SparseFile f;
VT_REQUIRE(f.open(tp.path, {.total_size = 64}).has_value());
VT_CHECK(f.write_at(10, bytes("hello")).has_value());
VT_CHECK(f.write_at(40, bytes("world")).has_value());
VT_CHECK(f.sync().has_value());
auto data = read_all(tp.path);
VT_REQUIRE(data.size() == 64);
VT_CHECK_EQ(std::memcmp(data.data() + 10, "hello", 5), 0);
VT_CHECK_EQ(std::memcmp(data.data() + 40, "world", 5), 0);
f.close().value();
}
VT_TEST(sparse_write_past_end_grows_file) {
TempPath tp;
SparseFile f;
VT_REQUIRE(f.open(tp.path, {.total_size = 16}).has_value());
VT_CHECK(f.write_at(1000, bytes("tail")).has_value());
VT_CHECK_EQ(file_size(tp.path), 1004u);
f.close().value();
}
VT_TEST(sparse_resize_trims) {
TempPath tp;
SparseFile f;
VT_REQUIRE(f.open(tp.path, {.total_size = 4096}).has_value());
VT_CHECK(f.resize(100).has_value());
VT_CHECK_EQ(file_size(tp.path), 100u);
f.close().value();
}
VT_TEST(sparse_open_bad_path_is_path_rejected) {
SparseFile f;
auto r = f.open("/vdm_no_such_dir_xyz/file.part", {.total_size = 10});
VT_REQUIRE(!r.has_value());
VT_CHECK_EQ(r.error().code, Error::path_rejected);
VT_CHECK(!f.is_open());
}
VT_TEST(sparse_symlinked_target_is_rejected) {
// A symlink swapped in as the final path component after DAEMON's canonicalise-and-check
// must not be followed: the open is O_NOFOLLOW, so it fails with ELOOP -> path_rejected
// rather than redirecting our writes through the link.
TempPath link; // the download target the caller hands us
TempPath target; // where the symlink points (would-be victim, outside allowed roots)
VT_REQUIRE(::symlink(target.path.c_str(), link.path.c_str()) == 0);
SparseFile f;
auto r = f.open(link.path, {.total_size = 4096});
VT_REQUIRE(!r.has_value());
VT_CHECK_EQ(r.error().code, Error::path_rejected);
VT_CHECK(!f.is_open());
// the link target was never created/written through
VT_CHECK_EQ(::access(target.path.c_str(), F_OK), -1);
}
VT_TEST(sparse_ops_on_closed_file_error) {
SparseFile f;
VT_CHECK_EQ(f.write_at(0, bytes("x")).error().code, Error::internal);
VT_CHECK_EQ(f.sync().error().code, Error::internal);
VT_CHECK(f.close().has_value()); // close on a closed file is ok
}
VT_TEST(sparse_advise_dontneed_is_safe) {
TempPath tp;
SparseFile f;
VT_REQUIRE(f.open(tp.path, {.total_size = 8192}).has_value());
VT_CHECK(f.write_at(0, bytes("data")).has_value());
VT_CHECK(f.sync().has_value());
f.advise_dontneed(0, 4096); // must not crash / must be a no-op-safe call
f.advise_dontneed(0, 0);
f.close().value();
}
VT_TEST(sparse_move_transfers_fd) {
TempPath tp;
SparseFile a;
VT_REQUIRE(a.open(tp.path, {.total_size = 32}).has_value());
SparseFile b = std::move(a);
VT_CHECK(!a.is_open());
VT_CHECK(b.is_open());
VT_CHECK(b.write_at(0, bytes("moved")).has_value());
b.close().value();
}
VT_TEST(sparse_concurrent_nonoverlapping_writes) {
TempPath tp;
SparseFile f;
constexpr int kSegs = 8;
constexpr std::size_t kSeg = 64 * 1024;
VT_REQUIRE(f.open(tp.path, {.total_size = kSegs * kSeg}).has_value());
std::vector<std::jthread> ts;
for (int s = 0; s < kSegs; ++s) {
ts.emplace_back([&, s] {
std::vector<std::byte> chunk(kSeg, static_cast<std::byte>('A' + s));
for (std::size_t off = 0; off < kSeg; off += 4096) {
auto r = f.write_at(static_cast<std::uint64_t>(s) * kSeg + off,
vdm::ConstByteSpan(chunk.data() + off, 4096));
if (!r.has_value())
VT_FAIL("concurrent write_at failed");
}
});
}
ts.clear(); // join
VT_CHECK(f.sync().has_value());
auto data = read_all(tp.path);
VT_REQUIRE(data.size() == kSegs * kSeg);
for (int s = 0; s < kSegs; ++s) {
bool ok = true;
for (std::size_t i = 0; i < kSeg; ++i)
if (data[s * kSeg + i] != static_cast<std::byte>('A' + s))
ok = false;
VT_CHECK(ok);
}
f.close().value();
}
+202
View File
@@ -0,0 +1,202 @@
#include "vdm/io/write_buffer.hpp"
#include <atomic>
#include <cstdlib>
#include <cstring>
#include <new>
#include <string>
#include <vector>
#include "vtest.hpp"
using vdm::ConstByteSpan;
using vdm::Error;
using vdm::Result;
using vdm::io::WriteBuffer;
// --- global allocation counter, for the "no alloc in append()" test -----------------
namespace {
std::atomic<long> g_alloc_calls{0};
std::atomic<bool> g_count_allocs{false};
} // namespace
void *operator new(std::size_t n) {
if (g_count_allocs.load(std::memory_order_relaxed))
g_alloc_calls.fetch_add(1, std::memory_order_relaxed);
void *p = std::malloc(n ? n : 1);
if (!p)
throw std::bad_alloc();
return p;
}
void operator delete(void *p) noexcept {
std::free(p);
}
void operator delete(void *p, std::size_t) noexcept {
std::free(p);
}
void *operator new[](std::size_t n) {
return ::operator new(n);
}
void operator delete[](void *p) noexcept {
std::free(p);
}
void operator delete[](void *p, std::size_t) noexcept {
std::free(p);
}
namespace {
// A flush sink that records (offset, bytes) and never allocates after construction.
struct Sink {
std::vector<std::byte> data; // pre-reserved
std::vector<std::uint64_t> offs; // pre-reserved
std::vector<std::size_t> lens;
bool fail_next = false;
WriteBuffer::FlushFn fn() {
return [this](std::uint64_t off, ConstByteSpan s) -> Result<void> {
if (fail_next) {
fail_next = false;
return vdm::Err{Error::io_error, "sink forced failure"};
}
offs.push_back(off);
lens.push_back(s.size());
data.insert(data.end(), s.begin(), s.end());
return vdm::ok();
};
}
};
ConstByteSpan sv(const char *s) {
return {reinterpret_cast<const std::byte *>(s), std::strlen(s)};
}
} // namespace
VT_TEST(wb_accumulates_then_flushes_on_fill) {
Sink sink;
sink.data.reserve(1 << 16);
sink.offs.reserve(64);
sink.lens.reserve(64);
WriteBuffer wb(0, 8, sink.fn());
VT_CHECK(wb.append(sv("abc")).has_value()); // 3 buffered
VT_CHECK_EQ(wb.pending(), 3u);
VT_CHECK(sink.offs.empty()); // no flush yet
VT_CHECK(wb.append(sv("defgh")).has_value()); // fills to 8 -> flush
VT_REQUIRE(sink.offs.size() == 1);
VT_CHECK_EQ(sink.offs[0], 0u);
VT_CHECK_EQ(sink.lens[0], 8u);
VT_CHECK_EQ(wb.pending(), 0u);
VT_CHECK_EQ(wb.next_offset(), 8u);
VT_CHECK(wb.append(sv("ij")).has_value());
VT_CHECK(wb.flush().has_value()); // explicit tail flush
VT_REQUIRE(sink.offs.size() == 2);
VT_CHECK_EQ(sink.offs[1], 8u);
VT_CHECK_EQ(sink.lens[1], 2u);
VT_CHECK_EQ(std::string(reinterpret_cast<const char *>(sink.data.data()), sink.data.size()),
std::string("abcdefghij"));
VT_CHECK_EQ(wb.total_appended(), 10u);
}
VT_TEST(wb_flush_is_noop_when_empty) {
Sink sink;
sink.offs.reserve(4);
WriteBuffer wb(100, 16, sink.fn());
VT_CHECK(wb.flush().has_value());
VT_CHECK(sink.offs.empty());
}
VT_TEST(wb_oversized_chunk_writes_through) {
Sink sink;
sink.data.reserve(1 << 16);
sink.offs.reserve(16);
sink.lens.reserve(16);
WriteBuffer wb(0, 8, sink.fn());
VT_CHECK(wb.append(sv("ab")).has_value()); // 2 buffered
// 20 bytes arriving: buffer isn't empty, so first 6 top it off + flush(8), then the
// remaining 14 (>= capacity, buffer now empty) write straight through.
std::string big(20, 'x');
VT_CHECK(wb.append(sv(big.c_str())).has_value());
VT_CHECK(wb.flush().has_value());
// reconstruct
std::string got(reinterpret_cast<const char *>(sink.data.data()), sink.data.size());
VT_CHECK_EQ(got, std::string("ab") + big);
VT_CHECK_EQ(wb.total_appended(), 22u);
// one full-buffer flush + one passthrough; order preserved
VT_CHECK(sink.offs.size() >= 2);
VT_CHECK_EQ(sink.offs.front(), 0u);
}
VT_TEST(wb_exact_capacity_chunk_from_empty_writes_through) {
Sink sink;
sink.data.reserve(64);
sink.offs.reserve(4);
sink.lens.reserve(4);
WriteBuffer wb(0, 4, sink.fn());
VT_CHECK(wb.append(sv("wxyz")).has_value()); // == capacity, empty -> passthrough
VT_REQUIRE(sink.offs.size() == 1);
VT_CHECK_EQ(sink.lens[0], 4u);
VT_CHECK_EQ(wb.pending(), 0u);
}
VT_TEST(wb_flush_error_propagates_without_advancing_durable_offset) {
Sink sink;
sink.data.reserve(64);
sink.offs.reserve(4);
sink.lens.reserve(4);
WriteBuffer wb(0, 8, sink.fn());
VT_CHECK(wb.append(sv("abc")).has_value()); // 3 buffered, nothing durable yet
VT_CHECK_EQ(wb.next_offset(), 0u);
sink.fail_next = true;
auto r = wb.flush(); // forced failure
VT_REQUIRE(!r.has_value());
VT_CHECK_EQ(r.error().code, Error::io_error);
VT_CHECK_EQ(wb.next_offset(), 0u); // durable offset did NOT move
VT_CHECK(sink.offs.empty());
// a retry flush succeeds and advances
VT_CHECK(wb.flush().has_value());
VT_CHECK_EQ(wb.next_offset(), 3u);
VT_REQUIRE(sink.lens.size() == 1);
VT_CHECK_EQ(sink.lens[0], 3u);
}
VT_TEST(wb_append_does_not_allocate) {
// flush sink that never allocates: just sum sizes.
std::atomic<std::uint64_t> total{0};
auto flush = [&total](std::uint64_t, ConstByteSpan s) -> Result<void> {
total.fetch_add(s.size());
return vdm::ok();
};
WriteBuffer wb(0, 4096, flush);
// Warm up (any first-call lazy init happens now, outside the measured window).
std::string warm(100, 'w');
(void)wb.append(sv(warm.c_str()));
(void)wb.flush();
std::vector<std::byte> chunk(512, std::byte{7});
g_alloc_calls.store(0);
g_count_allocs.store(true);
for (int i = 0; i < 5000; ++i) {
auto r = wb.append(ConstByteSpan(chunk.data(), 137 + (i % 200)));
if (!r.has_value()) {
g_count_allocs.store(false);
VT_FAIL("append failed");
return;
}
}
(void)wb.flush();
g_count_allocs.store(false);
VT_CHECK_EQ(g_alloc_calls.load(), 0L);
VT_CHECK(total.load() > 0);
}
+246
View File
@@ -0,0 +1,246 @@
#include "vdm/meta/veloxpart.hpp"
#include <unistd.h>
#include <cstdint>
#include <cstdlib>
#include <cstring>
#include <string>
#include <vector>
#include "vdm/util/crc32.hpp"
#include "vtest.hpp"
using namespace vdm;
using namespace vdm::meta;
namespace {
VeloxPart sample_full() {
VeloxPart vp;
vp.version = kVersion;
vp.total_size = 5'000'000'000ull;
vp.downloaded = 1'234'567;
vp.urls = {"https://origin.example/big.iso", "https://cdn.example/big.iso?sig=abc",
"https://mirror1.example/big.iso"};
vp.etag = "\"deadbeef-1234\"";
vp.last_modified = "Wed, 01 Jan 2025 00:00:00 GMT";
vp.content_type = "application/octet-stream";
for (int i = 0; i < 8; ++i) {
SegmentRecord s;
s.start = static_cast<std::uint64_t>(i) * 625'000'000ull;
s.end = s.start + 625'000'000ull - 1; // inclusive
s.completed = (i < 2) ? s.end - s.start + 1 : 100'000ull * (i + 1);
vp.segments.push_back(s);
}
vp.flags = kFlagHasShaState;
vp.sha256_state = {std::byte{1}, std::byte{2}, std::byte{3}, std::byte{0xAA}, std::byte{0xFF}};
return vp;
}
VeloxPart sample_minimal() {
VeloxPart vp;
vp.total_size = 0; // chunked / unknown
vp.urls = {"http://x/y"};
SegmentRecord s;
s.start = 0;
s.end = 0; // 1-byte resource
s.completed = 0;
vp.segments.push_back(s);
return vp;
}
// Re-CRC a mutated body (everything except the final u32).
std::vector<std::byte> refresh_crc(std::vector<std::byte> image) {
std::uint32_t c = crc32(ConstByteSpan(image.data(), image.size() - 4));
for (int i = 0; i < 4; ++i)
image[image.size() - 4 + i] = static_cast<std::byte>((c >> (8 * i)) & 0xFF);
return image;
}
struct TempPath {
std::string path;
TempPath() {
const char *d = std::getenv("TMPDIR");
path = (d ? d : "/tmp");
path += "/vdm_vp_test_XXXXXX";
int fd = ::mkstemp(path.data());
if (fd >= 0) {
::close(fd);
::unlink(path.c_str());
}
}
~TempPath() {
::unlink(path.c_str());
::unlink((path + ".tmp").c_str());
}
};
} // namespace
VT_TEST(crc32_known_vector) {
const char *s = "123456789";
VT_CHECK_EQ(crc32(ConstByteSpan(reinterpret_cast<const std::byte *>(s), 9)), 0xCBF43926u);
VT_CHECK_EQ(crc32(ConstByteSpan{}), 0u);
}
VT_TEST(vp_roundtrip_full) {
VeloxPart in = sample_full();
auto image = serialize_veloxpart(in);
auto out = parse_veloxpart(ConstByteSpan(image.data(), image.size()));
VT_REQUIRE(out.has_value());
VT_CHECK(out.value() == in);
VT_CHECK_EQ(out.value().effective_url(),
std::string_view("https://cdn.example/big.iso?sig=abc"));
VT_CHECK_EQ(out.value().segments.size(), 8u);
}
VT_TEST(vp_roundtrip_minimal) {
VeloxPart in = sample_minimal();
auto image = serialize_veloxpart(in);
auto out = parse_veloxpart(ConstByteSpan(image.data(), image.size()));
VT_REQUIRE(out.has_value());
VT_CHECK(out.value() == in);
VT_CHECK(out.value().sha256_state.empty());
}
VT_TEST(vp_serialize_is_deterministic) {
VeloxPart in = sample_full();
auto a = serialize_veloxpart(in);
auto b = serialize_veloxpart(in);
VT_CHECK(a == b);
}
VT_TEST(vp_file_roundtrip) {
TempPath tp;
VeloxPart in = sample_full();
VT_REQUIRE(write_veloxpart_file(tp.path, in, /*fsync=*/true).has_value());
auto out = read_veloxpart_file(tp.path);
VT_REQUIRE(out.has_value());
VT_CHECK(out.value() == in);
// the temp file must be gone after the atomic rename
VT_CHECK_EQ(::access((tp.path + ".tmp").c_str(), F_OK), -1);
}
VT_TEST(vp_read_missing_file_errors) {
auto out = read_veloxpart_file("/vdm_no_such_dir_zz/x.veloxpart.meta");
VT_REQUIRE(!out.has_value());
VT_CHECK_EQ(out.error().code, Error::path_rejected);
}
// --- truncation / corruption table -------------------------------------------------
VT_TEST(vp_reject_empty_and_tiny) {
VT_CHECK_EQ(parse_veloxpart(ConstByteSpan{}).error().code, Error::meta_corrupt);
std::array<std::byte, 3> three{};
VT_CHECK_EQ(parse_veloxpart(ConstByteSpan(three.data(), 3)).error().code, Error::meta_corrupt);
std::array<std::byte, 51> almost{};
VT_CHECK_EQ(parse_veloxpart(ConstByteSpan(almost.data(), almost.size())).error().code,
Error::meta_corrupt);
}
VT_TEST(vp_reject_bad_magic) {
auto image = serialize_veloxpart(sample_minimal());
image[1] = std::byte{'X'}; // "VDMP" -> "VXMP"
image = refresh_crc(std::move(image)); // fix CRC so we're testing the magic check
auto r = parse_veloxpart(ConstByteSpan(image.data(), image.size()));
VT_REQUIRE(!r.has_value());
VT_CHECK_EQ(r.error().code, Error::meta_corrupt);
}
VT_TEST(vp_reject_crc_mismatch) {
auto image = serialize_veloxpart(sample_minimal());
image[20] ^= std::byte{0x40}; // flip a payload bit, do NOT refresh the CRC
auto r = parse_veloxpart(ConstByteSpan(image.data(), image.size()));
VT_REQUIRE(!r.has_value());
VT_CHECK_EQ(r.error().code, Error::meta_corrupt);
// flipping the CRC field itself is also a mismatch
auto image2 = serialize_veloxpart(sample_minimal());
image2.back() ^= std::byte{0xFF};
VT_CHECK_EQ(parse_veloxpart(ConstByteSpan(image2.data(), image2.size())).error().code,
Error::meta_corrupt);
}
VT_TEST(vp_future_version_is_unsupported) {
auto image = serialize_veloxpart(sample_minimal());
image[4] = std::byte{99}; // version u16 low byte
image[5] = std::byte{0};
image = refresh_crc(std::move(image));
auto r = parse_veloxpart(ConstByteSpan(image.data(), image.size()));
VT_REQUIRE(!r.has_value());
VT_CHECK_EQ(r.error().code, Error::meta_version_unsupported);
}
VT_TEST(vp_reject_truncation_at_every_stage) {
auto full = serialize_veloxpart(sample_full());
// Chop the image at many lengths; each must be meta_corrupt, never a crash.
for (std::size_t len = full.size() - 1; len >= 1; len = (len > 8 ? len - 7 : len - 1)) {
auto r = parse_veloxpart(ConstByteSpan(full.data(), len));
VT_CHECK(!r.has_value());
if (r.has_value())
break;
VT_CHECK_EQ(r.error().code, Error::meta_corrupt);
if (len == 1)
break;
}
}
VT_TEST(vp_reject_hostile_url_count) {
auto image = serialize_veloxpart(sample_minimal());
// url_count u32 sits right after magic(4)+ver(2)+flags(2)+total(8)+downloaded(8) = 24
for (int i = 0; i < 4; ++i)
image[24 + i] = std::byte{0xFF};
image = refresh_crc(std::move(image));
auto r = parse_veloxpart(ConstByteSpan(image.data(), image.size()));
VT_REQUIRE(!r.has_value());
VT_CHECK_EQ(r.error().code, Error::meta_corrupt);
}
VT_TEST(vp_reject_hostile_lp_string_length) {
// The case AGENT-CORE singles out: a length prefix that reaches past the buffer.
// Build a minimal image, then overwrite url[0]'s length prefix with a huge value.
auto image = serialize_veloxpart(sample_minimal());
// layout up to url[0] length: magic4 ver2 flags2 total8 downloaded8 url_count4 = 28
for (int i = 0; i < 4; ++i)
image[28 + i] = std::byte{0xFF}; // url[0] len = 4 GiB - 1
image = refresh_crc(std::move(image));
auto r = parse_veloxpart(ConstByteSpan(image.data(), image.size()));
VT_REQUIRE(!r.has_value());
VT_CHECK_EQ(r.error().code, Error::meta_corrupt);
}
VT_TEST(vp_reject_hostile_segment_count) {
auto image = serialize_veloxpart(sample_minimal());
// find the segment_count: it's u32 right before the (24-byte) segment records and the
// trailing crc. minimal sample has 1 segment and no sha state, so:
// segment_count is at size - 4(crc) - 24(one segment) - 4 = size - 32
std::size_t sc = image.size() - 32;
for (int i = 0; i < 4; ++i)
image[sc + i] = std::byte{0xFF};
image = refresh_crc(std::move(image));
auto r = parse_veloxpart(ConstByteSpan(image.data(), image.size()));
VT_REQUIRE(!r.has_value());
VT_CHECK_EQ(r.error().code, Error::meta_corrupt);
}
VT_TEST(vp_reject_trailing_bytes) {
auto image = serialize_veloxpart(sample_minimal());
image.push_back(std::byte{0});
image.push_back(std::byte{0});
image = refresh_crc(std::move(image)); // CRC now covers the padding too
auto r = parse_veloxpart(ConstByteSpan(image.data(), image.size()));
VT_REQUIRE(!r.has_value());
VT_CHECK_EQ(r.error().code, Error::meta_corrupt);
}
VT_TEST(vp_reject_segment_completed_over_length) {
VeloxPart in = sample_minimal();
in.segments[0].start = 0;
in.segments[0].end = 99; // length 100
in.segments[0].completed = 500; // impossible
auto image = serialize_veloxpart(in);
auto r = parse_veloxpart(ConstByteSpan(image.data(), image.size()));
VT_REQUIRE(!r.has_value());
VT_CHECK_EQ(r.error().code, Error::meta_corrupt);
}
+165
View File
@@ -0,0 +1,165 @@
#include "vdm/net/content_disposition.hpp"
#include <string>
#include <string_view>
#include "vtest.hpp"
using vdm::net::ContentDisposition;
using vdm::net::parse_content_disposition;
using Type = vdm::net::ContentDisposition::Type;
namespace {
// "€" is U+20AC -> UTF-8 E2 82 AC. "£" is U+00A3 -> UTF-8 C2 A3.
const std::string kEuro = "\xE2\x82\xAC";
const std::string kPound = "\xC2\xA3";
const std::string kEAcute = "\xC3\xA9"; // é U+00E9
} // namespace
VT_TEST(cd_plain_quoted) {
auto cd = parse_content_disposition(R"(attachment; filename="report.pdf")");
VT_CHECK(cd.type == Type::attachment);
VT_CHECK_EQ(cd.filename, std::string("report.pdf"));
VT_CHECK(!cd.filename_from_ext);
}
VT_TEST(cd_plain_token_unquoted) {
auto cd = parse_content_disposition("attachment; filename=report.pdf");
VT_CHECK_EQ(cd.filename, std::string("report.pdf"));
}
VT_TEST(cd_inline_no_filename) {
auto cd = parse_content_disposition("inline");
VT_CHECK(cd.type == Type::inline_);
VT_CHECK(!cd.has_filename());
}
VT_TEST(cd_rfc5987_utf8_ext_value) {
auto cd = parse_content_disposition("attachment; filename*=UTF-8''%e2%82%ac%20rates.pdf");
VT_CHECK_EQ(cd.filename, kEuro + " rates.pdf");
VT_CHECK(cd.filename_from_ext);
}
VT_TEST(cd_prefers_ext_over_plain) {
auto cd = parse_content_disposition(
R"(attachment; filename="EURO rates.pdf"; filename*=UTF-8''%e2%82%ac%20rates.pdf)");
VT_CHECK_EQ(cd.filename, kEuro + " rates.pdf");
VT_CHECK(cd.filename_from_ext);
}
VT_TEST(cd_rfc5987_latin1_ext_value) {
// %A3 = £ in ISO-8859-1
auto cd = parse_content_disposition("attachment; filename*=ISO-8859-1''%A3rates.pdf");
VT_CHECK_EQ(cd.filename, kPound + "rates.pdf");
}
VT_TEST(cd_legacy_rfc2047_base64) {
// base64("<euro> rates.pdf") with euro as UTF-8
// "€ rates.pdf" -> bytes E2 82 AC 20 72 61 74 65 73 2E 70 64 66 -> base64:
auto cd =
parse_content_disposition(R"(attachment; filename="=?UTF-8?B?4oKsIHJhdGVzLnBkZg==?=")");
VT_CHECK_EQ(cd.filename, kEuro + " rates.pdf");
}
VT_TEST(cd_legacy_rfc2047_qencoded_latin1) {
// =?ISO-8859-1?Q?=A3rates.pdf?= -> £rates.pdf
auto cd = parse_content_disposition(R"(attachment; filename="=?ISO-8859-1?Q?=A3rates.pdf?=")");
VT_CHECK_EQ(cd.filename, kPound + "rates.pdf");
}
VT_TEST(cd_raw_utf8_bytes_in_quotes) {
std::string h = "attachment; filename=\"caf" + kEAcute + ".txt\"";
auto cd = parse_content_disposition(h);
VT_CHECK_EQ(cd.filename, "caf" + kEAcute + ".txt");
}
VT_TEST(cd_raw_latin1_byte_in_quotes) {
// 0xE9 is 'é' in Latin-1; not valid UTF-8 alone -> transcoded
std::string h = "attachment; filename=\"caf\xE9.txt\"";
auto cd = parse_content_disposition(h);
VT_CHECK_EQ(cd.filename, "caf" + kEAcute + ".txt");
}
VT_TEST(cd_strips_path_components) {
VT_CHECK_EQ(parse_content_disposition(R"(attachment; filename="../../etc/passwd")").filename,
std::string("passwd"));
// real Windows paths in the wild use unescaped backslashes as separators
VT_CHECK_EQ(parse_content_disposition(R"(attachment; filename="C:\Windows\evil.exe")").filename,
std::string("evil.exe"));
// a base64 payload can contain '/', so decode must happen before path stripping
VT_CHECK_EQ(parse_content_disposition(R"(attachment; filename="=?UTF-8?B?Li4vLi4vc2VjcmV0?=")")
.filename,
std::string("secret")); // decodes to "../../secret", then stripped
}
VT_TEST(cd_quoted_dquote_escape) {
// \" is the one escape we resolve, so a quote can appear mid-name
auto cd = parse_content_disposition(R"(attachment; filename="quote\"here.txt")");
VT_CHECK_EQ(cd.filename, std::string("quote\"here.txt"));
}
VT_TEST(cd_semicolon_inside_quotes_is_not_a_separator) {
auto cd = parse_content_disposition(R"(attachment; filename="a;b;c.txt")");
VT_CHECK_EQ(cd.filename, std::string("a;b;c.txt"));
}
VT_TEST(cd_form_data) {
auto cd = parse_content_disposition(R"(form-data; name="file"; filename="upload.bin")");
VT_CHECK(cd.type == Type::form_data);
VT_CHECK_EQ(cd.filename, std::string("upload.bin"));
}
VT_TEST(cd_rfc2231_continuations) {
auto cd = parse_content_disposition(
"attachment; filename*0*=UTF-8''%e2%82%ac; filename*1*=%20rates; filename*2=.pdf");
VT_CHECK_EQ(cd.filename, kEuro + " rates.pdf");
}
VT_TEST(cd_empty_and_garbage_do_not_crash) {
VT_CHECK(!parse_content_disposition("").has_filename());
VT_CHECK(!parse_content_disposition(";;;;").has_filename());
VT_CHECK(!parse_content_disposition("attachment;").has_filename());
VT_CHECK(!parse_content_disposition(R"(attachment; filename=)").has_filename());
VT_CHECK(!parse_content_disposition(R"(attachment; filename="")").has_filename());
// truncated ext-value
auto cd = parse_content_disposition("attachment; filename*=UTF-8''%e2%82");
VT_CHECK(cd.type == Type::attachment); // no crash; filename is whatever fell out
// truncated encoded-word
auto trunc = parse_content_disposition(R"(attachment; filename="=?UTF-8?B?4oKs")");
(void)trunc;
}
VT_TEST(cd_bad_percent_escapes_in_ext_value) {
// stray % and non-hex digits are emitted literally, no crash
auto cd = parse_content_disposition("attachment; filename*=UTF-8''%ZZ%%file%2");
VT_CHECK(cd.type == Type::attachment);
}
VT_TEST(cd_strips_control_bytes_and_nul) {
// A mangled ext-value that decodes to bytes with embedded NULs (fuzz-found).
std::string h1("attachment; filename*=x''%e2%82%a");
h1.push_back('\0');
h1.push_back('\0');
h1 += "ff.pdf";
auto cd = parse_content_disposition(h1);
for (unsigned char c : cd.filename)
VT_CHECK(c >= 0x20 && c != 0x7F);
// a plain filename with a tab / newline / SOH loses them
std::string h2("attachment; filename=\"a\tb\nc");
h2.push_back('\x01');
h2 += ".txt\"";
auto cd2 = parse_content_disposition(h2);
VT_CHECK_EQ(cd2.filename, std::string("abc.txt"));
}
VT_TEST(cd_case_insensitive_keys_and_type) {
auto cd = parse_content_disposition(R"(ATTACHMENT; FileName="x.txt")");
VT_CHECK(cd.type == Type::attachment);
VT_CHECK_EQ(cd.filename, std::string("x.txt"));
}
VT_TEST(cd_unknown_type_is_other) {
auto cd = parse_content_disposition(R"(signal; filename="x.txt")");
VT_CHECK(cd.type == Type::other);
VT_CHECK_EQ(cd.filename, std::string("x.txt"));
}
+163
View File
@@ -0,0 +1,163 @@
#include "vdm/net/probe.hpp"
#include <chrono>
#include <future>
#include <string>
#include "testserver_fixture.hpp"
#include "vtest.hpp"
using namespace vdm;
using namespace vdm::net;
using vdm::testing::TestServer;
namespace {
Result<ProbeResult> run_probe(Prober &p, const std::string &url) {
std::promise<Result<ProbeResult>> prom;
auto fut = prom.get_future();
ProbeRequest req;
req.url = url;
req.overall_timeout_ms = 15000;
p.probe(std::move(req), [&](Result<ProbeResult> r) { prom.set_value(std::move(r)); });
if (fut.wait_for(std::chrono::seconds(25)) != std::future_status::ready)
return Err{Error::timeout, "probe test wait"};
return fut.get();
}
const std::string kEuro = "\xE2\x82\xAC";
} // namespace
VT_TEST(probe_plain_resumable_file) {
TestServer srv;
VT_REQUIRE(srv.available());
Prober p(4);
auto r = run_probe(p, srv.url("/plain/file/2M"));
VT_REQUIRE(r.has_value());
const auto &pr = r.value();
VT_CHECK_EQ(pr.http_status, 206); // proven via the ranged GET
VT_CHECK(pr.accept_ranges);
VT_CHECK(pr.resumable); // 206 + ETag validator
VT_REQUIRE(pr.total_size.has_value());
VT_CHECK_EQ(*pr.total_size, 2u * 1024 * 1024);
VT_CHECK(!pr.etag.empty());
VT_CHECK_EQ(pr.filename_from_url, std::string("2M"));
VT_CHECK(!pr.requires_auth);
}
VT_TEST(probe_no_range_server_is_not_resumable) {
TestServer srv;
VT_REQUIRE(srv.available());
Prober p(4);
auto r = run_probe(p, srv.url("/no-range/file/1M"));
VT_REQUIRE(r.has_value());
const auto &pr = r.value();
VT_CHECK(!pr.resumable); // no 206 ever
VT_CHECK(!pr.accept_ranges);
VT_REQUIRE(pr.total_size.has_value());
VT_CHECK_EQ(*pr.total_size, 1u * 1024 * 1024);
}
VT_TEST(probe_lying_accept_ranges_still_not_resumable) {
TestServer srv;
VT_REQUIRE(srv.available());
Prober p(4);
// advertises Accept-Ranges: bytes but never returns 206
auto r = run_probe(p, srv.url("/lies-about-accept-ranges/file/1M"));
VT_REQUIRE(r.has_value());
const auto &pr = r.value();
VT_CHECK(pr.accept_ranges); // it advertised
VT_CHECK(!pr.resumable); // ...but never proved it -> resume is off
}
VT_TEST(probe_reads_utf8_content_disposition) {
TestServer srv;
VT_REQUIRE(srv.available());
Prober p(4);
auto r = run_probe(p, srv.url("/utf8-content-disposition/file/8K"));
VT_REQUIRE(r.has_value());
VT_CHECK_EQ(r.value().filename_from_disposition, kEuro + " rates.pdf");
VT_CHECK_EQ(suggest_filename(r.value()), kEuro + " rates.pdf");
}
VT_TEST(probe_reads_legacy_content_disposition) {
TestServer srv;
VT_REQUIRE(srv.available());
Prober p(4);
auto r = run_probe(p, srv.url("/legacy-content-disposition/file/8K"));
VT_REQUIRE(r.has_value());
VT_CHECK(!r.value().filename_from_disposition.empty()); // decoded, not the raw =?...?=
VT_CHECK(r.value().filename_from_disposition.find("=?") == std::string::npos);
}
VT_TEST(probe_follows_redirect_and_reports_effective_url) {
TestServer srv;
VT_REQUIRE(srv.available());
Prober p(4);
auto r = run_probe(p, srv.url("/redirect-chain/file/64K"));
VT_REQUIRE(r.has_value());
const auto &pr = r.value();
VT_CHECK(pr.effective_url != srv.url("/redirect-chain/file/64K"));
VT_REQUIRE(pr.redirect_chain.size() >= 2);
VT_CHECK_EQ(pr.redirect_chain.front(), srv.url("/redirect-chain/file/64K"));
VT_CHECK_EQ(pr.redirect_chain.back(), pr.effective_url);
}
VT_TEST(probe_401_is_requires_auth_not_error) {
TestServer srv;
VT_REQUIRE(srv.available());
Prober p(4);
auto r = run_probe(p, srv.url("/401-basic/file/64K"));
VT_REQUIRE(r.has_value()); // success result...
VT_CHECK(r.value().requires_auth); // ...flagged for the credential dialog
}
VT_TEST(probe_404_is_an_error) {
TestServer srv;
VT_REQUIRE(srv.available());
Prober p(4);
auto r = run_probe(p, srv.url("/plain/nope"));
VT_REQUIRE(!r.has_value());
VT_CHECK_EQ(r.error().code, Error::not_found);
}
VT_TEST(probe_dead_host_is_connect_failed) {
Prober p(4);
std::promise<Result<ProbeResult>> prom;
auto fut = prom.get_future();
ProbeRequest req;
req.url = "http://127.0.0.1:1/x";
req.connect_timeout_ms = 2000;
p.probe(std::move(req), [&](Result<ProbeResult> r) { prom.set_value(std::move(r)); });
VT_REQUIRE(fut.wait_for(std::chrono::seconds(10)) == std::future_status::ready);
auto r = fut.get();
VT_REQUIRE(!r.has_value());
VT_CHECK_EQ(r.error().code, Error::connect_failed);
}
VT_TEST(probe_pool_serialises_excess_requests) {
TestServer srv;
VT_REQUIRE(srv.available());
Prober p(2); // only 2 at a time
constexpr int kN = 8;
std::vector<std::future<Result<ProbeResult>>> futs;
std::vector<std::promise<Result<ProbeResult>>> proms(kN);
for (int i = 0; i < kN; ++i) {
futs.push_back(proms[i].get_future());
ProbeRequest req;
req.url = srv.url("/plain/file/4K");
p.probe(std::move(req),
[pr = &proms[i]](Result<ProbeResult> r) { pr->set_value(std::move(r)); });
}
for (auto &f : futs) {
VT_REQUIRE(f.wait_for(std::chrono::seconds(30)) == std::future_status::ready);
VT_CHECK(f.get().has_value());
}
}
+10 -2
View File
@@ -28,7 +28,14 @@ namespace vdm::testing {
class TestServer {
public:
TestServer() {
TestServer() : TestServer(1.0) {}
// loris_seconds overrides the dribble duration slow-loris mode uses (default matches the
// no-arg ctor's long-standing 1s). A test that needs curl's stall detector
// (CURLOPT_LOW_SPEED_TIME, hardcoded to 30s in download_task.cpp) to actually fire needs a
// dribble that outlasts that threshold, not the short one every other test relies on to
// keep runtime down.
explicit TestServer(double loris_seconds) {
const char *script = VDM_TESTSERVER_PY;
if (!script || !*script || ::access(script, R_OK) != 0)
return;
@@ -50,8 +57,9 @@ class TestServer {
int devnull = ::open("/dev/null", O_WRONLY);
if (devnull >= 0)
::dup2(devnull, STDERR_FILENO);
std::string loris_str = std::to_string(loris_seconds);
::execlp("python3", "python3", script, "--port", "0", "--seed", "9", "--loris-seconds",
"1", "--throttle-bps", "131072", static_cast<char *>(nullptr));
loris_str.c_str(), "--throttle-bps", "131072", static_cast<char *>(nullptr));
::_exit(127);
}
::close(pipefd[1]);
+71
View File
@@ -0,0 +1,71 @@
#include "vdm/net/url.hpp"
#include <string>
#include "vtest.hpp"
using vdm::net::split_url;
using vdm::net::SplitUrl;
using vdm::net::url_filename;
VT_TEST(url_basic_https) {
auto u = split_url("https://example.com/path/to/file.iso?x=1#frag");
VT_CHECK(u.valid);
VT_CHECK_EQ(u.scheme, std::string("https"));
VT_CHECK_EQ(u.host, std::string("example.com"));
VT_CHECK(!u.port.has_value());
VT_CHECK_EQ(u.path, std::string("/path/to/file.iso"));
VT_CHECK_EQ(u.query, std::string("x=1"));
VT_CHECK_EQ(u.fragment, std::string("frag"));
}
VT_TEST(url_port_userinfo_lowercasing) {
auto u = split_url("HTTP://User:[email protected]:8080/a");
VT_CHECK_EQ(u.scheme, std::string("http"));
VT_CHECK_EQ(u.userinfo, std::string("User:pw"));
VT_CHECK_EQ(u.host, std::string("host.example.com"));
VT_REQUIRE(u.port.has_value());
VT_CHECK_EQ(*u.port, 8080);
}
VT_TEST(url_ipv6_host) {
auto u = split_url("http://[2001:db8::1]:9000/file");
VT_CHECK_EQ(u.host, std::string("2001:db8::1"));
VT_REQUIRE(u.port.has_value());
VT_CHECK_EQ(*u.port, 9000);
}
VT_TEST(url_no_path) {
auto u = split_url("https://example.com");
VT_CHECK(u.valid);
VT_CHECK_EQ(u.path, std::string(""));
}
VT_TEST(url_non_http_is_invalid_but_parsed) {
auto u = split_url("ftp://host/file");
VT_CHECK(!u.valid); // not http/https
VT_CHECK_EQ(u.scheme, std::string("ftp"));
VT_CHECK(!u.is_http());
}
VT_TEST(url_garbage_does_not_crash) {
VT_CHECK(!split_url("").valid);
VT_CHECK(!split_url("not a url").valid);
VT_CHECK(!split_url("://noscheme/x").valid);
VT_CHECK(!split_url("http://").valid);
auto a = split_url("http://////");
auto b = split_url("https://h/%%%/%");
(void)a;
(void)b;
}
VT_TEST(url_filename_extraction) {
VT_CHECK_EQ(url_filename("https://x.com/a/b/report%20final.pdf"),
std::string("report final.pdf"));
VT_CHECK_EQ(url_filename("https://x.com/a/b/file.iso?sig=abc"), std::string("file.iso"));
VT_CHECK_EQ(url_filename("https://x.com/dir/"), std::string(""));
VT_CHECK_EQ(url_filename("https://x.com"), std::string(""));
VT_CHECK_EQ(url_filename("https://x.com/%2e%2e"), std::string("")); // ".." rejected
VT_CHECK_EQ(url_filename("https://x.com/a%2Fb"),
std::string("a_b")); // decoded '/' neutralised
}
+145
View File
@@ -0,0 +1,145 @@
#include "vdm/rate/token_bucket.hpp"
#include <atomic>
#include <chrono>
#include <thread>
#include <vector>
#include "vtest.hpp"
using namespace vdm;
using namespace vdm::rate;
using namespace std::chrono_literals;
namespace {
TaskId tid(std::uint64_t v) {
return TaskId{v};
}
QueueId qid(std::uint64_t v) {
return QueueId{v};
}
} // namespace
VT_TEST(tb_unlimited_never_waits) {
TokenBucket b(0);
for (int i = 0; i < 1000; ++i)
VT_CHECK_EQ(b.consume(1'000'000).count(), 0);
}
VT_TEST(tb_burst_then_throttle) {
// 1000 B/s, default burst = 1 s = 1000 tokens.
TokenBucket b(1000);
VT_CHECK_EQ(b.consume(1000).count(), 0); // drains the burst
auto w = b.consume(1000); // empty now: must wait ~1 s
VT_CHECK(w >= 900ms && w <= 1100ms);
}
VT_TEST(tb_refills_over_time) {
TokenBucket b(10'000, /*burst=*/10'000);
VT_CHECK_EQ(b.consume(10'000).count(), 0);
std::this_thread::sleep_for(120ms); // ~1200 tokens back
auto w = b.consume(1000);
VT_CHECK_EQ(w.count(), 0); // affordable from the refill
auto w2 = b.consume(5000);
VT_CHECK(w2.count() > 0); // not that much yet
}
VT_TEST(tb_burst_caps_accumulation) {
TokenBucket b(1000, /*burst=*/2000);
std::this_thread::sleep_for(100ms); // idle far longer than burst/rate would fill
std::this_thread::sleep_for(100ms);
VT_CHECK_EQ(b.consume(2000).count(), 0); // at most the 2000 cap accumulated
VT_CHECK(b.consume(1).count() > 0); // and no more
}
VT_TEST(tb_set_rate_zero_makes_unlimited) {
TokenBucket b(1000);
VT_CHECK_EQ(b.consume(1000).count(), 0);
VT_CHECK(b.consume(1000).count() > 0);
b.set_rate(0);
VT_CHECK_EQ(b.consume(1'000'000).count(), 0);
}
// --- the hierarchy --------------------------------------------------------------------
VT_TEST(rl_all_unlimited_by_default) {
RateLimiter rl;
rl.attach_task(tid(1), std::nullopt);
for (int i = 0; i < 100; ++i)
VT_CHECK_EQ(rl.acquire(tid(1), 1'000'000).count(), 0);
}
VT_TEST(rl_tightest_level_binds) {
RateLimiter rl;
rl.set_global_limit(100'000);
rl.set_queue_limit(qid(9), 20'000);
rl.set_task_limit(tid(1), 50'000);
rl.attach_task(tid(1), qid(9));
// burst: task 50k, queue 20k, global 100k -> the queue's 20k is the ceiling
VT_CHECK_EQ(rl.acquire(tid(1), 20'000).count(), 0);
auto w = rl.acquire(tid(1), 5'000);
VT_CHECK(w.count() > 0); // queue bucket is dry even though task & global aren't
}
VT_TEST(rl_no_partial_consumption_on_miss) {
RateLimiter rl;
rl.set_global_limit(1'000'000); // plenty
rl.set_task_limit(tid(1), 1000); // 1 s burst
rl.attach_task(tid(1), std::nullopt);
VT_CHECK_EQ(rl.acquire(tid(1), 1000).count(), 0); // drain the task bucket
for (int i = 0; i < 5; ++i)
VT_CHECK(rl.acquire(tid(1), 1000).count() > 0); // task bucket blocks, repeatedly
// global must NOT have been charged for any of those blocked attempts: a fresh task
// limited only by the global bucket can still spend nearly its whole burst (only the
// one *successful* 1000-byte acquire above was charged).
rl.attach_task(tid(2), std::nullopt);
VT_CHECK_EQ(rl.acquire(tid(2), 990'000).count(), 0);
}
VT_TEST(rl_detach_then_acquire_is_safe_and_unlimited) {
RateLimiter rl;
rl.set_task_limit(tid(1), 1000);
rl.attach_task(tid(1), std::nullopt);
VT_CHECK_EQ(rl.acquire(tid(1), 1000).count(), 0);
rl.detach_task(tid(1));
// unknown task -> no task/queue bucket, only global (unlimited here)
VT_CHECK_EQ(rl.acquire(tid(1), 1'000'000).count(), 0);
}
VT_TEST(rl_enforces_aggregate_rate_under_load) {
RateLimiter rl;
const std::uint64_t rate = 4'000'000; // 4 MB/s global
rl.set_global_limit(rate);
for (std::uint64_t i = 1; i <= 8; ++i)
rl.attach_task(tid(i), std::nullopt);
std::atomic<std::uint64_t> moved{0};
auto t0 = std::chrono::steady_clock::now();
std::vector<std::jthread> ws;
for (std::uint64_t i = 1; i <= 8; ++i) {
ws.emplace_back([&, id = tid(i)] {
for (int k = 0; k < 400; ++k) {
std::uint64_t chunk = 16 * 1024;
for (;;) {
auto w = rl.acquire(id, chunk);
if (w.count() == 0)
break;
std::this_thread::sleep_for(
std::min<std::chrono::nanoseconds>(w, std::chrono::milliseconds(20)));
}
moved.fetch_add(chunk);
}
});
}
ws.clear(); // join
auto secs = std::chrono::duration<double>(std::chrono::steady_clock::now() - t0).count();
double effective = moved.load() / secs;
// Allow one burst's worth of slop plus scheduling noise: effective rate should sit
// within ~2x of the configured limit, never wildly above.
VT_CHECK(effective <= rate * 2.5);
VT_CHECK(moved.load() == 8u * 400u * 16u * 1024u);
}
+55
View File
@@ -0,0 +1,55 @@
#include "vdm/rules/collision.hpp"
#include <functional>
#include <set>
#include <string>
#include <string_view>
#include "vtest.hpp"
using vdm::rules::CollisionPolicy;
using vdm::rules::resolve_collision;
namespace {
std::function<bool(std::string_view)> exists_in(const std::set<std::string> &names) {
return [&names](std::string_view s) { return names.count(std::string(s)) > 0; };
}
} // namespace
VT_TEST(collision_no_collision_returns_desired) {
std::set<std::string> existing = {"other.txt"};
VT_CHECK_EQ(resolve_collision("file.txt", exists_in(existing)), std::string("file.txt"));
}
VT_TEST(collision_renames_on_conflict) {
std::set<std::string> existing = {"file.txt"};
VT_CHECK_EQ(resolve_collision("file.txt", exists_in(existing)), std::string("file (1).txt"));
}
VT_TEST(collision_finds_first_free_slot) {
std::set<std::string> existing = {"file.txt", "file (1).txt", "file (2).txt"};
VT_CHECK_EQ(resolve_collision("file.txt", exists_in(existing)), std::string("file (3).txt"));
}
VT_TEST(collision_no_extension) {
std::set<std::string> existing = {"README"};
VT_CHECK_EQ(resolve_collision("README", exists_in(existing)), std::string("README (1)"));
}
VT_TEST(collision_dotfile_treated_as_no_extension) {
std::set<std::string> existing = {".gitignore"};
VT_CHECK_EQ(resolve_collision(".gitignore", exists_in(existing)),
std::string(".gitignore (1)"));
}
VT_TEST(collision_overwrite_policy_ignores_existence) {
std::set<std::string> existing = {"file.txt"};
VT_CHECK_EQ(resolve_collision("file.txt", exists_in(existing), CollisionPolicy::overwrite),
std::string("file.txt"));
}
VT_TEST(collision_gives_up_after_max_attempts_without_fabricating) {
auto always_exists = [](std::string_view) { return true; };
auto out = resolve_collision("file.txt", always_exists, CollisionPolicy::rename, 3);
VT_CHECK_EQ(out, std::string("file (3).txt")); // last attempted, still colliding
}
+83
View File
@@ -0,0 +1,83 @@
#include "vdm/rules/filename.hpp"
#include <string>
#include "vtest.hpp"
using vdm::rules::sanitize_filename;
VT_TEST(filename_passthrough_when_already_clean) {
VT_CHECK_EQ(sanitize_filename("report.pdf"), std::string("report.pdf"));
}
VT_TEST(filename_strips_path_separators) {
VT_CHECK_EQ(sanitize_filename("a/b\\c.txt"), std::string("abc.txt"));
}
VT_TEST(filename_collapses_dotdot_after_separator_strip) {
// "../../etc/passwd" -> separators stripped, then the resulting ".." runs collapse to
// a single '.', which strip-trailing-dot then removes entirely.
auto out = sanitize_filename("../../etc/passwd");
VT_CHECK(out.find("..") == std::string::npos);
}
VT_TEST(filename_strips_control_bytes) {
std::string raw = "bad";
raw.push_back('\0');
raw += "name.txt";
auto out = sanitize_filename(raw);
VT_CHECK_EQ(out, std::string("badname.txt"));
}
VT_TEST(filename_replaces_ntfs_illegal_chars) {
VT_CHECK_EQ(sanitize_filename("a<b>c:d\"e|f?g*h.txt"), std::string("a_b_c_d_e_f_g_h.txt"));
}
VT_TEST(filename_strips_trailing_dot_and_space) {
VT_CHECK_EQ(sanitize_filename("name. "), std::string("name"));
}
VT_TEST(filename_empty_falls_back_to_download) {
VT_CHECK_EQ(sanitize_filename(""), std::string("download"));
}
VT_TEST(filename_all_stripped_falls_back_to_download) {
VT_CHECK_EQ(sanitize_filename("/\\"), std::string("download"));
}
VT_TEST(filename_reserved_device_name_bare) {
VT_CHECK_EQ(sanitize_filename("CON"), std::string("CON_"));
VT_CHECK_EQ(sanitize_filename("con"), std::string("con_"));
}
VT_TEST(filename_reserved_device_name_with_extension) {
VT_CHECK_EQ(sanitize_filename("NUL.txt"), std::string("NUL_.txt"));
VT_CHECK_EQ(sanitize_filename("com3.tar.gz"), std::string("com3_.tar.gz"));
}
VT_TEST(filename_reserved_device_name_not_a_false_positive) {
// "CONTEST" is not "CON" — must not get mangled.
VT_CHECK_EQ(sanitize_filename("CONTEST.txt"), std::string("CONTEST.txt"));
VT_CHECK_EQ(sanitize_filename("COM99.txt"), std::string("COM99.txt")); // not COM1-9
}
VT_TEST(filename_truncates_long_name_keeping_extension) {
std::string stem(500, 'a');
auto out = sanitize_filename(stem + ".txt", 255);
VT_CHECK(out.size() <= 255);
VT_CHECK(out.ends_with(".txt"));
}
VT_TEST(filename_truncation_is_utf8_safe) {
// Each "é" is 2 bytes (C3 A9); a 5-byte budget can fit 2 whole codepoints (4 bytes) but
// not a 3rd (needs 6) -- an unguarded byte-length cut at 5 would split the 3rd
// codepoint's C3 from its A9, leaving a dangling lead byte.
std::string stem;
for (int i = 0; i < 20; ++i) stem += "\xC3\xA9";
auto out = sanitize_filename(stem, 5);
VT_CHECK_EQ(out, std::string("\xC3\xA9\xC3\xA9")); // 2 whole codepoints, 4 bytes
}
VT_TEST(filename_preserves_non_ascii) {
VT_CHECK_EQ(sanitize_filename("caf\xC3\xA9.pdf"), std::string("caf\xC3\xA9.pdf"));
}
+192
View File
@@ -0,0 +1,192 @@
#include "vdm/rules/match.hpp"
#include <string>
#include "vtest.hpp"
using namespace vdm::rules;
namespace {
Rule make_rule(std::string id, std::int64_t priority, RuleMatch m, RuleAction a,
bool enabled = true) {
Rule r;
r.rule_id = std::move(id);
r.priority = priority;
r.enabled = enabled;
r.match = std::move(m);
r.action = std::move(a);
return r;
}
// -Wmissing-field-initializers (part of -Wextra) flags a designated-initializer list that
// skips a member, even one this repo's designated-init style would normally leave
// implicit -- these small builders keep the tests below readable without tripping it.
RuleAction action_with_category(std::string id) {
RuleAction a;
a.category_id = std::move(id);
return a;
}
MatchInput input_with_extension(std::string ext) {
MatchInput in;
in.extension = std::move(ext);
return in;
}
MatchInput input_with_host(std::string host) {
MatchInput in;
in.host = std::move(host);
return in;
}
MatchInput input_with_size(std::optional<std::uint64_t> size) {
MatchInput in;
in.size_bytes = size;
return in;
}
MatchInput input_with_extension_and_host(std::string ext, std::string host) {
MatchInput in;
in.extension = std::move(ext);
in.host = std::move(host);
return in;
}
} // namespace
// --- glob_match ---------------------------------------------------------------------
VT_TEST(glob_exact_match) {
VT_CHECK(glob_match("example.com", "example.com"));
VT_CHECK(!glob_match("example.com", "example.org"));
}
VT_TEST(glob_star_suffix) {
VT_CHECK(glob_match("*.example.com", "cdn.example.com"));
VT_CHECK(glob_match("*.example.com", "a.b.example.com"));
VT_CHECK(!glob_match("*.example.com", "example.com")); // no room for the literal '.'
}
VT_TEST(glob_star_matches_empty) {
VT_CHECK(glob_match("file*.zip", "file.zip"));
VT_CHECK(glob_match("file*.zip", "file123.zip"));
}
VT_TEST(glob_question_mark) {
VT_CHECK(glob_match("file?.txt", "file1.txt"));
VT_CHECK(!glob_match("file?.txt", "file.txt"));
VT_CHECK(!glob_match("file?.txt", "file12.txt"));
}
VT_TEST(glob_case_insensitive) {
VT_CHECK(glob_match("*.EXAMPLE.com", "cdn.example.COM"));
}
VT_TEST(glob_multiple_stars) {
VT_CHECK(glob_match("*foo*bar*", "xxfooyybarzz"));
VT_CHECK(!glob_match("*foo*bar*", "xxbarzzfooyy")); // order matters
}
VT_TEST(glob_pathological_stars_terminate) {
// A pattern of nothing but '*' against a long text must not blow up (bounded work).
std::string pattern(50, '*');
std::string text(10000, 'x');
VT_CHECK(glob_match(pattern, text));
}
// --- match_rules ---------------------------------------------------------------------
VT_TEST(match_empty_table_yields_nullopt) {
VT_CHECK(!match_rules({}, input_with_extension("zip")).has_value());
}
VT_TEST(match_no_clause_matches_everything) {
auto rules = {make_rule("r1", 0, RuleMatch{}, action_with_category("default"))};
auto r = match_rules(rules, input_with_extension("anything"));
VT_REQUIRE(r.has_value());
VT_CHECK_EQ(*r->category_id, std::string("default"));
}
VT_TEST(match_by_extension_case_insensitive) {
RuleMatch m;
m.extensions = std::vector<std::string>{"zip", "rar"};
auto rules = {make_rule("r1", 0, m, action_with_category("archives"))};
VT_CHECK(match_rules(rules, input_with_extension("ZIP")).has_value());
VT_CHECK(!match_rules(rules, input_with_extension("txt")).has_value());
}
VT_TEST(match_priority_order_lower_wins) {
std::vector<Rule> rules = {
make_rule("hi", 10, RuleMatch{}, action_with_category("first")),
make_rule("lo", 0, RuleMatch{}, action_with_category("second")),
};
auto r = match_rules(rules, MatchInput{});
VT_REQUIRE(r.has_value());
VT_CHECK_EQ(*r->category_id, std::string("second")); // priority 0 runs first
}
VT_TEST(match_ties_keep_table_order) {
std::vector<Rule> rules = {
make_rule("a", 5, RuleMatch{}, action_with_category("first")),
make_rule("b", 5, RuleMatch{}, action_with_category("second")),
};
auto r = match_rules(rules, MatchInput{});
VT_REQUIRE(r.has_value());
VT_CHECK_EQ(*r->category_id, std::string("first"));
}
VT_TEST(match_skips_disabled_rules) {
std::vector<Rule> rules = {
make_rule("a", 0, RuleMatch{}, action_with_category("disabled"), /*enabled=*/false),
make_rule("b", 1, RuleMatch{}, action_with_category("enabled")),
};
auto r = match_rules(rules, MatchInput{});
VT_REQUIRE(r.has_value());
VT_CHECK_EQ(*r->category_id, std::string("enabled"));
}
VT_TEST(match_host_pattern) {
RuleMatch m;
m.host_pattern = "*.cdn.example.com";
auto rules = {make_rule("r1", 0, m, action_with_category("cdn"))};
VT_CHECK(match_rules(rules, input_with_host("a.cdn.example.com")).has_value());
VT_CHECK(!match_rules(rules, input_with_host("example.com")).has_value());
}
VT_TEST(match_size_bounds) {
RuleMatch m;
m.min_size_bytes = 1000;
m.max_size_bytes = 2000;
auto rules = {make_rule("r1", 0, m, action_with_category("midsize"))};
VT_CHECK(match_rules(rules, input_with_size(1500)).has_value());
VT_CHECK(!match_rules(rules, input_with_size(500)).has_value());
VT_CHECK(!match_rules(rules, input_with_size(5000)).has_value());
}
VT_TEST(match_size_clause_with_unknown_size_does_not_match) {
RuleMatch m;
m.min_size_bytes = 1000;
auto rules = {make_rule("r1", 0, m, action_with_category("big"))};
// size_bytes left absent (pre-probe) -- a size clause must not match speculatively.
VT_CHECK(!match_rules(rules, MatchInput{}).has_value());
}
VT_TEST(match_all_clauses_must_hold) {
RuleMatch m;
m.extensions = std::vector<std::string>{"iso"};
m.host_pattern = "*.trusted.example";
auto rules = {make_rule("r1", 0, m, action_with_category("isos"))};
VT_CHECK(match_rules(rules, input_with_extension_and_host("iso", "mirror.trusted.example"))
.has_value());
// Extension matches but host doesn't -- must not match.
VT_CHECK(!match_rules(rules, input_with_extension_and_host("iso", "evil.example"))
.has_value());
}
VT_TEST(match_falls_through_to_default_when_nothing_matches) {
RuleMatch m;
m.extensions = std::vector<std::string>{"exe"};
auto rules = {make_rule("r1", 0, m, action_with_category("installers"))};
VT_CHECK(!match_rules(rules, input_with_extension("pdf")).has_value());
}
+514
View File
@@ -0,0 +1,514 @@
#include "vdm/segment/budget.hpp"
#include <atomic>
#include <chrono>
#include <condition_variable>
#include <functional>
#include <mutex>
#include <optional>
#include <thread>
#include <vector>
#include "vtest.hpp"
using namespace vdm;
using namespace vdm::segment;
using EB = SegmentBudget::EngineBudget;
namespace {
TaskId tid(std::uint64_t v) {
return TaskId{v};
}
// A test task that reacts to slot targets the way stage 8's download_task will: start
// workers up to the target, release them when the target drops. Purely bookkeeping.
//
// Production's real equivalent (register_task()'s on_target lambda, download_task.cpp)
// never calls back in synchronously -- it posts through host.schedule() (engine.cpp), so
// apply_slot_target() never runs on the same call stack as whatever budget call triggered
// it. wake_one_waiter_locked()'s wakeups, though, mean on_target() here *can* now be
// reentered on the same thread (e.g. this task's own release_slot() call, deep in the grow
// loop below, can cascade into waking a *different* task whose own release, in turn,
// cascades back to this one) -- a plain std::mutex would self-deadlock on that, the way
// FakeTask::mu almost did until this was written. A recursive_mutex plus coalescing the
// reentrant call's target into `pending` (processed by the outer call's loop once its
// current pass finishes) keeps this a faithful, deterministic stand-in without actually
// needing real threads: no callback ever runs nested inside a still-running instance of
// itself, and a burst of reentrant retargeting converges to the latest value instead of
// each one doing its own redundant grow/shrink pass.
struct FakeTask {
SegmentBudget *budget = nullptr;
TaskId id{};
std::recursive_mutex mu;
std::uint32_t workers = 0;
std::uint32_t target = 0;
bool running = false;
std::optional<std::uint32_t> pending;
FakeTask() = default;
FakeTask(SegmentBudget *b, TaskId i) : budget(b), id(i) {}
void on_target(std::uint32_t t) {
std::lock_guard lk(mu);
if (running) {
pending = t; // reentrant call on this thread -- the running instance picks it up
return;
}
running = true;
std::uint32_t want = t;
for (;;) {
target = want;
while (workers < target) {
if (!budget->confirm_slot(id))
break;
++workers;
}
// over target -> yield the excess immediately (a real task waits for a
// boundary)
while (workers > target) {
budget->release_slot(id);
--workers;
}
if (!pending)
break;
want = *pending;
pending.reset();
}
running = false;
}
std::uint32_t held() {
std::lock_guard lk(mu);
return workers;
}
};
// Mirrors Engine's real timer thread (engine.cpp: timer_loop) and how register_task()'s
// on_target lambda actually reaches a task (download_task.cpp: host.schedule(), never a
// direct call). One dedicated thread drains queued closures one at a time -- the
// serialization that makes production immune to the cross-task deadlock risk FakeTask's
// synchronous, whatever-thread-triggered-it delivery has under heavy *concurrent* driving
// (budget_concurrent_confirm_release_stays_consistent, below, is the one test that
// actually exercises this: multiple threads calling set_want() for *different* tasks at
// once, each able to cascade into a wake for another task's callback -- two such cascades
// landing on two different FakeTask mutexes in opposite orders on two different threads is
// a real AB-BA deadlock a recursive_mutex alone doesn't prevent, since that only guards a
// single thread against re-entering itself). Every other test in this file drives the
// budget from one thread at a time, where that risk can't arise, so plain FakeTask is
// still the simpler, sufficient double there.
class TestTimer {
public:
TestTimer() {
worker_ = std::jthread([this](std::stop_token st) { run(st); });
}
~TestTimer() {
worker_.request_stop();
cv_.notify_all();
}
void post(std::function<void()> fn) {
{
std::lock_guard lk(mu_);
queue_.push_back(std::move(fn));
}
cv_.notify_one();
}
// Blocks until the queue is empty and nothing is mid-run -- what a test needs before
// asserting on state this timer's closures mutate.
void drain() {
std::unique_lock lk(mu_);
cv_done_.wait(lk, [&] { return queue_.empty() && !running_; });
}
private:
void run(std::stop_token st) {
std::unique_lock lk(mu_);
while (true) {
cv_.wait(lk, st, [&] { return !queue_.empty(); });
if (st.stop_requested())
return;
auto fn = std::move(queue_.front());
queue_.erase(queue_.begin());
running_ = true;
lk.unlock();
fn();
lk.lock();
running_ = false;
if (queue_.empty())
cv_done_.notify_all();
}
}
std::mutex mu_;
std::condition_variable_any cv_; // _any: wait() below takes a stop_token predicate
std::condition_variable cv_done_;
std::vector<std::function<void()>> queue_;
bool running_ = false;
std::jthread worker_;
};
// Same grow/shrink logic as FakeTask, but on_target() only ever posts through a TestTimer
// instead of running synchronously -- see the comment above TestTimer for why that's the
// faithful model under concurrent driving.
struct AsyncFakeTask {
SegmentBudget *budget = nullptr;
TestTimer *timer = nullptr;
TaskId id{};
std::mutex mu; // only the timer thread ever touches workers/target -- plain suffices
std::uint32_t workers = 0;
std::uint32_t target = 0;
void on_target(std::uint32_t t) {
timer->post([this, t] {
std::lock_guard lk(mu);
target = t;
while (workers < target) {
if (!budget->confirm_slot(id))
break;
++workers;
}
while (workers > target) {
budget->release_slot(id);
--workers;
}
});
}
std::uint32_t held() {
std::lock_guard lk(mu);
return workers;
}
};
} // namespace
VT_TEST(budget_single_task_grows_to_cap) {
SegmentBudget b({.max_active_segments = 32});
FakeTask t{&b, tid(1)};
b.register_task(tid(1), {.host = "h", .per_task_cap = 8, .resumable = true},
[&](std::uint32_t n) { t.on_target(n); });
b.set_want(tid(1), 8);
VT_CHECK_EQ(t.held(), 8u);
VT_CHECK_EQ(b.segments_active(tid(1)), 8u);
VT_CHECK_EQ(b.budget().active, 8u);
VT_CHECK_EQ(b.budget().tasks_starved, 0u);
}
VT_TEST(budget_min_one_before_seconds) {
// Budget of 3, two tasks each wanting 8. min-1 first: each gets 1, then the higher-
// priority one grows to 2.
SegmentBudget b({.max_active_segments = 3});
FakeTask a{&b, tid(1)}, c{&b, tid(2)};
b.register_task(tid(1), {.host = "h1", .per_task_cap = 8, .resumable = true},
[&](std::uint32_t n) { a.on_target(n); });
b.register_task(tid(2), {.host = "h2", .per_task_cap = 8, .resumable = true},
[&](std::uint32_t n) { c.on_target(n); });
std::vector<TaskId> order = {tid(1), tid(2)};
b.set_task_order(order);
b.set_want(tid(1), 8);
b.set_want(tid(2), 8);
VT_CHECK(a.held() >= 1); // guarantee
VT_CHECK(c.held() >= 1); // guarantee — the load-bearing property
VT_CHECK_EQ(a.held() + c.held(), 3u);
VT_CHECK_EQ(a.held(), 2u); // higher priority took the growth slot
}
VT_TEST(budget_new_high_priority_task_gets_min_one_via_yield) {
SegmentBudget b({.max_active_segments = 4});
FakeTask a{&b, tid(1)};
b.register_task(tid(1), {.host = "h", .per_task_cap = 8, .resumable = true},
[&](std::uint32_t n) { a.on_target(n); });
b.set_task_order(std::vector<TaskId>{tid(1)});
b.set_want(tid(1), 8);
VT_CHECK_EQ(a.held(), 4u); // hogging the whole budget
// a second, higher-priority task arrives
FakeTask c{&b, tid(2)};
b.register_task(tid(2), {.host = "h2", .per_task_cap = 8, .resumable = true},
[&](std::uint32_t n) { c.on_target(n); });
b.set_task_order(std::vector<TaskId>{tid(2), tid(1)});
b.set_want(tid(2), 8);
// a yields so c gets at least its guaranteed slot; the surplus is shared round-robin.
VT_CHECK(c.held() >= 1); // min-1 — the load-bearing guarantee
VT_CHECK(a.held() >= 1); // a keeps its own min-1
VT_CHECK(a.held() < 4); // a really did yield at least one
VT_CHECK_EQ(a.held() + c.held(), 4u);
VT_CHECK_EQ(b.budget().active, 4u);
VT_CHECK_EQ(b.budget().tasks_starved, 0u);
}
VT_TEST(budget_host_cap_clamps_effective_target) {
SegmentBudget b({.max_active_segments = 32});
FakeTask t{&b, tid(1)};
b.set_host_segment_cap("slowcdn", 4);
b.register_task(tid(1), {.host = "slowcdn", .per_task_cap = 16, .resumable = true},
[&](std::uint32_t n) { t.on_target(n); });
b.set_want(tid(1), 16);
VT_CHECK_EQ(t.held(), 4u); // clamped by the host cap, not per_task_cap
b.set_host_segment_cap("slowcdn", 0); // clear
VT_CHECK_EQ(t.held(), 16u);
}
VT_TEST(budget_non_resumable_task_capped_at_one) {
SegmentBudget b({.max_active_segments = 32});
FakeTask t{&b, tid(1)};
b.register_task(tid(1), {.host = "h", .per_task_cap = 8, .resumable = false},
[&](std::uint32_t n) { t.on_target(n); });
b.set_want(tid(1), 8);
VT_CHECK_EQ(t.held(), 1u);
}
VT_TEST(budget_live_lower_sheds_via_yield_lowest_priority_first) {
SegmentBudget b({.max_active_segments = 24});
FakeTask a{&b, tid(1)}, c{&b, tid(2)}, d{&b, tid(3)};
for (auto *ft : {&a, &c, &d})
b.register_task(ft->id, {.host = "h", .per_task_cap = 8, .resumable = true},
[ft](std::uint32_t n) { ft->on_target(n); });
b.set_task_order(std::vector<TaskId>{tid(1), tid(2), tid(3)});
for (auto id : {tid(1), tid(2), tid(3)})
b.set_want(id, 8);
VT_CHECK_EQ(a.held() + c.held() + d.held(), 24u); // 8 + 8 + 8
b.set_max_active_segments(10); // live cut
VT_CHECK_EQ(a.held() + c.held() + d.held(), 10u);
VT_CHECK(a.held() >= c.held() && c.held() >= d.held()); // priority order preserved
VT_CHECK(a.held() >= 1 && c.held() >= 1 && d.held() >= 1); // min-1 still honoured
}
VT_TEST(budget_live_lower_below_task_count_starves_the_tail) {
SegmentBudget b({.max_active_segments = 6});
std::vector<FakeTask> ts(4);
for (std::uint32_t i = 0; i < 4; ++i) {
ts[i].budget = &b;
ts[i].id = tid(i + 1);
}
for (auto &ft : ts)
b.register_task(ft.id, {.host = "h", .per_task_cap = 4, .resumable = true},
[&ft](std::uint32_t n) { ft.on_target(n); });
b.set_task_order(std::vector<TaskId>{tid(1), tid(2), tid(3), tid(4)});
for (auto &ft : ts)
b.set_want(ft.id, 4);
VT_CHECK_EQ(b.budget().tasks_starved, 0u);
b.set_max_active_segments(3); // below the running-task count
VT_CHECK_EQ(ts[0].held(), 1u);
VT_CHECK_EQ(ts[3].held(), 0u); // lowest priority shed to zero
VT_CHECK_EQ(b.budget().tasks_starved, 1u);
VT_REQUIRE(b.starved_tasks().size() == 1);
VT_CHECK_EQ(b.starved_tasks()[0], tid(4));
VT_CHECK(b.starved_since(tid(4)).has_value());
VT_CHECK(!b.starved_since(tid(1)).has_value());
}
VT_TEST(budget_deregister_frees_slots_to_starved) {
SegmentBudget b({.max_active_segments = 4});
FakeTask a{&b, tid(1)}, c{&b, tid(2)};
b.register_task(tid(1), {.host = "h", .per_task_cap = 8, .resumable = true},
[&](std::uint32_t n) { a.on_target(n); });
b.set_task_order(std::vector<TaskId>{tid(1)});
b.set_want(tid(1), 8);
VT_CHECK_EQ(a.held(), 4u);
b.register_task(tid(2), {.host = "h", .per_task_cap = 8, .resumable = true},
[&](std::uint32_t n) { c.on_target(n); });
b.set_task_order(std::vector<TaskId>{tid(1), tid(2)});
b.set_want(tid(2), 8);
VT_CHECK(c.held() >= 1); // min-1 from a's yield
b.deregister_task(tid(1));
VT_CHECK_EQ(c.held(), 4u); // c grows into the whole freed budget
VT_CHECK_EQ(b.budget().active, 4u);
}
VT_TEST(budget_on_changed_fires_on_starved_edge) {
SegmentBudget b({.max_active_segments = 1, .notify_period = std::chrono::milliseconds{40}});
std::mutex m;
std::vector<EB> seen;
b.on_budget_changed([&](EB e) {
std::lock_guard lk(m);
seen.push_back(e);
});
FakeTask a{&b, tid(1)}, c{&b, tid(2)};
b.register_task(tid(1), {.host = "h", .per_task_cap = 4, .resumable = true},
[&](std::uint32_t n) { a.on_target(n); });
b.register_task(tid(2), {.host = "h", .per_task_cap = 4, .resumable = true},
[&](std::uint32_t n) { c.on_target(n); });
b.set_task_order(std::vector<TaskId>{tid(1), tid(2)});
b.set_want(tid(1), 4);
b.set_want(tid(2), 4); // budget is 1 -> tid(2) is starved: 0 -> nonzero edge
// the edge fire is synchronous on the triggering call
bool saw_starved = false;
{
std::lock_guard lk(m);
for (auto &e : seen)
if (e.tasks_starved > 0)
saw_starved = true;
}
VT_CHECK(saw_starved);
b.deregister_task(tid(1)); // frees the slot -> tid(2) no longer starved: edge back
std::this_thread::sleep_for(std::chrono::milliseconds(120));
bool saw_unstarved_after = false;
{
std::lock_guard lk(m);
VT_CHECK(!seen.empty());
saw_unstarved_after = seen.back().tasks_starved == 0;
}
VT_CHECK(saw_unstarved_after);
}
VT_TEST(budget_concurrent_confirm_release_stays_consistent) {
SegmentBudget b({.max_active_segments = 16});
constexpr int kTasks = 6;
TestTimer timer;
std::vector<std::unique_ptr<AsyncFakeTask>> ts;
for (int i = 0; i < kTasks; ++i) {
ts.push_back(std::make_unique<AsyncFakeTask>());
ts.back()->budget = &b;
ts.back()->timer = &timer;
ts.back()->id = tid(i + 1);
AsyncFakeTask *ft = ts.back().get();
b.register_task(ft->id, {.host = "h", .per_task_cap = 6, .resumable = true},
[ft](std::uint32_t n) { ft->on_target(n); });
}
std::vector<std::jthread> drivers;
for (int i = 0; i < kTasks; ++i) {
drivers.emplace_back([&, id = tid(i + 1)] {
for (int r = 0; r < 4000; ++r)
b.set_want(id, (r % 7));
});
}
drivers.clear(); // join
for (auto &ft : ts)
b.set_want(ft->id, 0);
timer.drain(); // let every queued on_target actually run before asserting
// With everyone wanting nothing, the budget must be fully released.
VT_CHECK_EQ(b.budget().active, 0u);
std::uint32_t sum = 0;
for (auto &ft : ts)
sum += b.segments_active(ft->id);
VT_CHECK_EQ(sum, 0u);
}
// --- wait-list wakeup: tools/bench heap-profile / load found a real task time out
// waiting on a slot its own target already said it should have (core/docs/m7-baseline.md,
// docs/adr/0012). Root cause: confirm_slot()'s engine-wide cap check (needed so active_
// never exceeds max_active_segments -- a real over-admission bug, not just this liveness
// gap) can deny a task whose target is already correct, when a *different* task is
// legitimately still holding more than its own just-lowered target (yield is deferred to
// a segment boundary, ADR 0011 A1). Nothing in the plain target-changed callback
// mechanism ever revisits a task whose target didn't change -- it was already right. ---
namespace {
// Unlike FakeTask above, on_target() here only enqueues -- it never calls back into the
// budget synchronously. This matches production exactly: register_task()'s on_target
// lambda (download_task.cpp) posts through host.schedule() (engine.cpp), so
// apply_slot_target() never runs on the same call stack as whatever budget call triggered
// it. The test drives delivery explicitly (deliver_one()) instead of a background thread
// so the race this test exists to force -- confirm_slot() denied before the task that's
// over its target has processed its own shrink -- is deterministic, not a timing gamble.
struct QueuedTask {
SegmentBudget *budget = nullptr;
TaskId id{};
std::uint32_t workers = 0;
std::uint32_t target = 0;
std::vector<std::uint32_t> pending;
QueuedTask(SegmentBudget *b, TaskId i) : budget(b), id(i) {}
void on_target(std::uint32_t n) { pending.push_back(n); }
// Delivers the oldest queued target, applying it the way a real task's
// apply_slot_target()/fill_slots_locked() would: try to grow to it (confirm_slot()
// may deny), or shed down to it. Returns false (nothing to deliver) if the queue was
// empty -- the condition VT_REQUIRE checks to prove a wakeup was actually queued.
bool deliver_one() {
if (pending.empty())
return false;
target = pending.front();
pending.erase(pending.begin());
while (workers < target) {
if (!budget->confirm_slot(id))
break;
++workers;
}
while (workers > target) {
budget->release_slot(id);
--workers;
}
return true;
}
};
} // namespace
VT_TEST(budget_release_wakes_a_denied_waiter) {
// Sanity baseline: max_active=1, A (higher priority) holds it, B wants one too and is
// fairly denied -- its target stays 0 while A outranks it and still wants its slot.
// Once A stops wanting one, B's target rises and B is woken via the plain
// target-changed path -- no wait-list needed for this simple case. The harder case
// below is what actually needs it.
SegmentBudget b({.max_active_segments = 1});
QueuedTask A{&b, tid(1)}, B{&b, tid(2)};
b.register_task(A.id, {.host = "h", .per_task_cap = 1, .resumable = true},
[&](std::uint32_t n) { A.on_target(n); });
b.set_task_order(std::vector<TaskId>{A.id, B.id});
b.set_want(A.id, 1);
VT_REQUIRE(A.deliver_one());
VT_CHECK_EQ(A.workers, 1u);
// B registers and wants one too, but with A (higher priority) already holding the
// only slot and still wanting it, B's fairly computed target stays 0 -- unchanged
// from its just-registered value, so no callback is queued for it yet.
b.register_task(B.id, {.host = "h", .per_task_cap = 1, .resumable = true},
[&](std::uint32_t n) { B.on_target(n); });
b.set_want(B.id, 1);
VT_CHECK(!B.deliver_one());
VT_CHECK_EQ(B.workers, 0u);
b.set_want(A.id, 0); // A is done wanting a slot
VT_REQUIRE(A.deliver_one()); // A's target dropped to 0 -- sheds its held slot
VT_CHECK_EQ(A.workers, 0u);
VT_REQUIRE(B.deliver_one()); // B's target rose to 1 -- the plain target-changed path
VT_CHECK_EQ(B.workers, 1u); // B took the freed slot
}
VT_TEST(budget_wait_list_wakes_a_task_whose_target_never_changed) {
// The real gap. max_active=2. Y alone, holds both (target=2). X arrives wanting 1:
// this recompute correctly drops Y's target to 1 (giving X its guaranteed slot) and
// raises X's target to 1 -- both real target changes, both queued. Deliver X's
// *first*: X's target says grow, but Y still physically holds 2 (hasn't processed
// its own shrink yet) -- confirm_slot() must deny X here (active_ == max_active_),
// which is the correctness fix (over-admission is the real RSS bug). Then Y
// processes its shrink and actually releases. X's target never changes again -- it
// was already correctly 1 -- so nothing in the plain mechanism ever revisits X.
SegmentBudget b({.max_active_segments = 2});
QueuedTask Y{&b, tid(1)}, X{&b, tid(2)};
b.register_task(Y.id, {.host = "h", .per_task_cap = 2, .resumable = true},
[&](std::uint32_t n) { Y.on_target(n); });
b.set_want(Y.id, 2);
VT_REQUIRE(Y.deliver_one());
VT_CHECK_EQ(Y.workers, 2u);
b.register_task(X.id, {.host = "h", .per_task_cap = 1, .resumable = true},
[&](std::uint32_t n) { X.on_target(n); });
b.set_task_order(std::vector<TaskId>{Y.id, X.id});
b.set_want(X.id, 1);
VT_REQUIRE(X.deliver_one());
VT_CHECK_EQ(X.workers, 0u); // denied: active_ == max_active_, even though X's target is 1
VT_REQUIRE(Y.deliver_one());
VT_CHECK_EQ(Y.workers, 1u); // Y actually releases its excess now
// The bug: without a wait-list, X.pending is empty here -- nothing was ever queued
// for it, because X's target never changed again. X would wait forever despite its
// target correctly saying it should hold a slot.
VT_REQUIRE(X.deliver_one());
VT_CHECK_EQ(X.workers, 1u);
VT_CHECK_EQ(b.budget().active, 2u);
}
+257
View File
@@ -0,0 +1,257 @@
#include "vdm/segment/segmenter.hpp"
#include <atomic>
#include <cstdint>
#include <thread>
#include <vector>
#include "vtest.hpp"
using namespace vdm::segment;
namespace {
constexpr std::uint64_t MiB = 1u << 20;
// Assign `n` slots (bounded by what the segmenter hands out) and return the indices.
std::vector<std::uint32_t> fill(Segmenter &s, int n) {
std::vector<std::uint32_t> idx;
for (int i = 0; i < n; ++i) {
auto a = s.assign_slot();
if (!a)
break;
idx.push_back(*a);
}
return idx;
}
// Do the segments (by their [start,end] at this instant) tile [0,total) with no overlap?
bool tiles_exactly(const Segmenter &s) {
auto snap = s.snapshot();
std::vector<std::pair<std::uint64_t, std::uint64_t>> r;
for (auto &v : snap)
if (v.state != SegState::failed)
r.emplace_back(v.start, v.end);
std::sort(r.begin(), r.end());
std::uint64_t cursor = 0;
for (auto [a, b] : r) {
if (a != cursor)
return false;
cursor = b + 1;
}
return cursor == s.total_size();
}
} // namespace
VT_TEST(seg_target_count_clamps) {
VT_CHECK_EQ(Segmenter(100 * MiB, 8, true).target_segment_count(), 8u);
VT_CHECK_EQ(Segmenter(100 * MiB, 64, true).target_segment_count(), 32u); // max 32
VT_CHECK_EQ(Segmenter(100 * MiB, 0, true).target_segment_count(), 8u); // default
VT_CHECK_EQ(Segmenter(3 * MiB + 1, 8, true).target_segment_count(), 3u); // total/min
VT_CHECK_EQ(Segmenter(100 * MiB, 8, false).target_segment_count(), 1u); // non-resumable
VT_CHECK_EQ(Segmenter(0, 8, true).target_segment_count(), 1u); // chunked
}
VT_TEST(seg_non_resumable_is_one_segment) {
Segmenter s(50 * MiB, 8, false);
auto idx = fill(s, 8);
VT_REQUIRE(idx.size() == 1);
auto snap = s.snapshot();
VT_REQUIRE(snap.size() == 1);
VT_CHECK_EQ(snap[0].start, 0u);
VT_CHECK_EQ(snap[0].end, 50u * MiB - 1);
}
VT_TEST(seg_initial_split_covers_range) {
Segmenter s(80 * MiB, 8, true);
auto idx = fill(s, 8);
VT_CHECK_EQ(idx.size(), 8u);
VT_CHECK(tiles_exactly(s));
// no segment below the 1 MiB floor
for (auto &v : s.snapshot())
VT_CHECK(v.length() >= MiB);
}
VT_TEST(seg_split_stops_at_min_floor) {
// 5 MiB, floor 1 MiB, ask for 8: only ~5 splits possible (each half >= 1 MiB needs
// the parent >= 2 MiB), so we get fewer than 8.
Segmenter s(5 * MiB, 8, true);
auto idx = fill(s, 8);
VT_CHECK(idx.size() >= 1 && idx.size() <= 5);
VT_CHECK(tiles_exactly(s));
}
VT_TEST(seg_steal_takes_second_half_of_largest_remaining) {
Segmenter s(80 * MiB, 4, true);
auto idx = fill(s, 4);
VT_REQUIRE(idx.size() == 4);
// Spread progress unevenly. Index != file position after splits, so identify the
// largest-remaining segment by scanning the snapshot, not by index.
s.advance(idx[1], 3 * MiB);
s.advance(idx[2], 7 * MiB);
s.advance(idx[3], 12 * MiB);
SegmentView pre_victim{};
std::uint64_t worst = 0;
for (auto &v : s.snapshot())
if (v.index != idx[0] && v.remaining() > worst) {
worst = v.remaining();
pre_victim = v;
}
auto cont = s.on_complete(idx[0], /*may_steal=*/true);
VT_REQUIRE(cont.has_value());
SegmentView victim{}, fresh{};
for (auto &v : s.snapshot()) {
if (v.index == pre_victim.index)
victim = v;
if (v.index == *cont)
fresh = v;
}
VT_CHECK_EQ(fresh.end, pre_victim.end); // fresh takes the tail of the victim's range
VT_CHECK_EQ(victim.end + 1, fresh.start); // contiguous, no gap / no overlap
VT_CHECK(victim.end < pre_victim.end); // the victim really did shrink
VT_CHECK(fresh.length() >= MiB);
VT_CHECK(victim.remaining() >= MiB);
// fresh got roughly the back half of what was remaining
VT_CHECK(fresh.length() >= worst / 2 - MiB && fresh.length() <= worst / 2 + MiB);
VT_CHECK(tiles_exactly(s));
}
VT_TEST(seg_complete_without_steal_releases) {
Segmenter s(4 * MiB, 2, true);
auto idx = fill(s, 2); // 2 x 2 MiB
VT_REQUIRE(idx.size() == 2);
s.advance(idx[1], 2 * MiB);
s.set_segment_state(idx[1], SegState::complete);
// idx[0] done, nothing left worth >= 1 MiB to steal -> release
s.advance(idx[0], 2 * MiB);
auto cont = s.on_complete(idx[0], true);
VT_CHECK(!cont.has_value());
}
VT_TEST(seg_yield_returns_nullopt) {
Segmenter s(80 * MiB, 4, true);
auto idx = fill(s, 4);
s.advance(idx[0], 20 * MiB);
auto cont = s.on_complete(idx[0], /*may_steal=*/false); // yielding
VT_CHECK(!cont.has_value());
}
VT_TEST(seg_third_connection_failure_with_mirror_requeues) {
Segmenter s(40 * MiB, 2, true);
auto idx = fill(s, 2);
s.advance(idx[0], 4 * MiB);
VT_CHECK(s.on_failed(idx[0], /*conn=*/true, /*mirror=*/true) == FailAction::retry);
VT_CHECK(s.on_failed(idx[0], true, true) == FailAction::retry);
VT_CHECK(s.on_failed(idx[0], true, true) == FailAction::requeue); // 3rd
VT_CHECK_EQ(s.segment_state(idx[0]), SegState::failed);
// the orphaned tail is now assignable again
auto again = s.assign_slot();
VT_REQUIRE(again.has_value());
auto snap = s.snapshot();
SegmentView reborn{};
for (auto &v : snap)
if (v.index == *again)
reborn = v;
VT_CHECK_EQ(reborn.start, 4u * MiB); // resumes where the failed one stopped
VT_CHECK_EQ(reborn.end, 20u * MiB - 1); // its half of the file
}
VT_TEST(seg_failure_without_mirror_always_retries) {
Segmenter s(40 * MiB, 2, true);
auto idx = fill(s, 2);
for (int i = 0; i < 6; ++i)
VT_CHECK(s.on_failed(idx[0], true, /*mirror=*/false) == FailAction::retry);
// a non-connection error also retries regardless of count
VT_CHECK(s.on_failed(idx[1], /*conn=*/false, /*mirror=*/true) == FailAction::retry);
}
VT_TEST(seg_note_connected_resets_failure_count) {
Segmenter s(40 * MiB, 2, true);
auto idx = fill(s, 2);
s.on_failed(idx[0], true, true);
s.on_failed(idx[0], true, true);
s.note_connected(idx[0]);
VT_CHECK(s.on_failed(idx[0], true, true) == FailAction::retry); // count restarted
}
VT_TEST(seg_resume_from_meta_table) {
std::vector<ResumedRange> table = {
{0, 9 * MiB - 1, 9 * MiB}, // fully done
{9 * MiB, 19 * MiB - 1, 3 * MiB}, // partial
{19 * MiB, 40 * MiB - 1, 0}, // untouched
};
Segmenter s(40 * MiB, 8, table, true);
auto snap = s.snapshot();
VT_REQUIRE(snap.size() == 3);
VT_CHECK_EQ(snap[0].state, SegState::complete);
VT_CHECK_EQ(snap[1].completed, 3u * MiB);
VT_CHECK_EQ(s.downloaded(), 12u * MiB);
VT_CHECK(tiles_exactly(s));
// assign hands out the two incomplete ranges before splitting
auto a = s.assign_slot();
auto b = s.assign_slot();
VT_REQUIRE(a && b);
}
VT_TEST(seg_resume_from_bad_table_falls_back) {
std::vector<ResumedRange> gappy = {{0, 4 * MiB - 1, 0}, {8 * MiB, 40 * MiB - 1, 0}};
Segmenter s(40 * MiB, 8, gappy, true);
VT_CHECK(s.snapshot().empty()); // lazy fresh layout
auto idx = fill(s, 8);
VT_CHECK(idx.size() >= 1);
VT_CHECK(tiles_exactly(s));
}
VT_TEST(seg_all_complete_and_downloaded) {
Segmenter s(8 * MiB, 4, true);
auto idx = fill(s, 4);
VT_CHECK(!s.all_complete());
for (auto i : idx) {
std::uint64_t len = s.segment_end(i) - s.segment_start(i) + 1;
s.advance(i, len);
s.set_segment_state(i, SegState::complete);
}
VT_CHECK(s.all_complete());
VT_CHECK_EQ(s.downloaded(), 8u * MiB);
}
// --- the steal path under the sanitizers -----------------------------------------------
VT_TEST(seg_concurrent_steal_and_advance) {
constexpr std::uint64_t total = 64 * MiB;
Segmenter s(total, 8, true);
auto idx = fill(s, 8);
VT_REQUIRE(idx.size() == 8);
std::vector<std::jthread> workers;
for (std::uint32_t w = 0; w < 8; ++w) {
workers.emplace_back([&s, seg = idx[w]]() mutable {
std::uint32_t cur = seg;
for (int guard = 0; guard < 200000; ++guard) {
const std::uint64_t start = s.segment_start(cur);
const std::uint64_t end = s.segment_end(cur); // may shrink under a steal
const std::uint64_t len = end - start + 1;
const std::uint64_t done = s.segment_completed(cur);
if (done >= len) {
auto nxt = s.on_complete(cur, /*may_steal=*/true);
if (!nxt)
return; // nothing left to steal — this worker is finished
cur = *nxt;
continue;
}
s.advance(cur, std::min(done + 128 * 1024, len));
}
});
}
workers.clear(); // join
VT_CHECK(s.all_complete());
VT_CHECK_EQ(s.downloaded(), total);
VT_CHECK(tiles_exactly(s));
}
+16 -5
View File
@@ -81,9 +81,17 @@ std::string show(const T &v) {
}
}
inline int run_all() {
inline int run_all(const std::vector<std::string> &filters = {}) {
int failed_cases = 0;
for (const auto &c : registry()) {
if (!filters.empty()) {
bool match = false;
for (const auto &f : filters)
if (std::string_view(c.name).find(f) != std::string_view::npos)
match = true;
if (!match)
continue;
}
int before = stats().failures;
stats().current_fatal = false;
std::fprintf(stderr, "[ RUN ] %s\n", c.name);
@@ -129,11 +137,14 @@ inline int run_all() {
::vt::report(__FILE__, __LINE__, #COND, {}, /*fatal=*/true); \
} while (0)
// NOTE: operands are copied (auto, not auto&&). A test assertion must never outlive a
// temporary the expression returned a reference into — the copy makes that safe. All
// compared types here are cheap to copy.
#define VT_CHECK_EQ(A, B) \
do { \
::vt::stats().checks++; \
auto &&_a = (A); \
auto &&_b = (B); \
auto _a = (A); \
auto _b = (B); \
if (!(_a == _b)) \
::vt::report(__FILE__, __LINE__, #A " == " #B, \
::vt::show(_a) + " vs " + ::vt::show(_b), false); \
@@ -142,8 +153,8 @@ inline int run_all() {
#define VT_CHECK_NE(A, B) \
do { \
::vt::stats().checks++; \
auto &&_a = (A); \
auto &&_b = (B); \
auto _a = (A); \
auto _b = (B); \
if (!(_a != _b)) \
::vt::report(__FILE__, __LINE__, #A " != " #B, \
::vt::show(_a) + " vs " + ::vt::show(_b), false); \
+25 -2
View File
@@ -1,6 +1,29 @@
// vtest_main.cpp — shared entry point for every CORE test binary.
#include <cstdlib>
#include <string>
#include <vector>
#include "vtest.hpp"
int main() {
return ::vt::run_all();
// Optional filters: each argv argument (or a comma-separated entry in $VT_ONLY) is a
// substring; a test runs only if its name contains one of them. No filters => run all.
int main(int argc, char **argv) {
std::vector<std::string> filters;
for (int i = 1; i < argc; ++i)
filters.emplace_back(argv[i]);
if (const char *env = std::getenv("VT_ONLY")) {
std::string cur;
for (const char *p = env;; ++p) {
if (*p == ',' || *p == '\0') {
if (!cur.empty())
filters.push_back(cur);
cur.clear();
if (*p == '\0')
break;
} else {
cur.push_back(*p);
}
}
}
return ::vt::run_all(filters);
}
+84
View File
@@ -0,0 +1,84 @@
// The engine API sketch must compile and its value types must behave. Engine /
// DownloadHandle bodies land in stage 8; this only exercises the data shapes DAEMON
// builds against.
#include "vdm/engine.hpp"
#include "vdm/task/download.hpp"
#include <type_traits>
#include "vtest.hpp"
using namespace vdm;
using namespace vdm::task;
VT_TEST(api_download_spec_defaults) {
DownloadSpec s;
s.url = "https://example.com/big.iso";
s.save_path = "/home/u/Downloads/big.iso";
VT_CHECK(s.mirrors.empty());
VT_CHECK(!s.segments.has_value());
VT_CHECK(!s.buffer_bytes.has_value());
VT_CHECK(!s.checksum.has_value());
VT_CHECK(!s.probe_hint.has_value());
VT_CHECK(s.allow_resume);
VT_CHECK(s.proxy.kind == net::ProxyKind::none);
VT_CHECK(s.auth.scheme == net::AuthScheme::none);
}
VT_TEST(api_state_helpers) {
VT_CHECK(is_terminal(EngineState::complete));
VT_CHECK(is_terminal(EngineState::failed));
VT_CHECK(is_terminal(EngineState::cancelled));
VT_CHECK(!is_terminal(EngineState::paused));
VT_CHECK(!is_terminal(EngineState::downloading));
}
VT_TEST(api_callbacks_are_all_optional) {
DownloadCallbacks cb; // every std::function default-constructs empty
VT_CHECK(!cb.on_progress);
VT_CHECK(!cb.on_state);
VT_CHECK(!cb.on_auth_required);
VT_CHECK(!cb.on_decision_needed);
VT_CHECK(!cb.on_finished);
cb.on_state = [](EngineState, EngineState, const std::optional<vdm::ErrorInfo> &) {};
cb.on_finished = [](Result<DownloadOutcome>) {};
VT_CHECK(cb.on_state && cb.on_finished);
}
VT_TEST(api_value_types_roundtrip) {
Progress p;
p.downloaded = 1234;
p.total = 5000;
p.effective_segments = 4;
SegmentProgress sp;
sp.index = 0;
sp.end = 1249;
sp.completed = 1234;
p.segments.push_back(sp);
VT_CHECK_EQ(p.segments.size(), 1u);
VT_CHECK_EQ(p.segments[0].end, 1249u);
DownloadOutcome o;
o.final_path = "/x";
o.bytes = 5000;
VT_CHECK_EQ(o.bytes, 5000u);
AuthChallenge a;
a.host = "h";
a.scheme = AuthChallenge::Scheme::digest;
VT_CHECK(a.scheme == AuthChallenge::Scheme::digest);
DecisionRequest d;
d.kind = DecisionRequest::Kind::server_file_changed;
d.detail = "changed";
VT_CHECK(d.kind == DecisionRequest::Kind::server_file_changed);
}
VT_TEST(api_handle_and_engine_are_move_only_shaped) {
static_assert(!std::is_copy_constructible_v<Engine>, "Engine is non-copyable");
static_assert(std::is_copy_constructible_v<DownloadHandle>, "handle is a shared handle");
DownloadHandle h; // default handle is invalid until Engine::start() fills it
VT_CHECK(!h.valid());
}
+674
View File
@@ -0,0 +1,674 @@
// End-to-end: a real Engine against tools/testserver, covering the CORE M1 DoD paths.
#include "vdm/engine.hpp"
#include <fcntl.h>
#include <unistd.h>
#include <atomic>
#include <chrono>
#include <cstdlib>
#include <future>
#include <string>
#include <vector>
#include "task/digest.hpp"
#include "testserver_fixture.hpp"
#include "vtest.hpp"
using namespace vdm;
using namespace vdm::task;
using vdm::testing::TestServer;
using namespace std::chrono_literals;
namespace {
struct TmpDir {
std::string path;
TmpDir() {
const char *d = std::getenv("TMPDIR");
path = (d ? d : "/tmp");
path += "/vdm_engine_XXXXXX";
path = ::mkdtemp(path.data()) ? path : "";
}
~TmpDir() {
// best-effort recursive cleanup of our flat dir
if (path.empty())
return;
std::string cmd = "rm -rf '" + path + "'";
(void)std::system(cmd.c_str());
}
std::string file(const std::string &name) const { return path + "/" + name; }
};
struct Recorder {
std::promise<Result<DownloadOutcome>> done;
std::future<Result<DownloadOutcome>> fut = done.get_future();
std::atomic<bool> fired{false};
std::vector<EngineState> states;
std::mutex mu;
std::atomic<int> auth_calls{0};
std::atomic<int> decision_calls{0};
// A probe callback can fire before the caller has stored the handle returned by
// eng.start(). Callbacks that reach back into the handle wait on this.
std::atomic<bool> handle_ready{false};
void arm(DownloadHandle &) { handle_ready.store(true, std::memory_order_release); }
DownloadCallbacks cbs(DownloadHandle *h = nullptr, std::string user = "",
std::string pass = "") {
DownloadCallbacks c;
c.on_state = [this](EngineState, EngineState to, const std::optional<ErrorInfo> &) {
std::lock_guard lk(mu);
states.push_back(to);
};
c.on_decision_needed = [this](const DecisionRequest &) { decision_calls.fetch_add(1); };
c.on_finished = [this](Result<DownloadOutcome> r) {
if (!fired.exchange(true))
done.set_value(std::move(r));
};
if (h) {
c.on_auth_required = [this, h, user, pass](const AuthChallenge &) {
auth_calls.fetch_add(1);
while (!handle_ready.load(std::memory_order_acquire))
std::this_thread::sleep_for(1ms);
h->provide_auth(user, pass, false);
};
}
return c;
}
Result<DownloadOutcome> wait(std::chrono::seconds to = 40s) {
if (fut.wait_for(to) != std::future_status::ready)
return Err{Error::timeout, "engine test wait"};
return fut.get();
}
bool saw(EngineState s) {
std::lock_guard lk(mu);
for (auto x : states)
if (x == s)
return true;
return false;
}
};
std::uint64_t file_size(const std::string &p) {
int fd = ::open(p.c_str(), O_RDONLY);
if (fd < 0)
return ~0ull;
off_t e = ::lseek(fd, 0, SEEK_END);
::close(fd);
return e < 0 ? ~0ull : static_cast<std::uint64_t>(e);
}
// The reference SHA-256 the testserver will report for a given path.
std::string server_sha(TestServer &srv, const std::string &mode, const std::string &size) {
// one-shot GET /<mode>/sha256/<size> via a throwaway Engine::probe? no — use raw curl
// through a small helper. Simplest: shell out.
std::string url = srv.url("/" + mode + "/sha256/" + size);
std::string cmd = "curl -s '" + url + "'";
std::string out;
if (FILE *f = ::popen(cmd.c_str(), "r")) {
char buf[512];
while (std::fgets(buf, sizeof buf, f))
out += buf;
::pclose(f);
}
auto q = out.find("\"sha256\"");
if (q == std::string::npos)
return {};
auto colon = out.find(':', q);
auto open = out.find('"', colon);
auto close = out.find('"', open + 1);
if (open == std::string::npos || close == std::string::npos)
return {};
return out.substr(open + 1, close - open - 1);
}
// Small, deliberately identical extraction to server_sha's: GET /<mode>/sign/<size>?ttl=N
// and pull the "url" field's value out of the {"url":..., "exp":...} JSON body.
std::string sign_url(TestServer &srv, const std::string &mode, const std::string &size,
int ttl_seconds) {
std::string url =
srv.url("/" + mode + "/sign/" + size + "?ttl=" + std::to_string(ttl_seconds));
std::string cmd = "curl -s '" + url + "'";
std::string out;
if (FILE *f = ::popen(cmd.c_str(), "r")) {
char buf[1024];
while (std::fgets(buf, sizeof buf, f))
out += buf;
::pclose(f);
}
auto q = out.find("\"url\"");
if (q == std::string::npos)
return {};
auto colon = out.find(':', q);
auto open = out.find('"', colon);
auto close = out.find('"', open + 1);
if (open == std::string::npos || close == std::string::npos)
return {};
return out.substr(open + 1, close - open - 1);
}
DownloadSpec spec_for(TestServer &srv, const std::string &urlpath, const std::string &save) {
DownloadSpec s;
s.url = srv.url(urlpath);
s.save_path = save;
return s;
}
} // namespace
VT_TEST(engine_plain_multisegment_download) {
TestServer srv;
VT_REQUIRE(srv.available());
TmpDir td;
VT_REQUIRE(!td.path.empty());
Recorder rec;
Engine eng;
auto h = eng.start(spec_for(srv, "/plain/file/4M", td.file("a.bin")), rec.cbs());
auto r = rec.wait();
VT_REQUIRE(r.has_value());
VT_CHECK_EQ(r.value().final_path, td.file("a.bin"));
VT_CHECK_EQ(r.value().bytes, 4u * 1024 * 1024);
VT_CHECK_EQ(file_size(td.file("a.bin")), 4u * 1024 * 1024);
VT_CHECK(rec.saw(EngineState::downloading));
VT_CHECK(rec.saw(EngineState::complete));
auto got = hash_file(td.file("a.bin"), Checksum::Algo::sha256);
VT_REQUIRE(got.has_value());
VT_CHECK_EQ(got.value(), server_sha(srv, "plain", "4M"));
// the sidecar is gone on success
VT_CHECK_EQ(::access((td.file("a.bin") + ".veloxpart.meta").c_str(), F_OK), -1);
}
VT_TEST(engine_checksum_pass_and_mismatch) {
TestServer srv;
VT_REQUIRE(srv.available());
TmpDir td;
Engine eng;
std::string want = server_sha(srv, "plain", "1M");
VT_REQUIRE(!want.empty());
{
Recorder rec;
auto s = spec_for(srv, "/plain/file/1M", td.file("ok.bin"));
s.checksum = Checksum{Checksum::Algo::sha256, want};
auto h = eng.start(std::move(s), rec.cbs());
auto r = rec.wait();
VT_REQUIRE(r.has_value());
VT_CHECK(rec.saw(EngineState::verifying));
}
{
Recorder rec;
auto s = spec_for(srv, "/plain/file/1M", td.file("bad.bin"));
s.checksum = Checksum{Checksum::Algo::sha256, std::string(64, 'a')};
auto h = eng.start(std::move(s), rec.cbs());
auto r = rec.wait();
VT_REQUIRE(!r.has_value());
VT_CHECK_EQ(r.error().code, Error::checksum_mismatch);
}
}
VT_TEST(engine_non_resumable_single_segment) {
TestServer srv;
VT_REQUIRE(srv.available());
TmpDir td;
Recorder rec;
Engine eng;
auto h = eng.start(spec_for(srv, "/no-range/file/2M", td.file("nr.bin")), rec.cbs());
auto r = rec.wait();
VT_REQUIRE(r.has_value());
VT_CHECK_EQ(file_size(td.file("nr.bin")), 2u * 1024 * 1024);
auto got = hash_file(td.file("nr.bin"), Checksum::Algo::sha256);
VT_CHECK_EQ(got.value(), server_sha(srv, "no-range", "2M"));
}
VT_TEST(engine_404_is_an_error) {
TestServer srv;
VT_REQUIRE(srv.available());
TmpDir td;
Recorder rec;
Engine eng;
auto h = eng.start(spec_for(srv, "/plain/nope", td.file("x.bin")), rec.cbs());
auto r = rec.wait();
VT_REQUIRE(!r.has_value());
VT_CHECK_EQ(r.error().code, Error::not_found);
VT_CHECK(rec.saw(EngineState::failed));
}
VT_TEST(engine_cancel_mid_download) {
TestServer srv;
VT_REQUIRE(srv.available());
TmpDir td;
Recorder rec;
Engine eng;
auto h = eng.start(spec_for(srv, "/throttled/file/8M", td.file("c.bin")), rec.cbs());
for (int i = 0; i < 200 && !rec.saw(EngineState::downloading); ++i)
std::this_thread::sleep_for(10ms);
VT_REQUIRE(rec.saw(EngineState::downloading));
h.cancel(/*discard_partial=*/true);
auto r = rec.wait();
VT_REQUIRE(!r.has_value());
VT_CHECK_EQ(r.error().code, Error::canceled);
VT_CHECK(rec.saw(EngineState::cancelled));
VT_CHECK_EQ(::access((td.file("c.bin") + ".veloxpart").c_str(), F_OK), -1); // discarded
}
VT_TEST(engine_pause_resume_completes) {
TestServer srv;
VT_REQUIRE(srv.available());
TmpDir td;
Recorder rec;
Engine eng;
auto h = eng.start(spec_for(srv, "/throttled/file/2M", td.file("pr.bin")), rec.cbs());
for (int i = 0; i < 200 && !rec.saw(EngineState::downloading); ++i)
std::this_thread::sleep_for(10ms);
h.pause();
for (int i = 0; i < 100 && h.state() != EngineState::paused; ++i)
std::this_thread::sleep_for(20ms);
VT_CHECK_EQ(h.state(), EngineState::paused);
h.resume();
auto r = rec.wait(90s);
VT_REQUIRE(r.has_value());
VT_CHECK_EQ(file_size(td.file("pr.bin")), 2u * 1024 * 1024);
auto got = hash_file(td.file("pr.bin"), Checksum::Algo::sha256);
VT_CHECK_EQ(got.value(), server_sha(srv, "throttled", "2M"));
}
VT_TEST(engine_resume_after_a_fresh_task) {
// Simulates kill -9: cancel WITHOUT discard, then a new task with allow_resume picks
// up the .veloxpart[.meta] and finishes with a byte-identical file.
TestServer srv;
VT_REQUIRE(srv.available());
TmpDir td;
std::string save = td.file("resume.bin");
std::string want = server_sha(srv, "throttled", "3M");
{
Recorder rec;
Engine eng;
auto h = eng.start(spec_for(srv, "/throttled/file/3M", save), rec.cbs());
for (int i = 0; i < 800 && (h.progress().downloaded < 512u * 1024); ++i)
std::this_thread::sleep_for(10ms);
VT_REQUIRE(h.progress().downloaded >= 512u * 1024);
h.cancel(/*discard_partial=*/false);
(void)rec.wait();
VT_CHECK_EQ(::access((save + ".veloxpart.meta").c_str(), F_OK), 0); // sidecar kept
}
{
Recorder rec;
Engine eng;
auto s = spec_for(srv, "/throttled/file/3M", save); // same source -> sidecar validates
s.allow_resume = true;
auto h = eng.start(std::move(s), rec.cbs());
auto r = rec.wait(120s);
VT_REQUIRE(r.has_value());
VT_CHECK_EQ(file_size(save), 3u * 1024 * 1024);
auto got = hash_file(save, Checksum::Algo::sha256);
VT_CHECK_EQ(got.value(), want); // byte-identical after resume
}
}
VT_TEST(engine_flaky_reset_retries_to_completion) {
TestServer srv;
VT_REQUIRE(srv.available());
TmpDir td;
Recorder rec;
Engine eng;
// flaky-reset RSTs the first two attempts per (path,range); a single-segment request
// therefore needs the retry loop.
auto s = spec_for(srv, "/flaky-reset/file/256K", td.file("fl.bin"));
s.segments = 1;
s.max_retries = 40; // the server RSTs at the halfway point every attempt -> ~18 halvings
auto h = eng.start(std::move(s), rec.cbs());
auto r = rec.wait(120s);
VT_REQUIRE(r.has_value());
VT_CHECK_EQ(file_size(td.file("fl.bin")), 256u * 1024);
auto got = hash_file(td.file("fl.bin"), Checksum::Algo::sha256);
VT_CHECK_EQ(got.value(), server_sha(srv, "flaky-reset", "256K"));
VT_CHECK(rec.saw(EngineState::retry_wait) || rec.saw(EngineState::connecting));
}
VT_TEST(engine_401_then_provide_auth_completes) {
TestServer srv;
VT_REQUIRE(srv.available());
TmpDir td;
Recorder rec;
Engine eng;
DownloadHandle h;
auto cbs = rec.cbs(&h, "test", "test");
h = eng.start(spec_for(srv, "/401-basic/file/1M", td.file("au.bin")), std::move(cbs));
rec.arm(h); // publish h to the auth callback (which may already be waiting)
auto r = rec.wait();
VT_REQUIRE(r.has_value());
VT_CHECK(rec.auth_calls.load() >= 1);
VT_CHECK_EQ(file_size(td.file("au.bin")), 1u * 1024 * 1024);
}
VT_TEST(engine_401_digest_then_provide_auth_completes) {
// Same shape as engine_401_then_provide_auth_completes, but the challenge is HTTP
// Digest (qop=auth) rather than Basic. provide_auth() doesn't know or care which --
// http_client.cpp always asks libcurl for CURLAUTH_ANY (net::AuthScheme::any) and lets
// curl negotiate against whatever WWW-Authenticate the server actually sent -- so this
// exists purely to prove that's true end-to-end, not just at the unit level.
TestServer srv;
VT_REQUIRE(srv.available());
TmpDir td;
Recorder rec;
Engine eng;
DownloadHandle h;
auto cbs = rec.cbs(&h, "test", "test");
h = eng.start(spec_for(srv, "/401-digest/file/1M", td.file("dg.bin")), std::move(cbs));
rec.arm(h);
auto r = rec.wait();
VT_REQUIRE(r.has_value());
VT_CHECK(rec.auth_calls.load() >= 1);
VT_CHECK_EQ(file_size(td.file("dg.bin")), 1u * 1024 * 1024);
auto got = hash_file(td.file("dg.bin"), Checksum::Algo::sha256);
VT_CHECK_EQ(got.value(), server_sha(srv, "401-digest", "1M"));
}
// --- hostile-mode matrix: the four where a bug is silent corruption, not a visible
// failure (docs/04 §5 "ask, never silently corrupt" / §7's failure-policy table). ---
VT_TEST(engine_etag_changes_asks_instead_of_splicing) {
// A server that revalidates with a different ETag on every response fails an If-Range
// on any retry or resume. That must surface as "ask the user" (server_file_changed),
// never as a silent restart-from-offset-0 spliced onto bytes already on disk.
TestServer srv;
VT_REQUIRE(srv.available());
TmpDir td;
Recorder rec;
Engine eng;
auto h = eng.start(spec_for(srv, "/throttled+etag-changes/file/2M", td.file("ec.bin")),
rec.cbs());
// Get real progress on at least one segment before pausing, so resume's If-Range (only
// sent once a segment has completed > 0) actually fires.
for (int i = 0; i < 300 && h.progress().downloaded < 64u * 1024; ++i)
std::this_thread::sleep_for(10ms);
VT_REQUIRE(h.progress().downloaded >= 64u * 1024);
h.pause();
for (int i = 0; i < 200 && h.state() != EngineState::paused; ++i)
std::this_thread::sleep_for(20ms);
VT_REQUIRE(h.state() == EngineState::paused);
h.resume();
for (int i = 0; i < 300 && rec.decision_calls.load() == 0; ++i)
std::this_thread::sleep_for(20ms);
VT_REQUIRE(rec.decision_calls.load() >= 1);
VT_CHECK_EQ(h.state(), EngineState::paused);
h.decide(Decision::restart);
auto r = rec.wait(90s);
VT_REQUIRE(r.has_value());
VT_CHECK_EQ(file_size(td.file("ec.bin")), 2u * 1024 * 1024);
auto got = hash_file(td.file("ec.bin"), Checksum::Algo::sha256);
VT_CHECK_EQ(got.value(), server_sha(srv, "throttled+etag-changes", "2M"));
}
VT_TEST(engine_416_mid_download_asks_instead_of_exhausting_retries) {
// 416-always 416s every ranged request, including the probe's own -- a live probe
// correctly concludes "not resumable" and a plain-GET download never touches Range
// (that path is the same shape as engine_non_resumable_single_segment). The failure
// mode docs/04 means -- a server that *was* proven resumable dropping Range support
// mid-download -- needs a worker to actually send Range against it, so force the
// resumable, multi-segment assumption directly via probe_hint.
TestServer srv;
VT_REQUIRE(srv.available());
TmpDir td;
Recorder rec;
Engine eng;
net::ProbeResult hint;
hint.total_size = 256u * 1024;
hint.last_modified = "Wed, 01 Jan 2025 00:00:00 GMT";
hint.accept_ranges = true;
hint.resumable = true;
auto s = spec_for(srv, "/416-always/file/256K", td.file("rb.bin"));
s.probe_hint = hint;
s.segments = 2;
auto h = eng.start(std::move(s), rec.cbs());
for (int i = 0; i < 300 && rec.decision_calls.load() == 0; ++i)
std::this_thread::sleep_for(20ms);
VT_REQUIRE(rec.decision_calls.load() >= 1);
VT_CHECK_EQ(h.state(), EngineState::paused);
// 416-always never recovers -- re-probing would just 416 again -- so the only sound
// resolution is to stop, honestly, rather than retry the stale range until exhaustion.
h.decide(Decision::abort);
auto r = rec.wait();
VT_REQUIRE(!r.has_value());
VT_CHECK_EQ(r.error().code, Error::range_not_satisfiable);
VT_CHECK_EQ(::access(td.file("rb.bin").c_str(), F_OK), -1); // never declared complete
}
VT_TEST(engine_lies_about_accept_ranges_demotes_without_asking) {
// Ranges are always ignored (a plain 200, full body) but ETag/Last-Modified are stable
// and honest -- unlike etag-changes, this is provably the *same* file, just a Range-
// blind connection. docs/04 §7: demote to 1 segment and continue, automatically, no
// user round-trip. As with 416-always, a live probe already gets this right up front
// (proven non-resumable), so probe_hint forces the interesting mid-download case.
TestServer srv;
VT_REQUIRE(srv.available());
TmpDir td;
Recorder rec;
Engine eng;
std::string want = server_sha(srv, "lies-about-accept-ranges", "512K");
VT_REQUIRE(!want.empty());
net::ProbeResult hint;
hint.total_size = 512u * 1024;
hint.last_modified = "Wed, 01 Jan 2025 00:00:00 GMT"; // testserver sends this verbatim
hint.accept_ranges = true;
hint.resumable = true;
auto s = spec_for(srv, "/lies-about-accept-ranges/file/512K", td.file("lar.bin"));
s.probe_hint = hint;
s.segments = 4;
auto h = eng.start(std::move(s), rec.cbs());
auto r = rec.wait(60s);
VT_REQUIRE(r.has_value());
VT_CHECK_EQ(rec.decision_calls.load(), 0); // demoted automatically, not asked
VT_CHECK_EQ(file_size(td.file("lar.bin")), 512u * 1024);
auto got = hash_file(td.file("lar.bin"), Checksum::Algo::sha256);
VT_CHECK_EQ(got.value(), want);
}
VT_TEST(engine_content_length_mismatch_fails_honestly) {
// Content-Length promises the true size but the connection always closes short of it.
// There is no recovery (unlike flaky-reset, this never "heals" on a later attempt), so
// the segment's remaining range shrinks every retry until it stalls at zero progress.
// The only correct outcome is a real, visible failure -- never a rename to save_path
// built from a file that is quietly missing bytes.
TestServer srv;
VT_REQUIRE(srv.available());
TmpDir td;
Recorder rec;
Engine eng;
auto s = spec_for(srv, "/content-length-mismatch/file/16K", td.file("clm.bin"));
s.segments = 1;
s.max_retries = 4;
auto h = eng.start(std::move(s), rec.cbs());
auto r = rec.wait(60s);
VT_REQUIRE(!r.has_value());
VT_CHECK_EQ(r.error().code, Error::max_retries_exhausted);
VT_CHECK(rec.saw(EngineState::failed));
VT_CHECK_EQ(::access(td.file("clm.bin").c_str(), F_OK), -1); // never renamed into place
}
// --- remaining hostile-mode matrix (tools/testserver/README.md's mode table). ---
VT_TEST(engine_expiring_signed_url_recovers_via_refresh_url) {
// A signed URL past its ttl 403s (tools/testserver's own JSON body distinguishes
// "expired" from "bad signature", but core never parses response bodies -- CLAUDE.md
// §3 -- so both just read as a 403). The one automatic referrer retry (see
// engine_403_without_referer_retries_with_origin, below) can't fix an expired
// signature, so the second 403 asks -- via the same auto_pause_locked(..., false,
// true) "ask, don't just fail" path as wrong_status/range_bad -- rather than
// terminally failing outright, specifically so DownloadHandle::refresh_url() (its own
// contract: works "on a live or paused task", never on a terminal one) stays usable:
// the README pairs this mode with exactly that recovery.
TestServer srv;
VT_REQUIRE(srv.available());
TmpDir td;
Recorder rec;
Engine eng;
std::string expired = sign_url(srv, "expiring-signed-url", "64K", /*ttl=*/1);
VT_REQUIRE(!expired.empty());
std::this_thread::sleep_for(1500ms); // let the ttl actually pass before the first request
DownloadSpec s;
s.url = expired;
s.save_path = td.file("exp.bin");
auto h = eng.start(std::move(s), rec.cbs());
for (int i = 0; i < 300 && rec.decision_calls.load() == 0; ++i)
std::this_thread::sleep_for(20ms);
VT_REQUIRE(rec.decision_calls.load() >= 1);
VT_CHECK_EQ(h.state(), EngineState::paused);
std::string fresh = sign_url(srv, "expiring-signed-url", "64K", /*ttl=*/60);
VT_REQUIRE(!fresh.empty());
h.refresh_url(fresh);
auto r = rec.wait(60s);
VT_REQUIRE(r.has_value());
VT_CHECK_EQ(file_size(td.file("exp.bin")), 64u * 1024);
auto got = hash_file(td.file("exp.bin"), Checksum::Algo::sha256);
VT_CHECK_EQ(got.value(), server_sha(srv, "expiring-signed-url", "64K"));
}
VT_TEST(engine_403_without_referer_retries_with_origin) {
// docs/04 §7: "403 after redirect: retry once with the original referrer -- many CDNs
// require it." No spec.referrer is set here (the common case for anything not
// initiated from a browser page, e.g. `velox add <url>`), so the first attempt 403s;
// the engine's own retry supplies the download URL's own origin as Referer, which
// this mode accepts, and the download completes with no decision ever asked.
TestServer srv;
VT_REQUIRE(srv.available());
TmpDir td;
Recorder rec;
Engine eng;
auto h = eng.start(spec_for(srv, "/403-without-referer/file/128K", td.file("ref.bin")),
rec.cbs());
auto r = rec.wait(30s);
VT_REQUIRE(r.has_value());
VT_CHECK_EQ(rec.decision_calls.load(), 0); // recovered automatically, not asked
VT_CHECK_EQ(file_size(td.file("ref.bin")), 128u * 1024);
auto got = hash_file(td.file("ref.bin"), Checksum::Algo::sha256);
VT_CHECK_EQ(got.value(), server_sha(srv, "403-without-referer", "128K"));
}
VT_TEST(engine_redirect_chain_follows_to_completion) {
// 5 hops (tools/testserver's own --redirect-depth default) of a plain 302, query
// string preserved across each. No CORE-side logic needed for this one -- libcurl's
// own CURLOPT_FOLLOWLOCATION (RequestOptions::follow_redirects, already on) and
// CURLOPT_MAXREDIRS (default 20, well over 5) do the whole thing -- this is here as
// the end-to-end check that they're actually wired through both the probe and every
// segment worker's own request, not just one of the two.
TestServer srv;
VT_REQUIRE(srv.available());
TmpDir td;
Recorder rec;
Engine eng;
auto h = eng.start(spec_for(srv, "/redirect-chain/file/1M", td.file("rc.bin")), rec.cbs());
auto r = rec.wait(30s);
VT_REQUIRE(r.has_value());
VT_CHECK_EQ(file_size(td.file("rc.bin")), 1u * 1024 * 1024);
auto got = hash_file(td.file("rc.bin"), Checksum::Algo::sha256);
VT_CHECK_EQ(got.value(), server_sha(srv, "redirect-chain", "1M"));
}
VT_TEST(engine_slow_loris_stall_timeout_fires) {
// Status line, headers, and body dribbled out one byte at a time for --loris-seconds,
// then (if the dribble hasn't already been cut off) normal streaming -- a connection
// that's technically alive (bytes ARE arriving, just far too slowly) but must not be
// allowed to hang the task forever. http_client.cpp sets CURLOPT_LOW_SPEED_LIMIT/_TIME
// (RequestOptions::low_speed_bytes_per_sec/low_speed_secs, hardcoded in
// download_task.cpp to 1024 B/s for 30s) for exactly this.
//
// Every other test in this file uses TestServer's default 1s loris dribble to keep
// runtime down, but 1s is far shorter than curl's 30s low_speed_time: a 1s trickle
// followed by full-speed streaming never accumulates 30 CONSECUTIVE seconds under the
// floor, so curl would never actually abort it -- the download would just complete
// slightly late, which would make this test pass for the wrong reason (or not exercise
// the stall timeout at all). Explicitly ask for a dribble that outlasts the 30s
// threshold so the stall timeout is the thing actually observed firing, not assumed.
TestServer srv(40.0);
VT_REQUIRE(srv.available());
TmpDir td;
Recorder rec;
Engine eng;
auto s = spec_for(srv, "/slow-loris/file/64K", td.file("sl.bin"));
s.segments = 1;
s.max_retries = 1;
auto h = eng.start(std::move(s), rec.cbs());
auto r = rec.wait(60s); // stall timeout fires ~30s in; must resolve, not hang to 60s
VT_REQUIRE(!r.has_value());
VT_CHECK(is_retryable(r.error().code) || r.error().code == Error::max_retries_exhausted);
}
VT_TEST(engine_chunked_no_length_completes_single_segment) {
// No Content-Length anywhere (HEAD gets none either, since it's the same handler path)
// -- the probe can't know total_size or prove resumability, so this should take the
// exact same "unknown size, one plain-GET segment" path as engine_non_resumable_single_
// segment, just arriving there via a chunked body instead of a server that plainly
// refuses Range. No core-side work needed if that demotion is already size-agnostic;
// this is here to prove it, since every other test's server tells the probe the size
// up front.
TestServer srv;
VT_REQUIRE(srv.available());
TmpDir td;
Recorder rec;
Engine eng;
auto h = eng.start(spec_for(srv, "/chunked-no-length/file/2M", td.file("ch.bin")),
rec.cbs());
auto r = rec.wait();
VT_REQUIRE(r.has_value());
VT_CHECK_EQ(rec.decision_calls.load(), 0);
VT_CHECK_EQ(file_size(td.file("ch.bin")), 2u * 1024 * 1024);
auto got = hash_file(td.file("ch.bin"), Checksum::Algo::sha256);
VT_CHECK_EQ(got.value(), server_sha(srv, "chunked-no-length", "2M"));
}
// --- DAEMON-reported bug: Progress.speed_bps reads 0 for the whole life of a live
// download while downloaded bytes visibly advance. DAEMON reads progress by polling
// DownloadHandle::progress() (engine_port_core.hpp), not the on_progress push callback --
// this exercises exactly that path. ---
VT_TEST(engine_polled_progress_reports_nonzero_speed) {
TestServer srv;
VT_REQUIRE(srv.available());
TmpDir td;
Recorder rec;
Engine eng;
auto h = eng.start(spec_for(srv, "/throttled/file/4M", td.file("sp.bin")), rec.cbs());
// Give it real, sustained progress: the speed estimate only updates on a >=0.5s
// sample window (seg_data), so a snapshot taken too early would legitimately read 0
// even with the bug fixed. Poll until downloaded has clearly advanced twice over.
std::uint64_t speed = 0;
for (int i = 0; i < 400 && speed == 0; ++i) {
std::this_thread::sleep_for(20ms);
auto p = h.progress();
if (p.downloaded >= 256u * 1024)
speed = p.speed_bps;
}
VT_CHECK(speed > 0);
h.cancel(/*discard_partial=*/true);
auto r = rec.wait();
VT_REQUIRE(!r.has_value());
}
+112
View File
@@ -0,0 +1,112 @@
# daemon/ produces the veloxd binary and the static libraries it is built from.
# Owned by lane DAEMON. Wired in by PKG via add_subdirectory(daemon) in the root file,
# guarded on this file existing.
#
# Layering (CLAUDE.md §3): depends on velox::core and velox::proto. No Qt. The engine
# (velox::core) is not linked yet it arrives when sched/ and the task glue land. This
# drop is the RPC transports (Unix socket + loopback WebSocket), the SQLite store, and a
# dispatcher skeleton so the CLI and GUI have a real server to talk to.
if(NOT TARGET nlohmann_json::nlohmann_json)
find_package(nlohmann_json 3.11 REQUIRED)
endif()
find_package(Threads REQUIRED)
find_package(SQLite3 REQUIRED)
find_package(OpenSSL REQUIRED) # libcrypto: WebSocket accept hash, pairing token hash
# --- generated: migrations_embedded.hpp from src/store/migrations/*.sql ---------------
set(_mig_dir ${CMAKE_CURRENT_SOURCE_DIR}/src/store/migrations)
set(_mig_hdr ${CMAKE_CURRENT_BINARY_DIR}/generated/migrations_embedded.hpp)
file(GLOB _mig_srcs ${_mig_dir}/*.sql)
add_custom_command(
OUTPUT ${_mig_hdr}
COMMAND ${CMAKE_COMMAND} -DMIG_DIR=${_mig_dir} -DOUT=${_mig_hdr}
-P ${CMAKE_CURRENT_SOURCE_DIR}/cmake/embed_migrations.cmake
DEPENDS ${_mig_srcs} ${CMAKE_CURRENT_SOURCE_DIR}/cmake/embed_migrations.cmake
COMMENT "Embedding SQL migrations"
VERBATIM)
add_custom_target(veloxd_migrations_hdr DEPENDS ${_mig_hdr})
# --- veloxd_store SQLite store, migrations, crypto helpers --------------------------
add_library(veloxd_store STATIC
src/util/crypto.cpp
src/store/sqlite.cpp
src/store/migrations.cpp
src/store/pairings.cpp
src/store/settings.cpp
src/store/tasks.cpp
src/store/categories.cpp
src/store/queues.cpp
src/store/rules.cpp
src/store/segments.cpp
${_mig_hdr}
)
add_library(velox::daemon_store ALIAS veloxd_store)
target_include_directories(veloxd_store
PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/src
PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/generated
)
target_compile_features(veloxd_store PUBLIC cxx_std_23)
target_compile_options(veloxd_store PRIVATE -Wall -Wextra -Wpedantic -Werror)
target_link_libraries(veloxd_store PUBLIC SQLite::SQLite3 velox::proto nlohmann_json::nlohmann_json PRIVATE OpenSSL::Crypto)
# --- veloxd_fs the saveDir/filename path-traversal boundary (security) -----------
# daemon/docs/safepath-adversarial.md is the spec; safepath_test.cpp is that table.
add_library(veloxd_fs STATIC src/fs/safepath.cpp)
add_library(velox::daemon_fs ALIAS veloxd_fs)
target_include_directories(veloxd_fs PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/src)
target_compile_features(veloxd_fs PUBLIC cxx_std_23)
target_compile_options(veloxd_fs PRIVATE -Wall -Wextra -Wpedantic -Werror)
# --- veloxd_sched the concurrency governor (pure; no engine link yet, see
# daemon/docs/deferrals.md D4) ---------------------------------------------------
add_library(veloxd_sched STATIC
src/sched/schedule_window.cpp
src/sched/governor.cpp
src/sched/scheduler.cpp
)
add_library(velox::daemon_sched ALIAS veloxd_sched)
target_include_directories(veloxd_sched PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/src)
target_compile_features(veloxd_sched PUBLIC cxx_std_23)
target_compile_options(veloxd_sched PRIVATE -Wall -Wextra -Wpedantic -Werror)
target_link_libraries(veloxd_sched PUBLIC velox::proto velox::core veloxd_store veloxd_rpc nlohmann_json::nlohmann_json)
# --- veloxd_rpc the RPC transports + dispatcher ------------------------------------
add_library(veloxd_rpc STATIC
src/rpc/runtime_dir.cpp
src/rpc/single_instance.cpp
src/rpc/event_loop.cpp
src/rpc/event_hub.cpp
src/rpc/uds_server.cpp
src/rpc/ws_frame.cpp
src/rpc/ws_handshake.cpp
src/rpc/ws_server.cpp
src/rpc/pairing.cpp
src/rpc/dispatcher.cpp
)
add_library(velox::daemon_rpc ALIAS veloxd_rpc)
target_include_directories(veloxd_rpc PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/src)
target_compile_features(veloxd_rpc PUBLIC cxx_std_23)
target_compile_options(veloxd_rpc PRIVATE -Wall -Wextra -Wpedantic -Werror)
# velox::core: dispatcher.hpp includes sched/scheduler.hpp for the Scheduler* it drives
# download.pause/resume/start/cancel and queue.start/stop through (D4b), which pulls in
# core/include's vdm/*.hpp. Interface-only from here (no .cpp in this library calls into
# CORE directly) the actual Scheduler symbols resolve at the veloxd executable's link
# step (veloxd links both veloxd_rpc and veloxd_sched), not here, so this does not create
# the veloxd_rpc <-> veloxd_sched cycle that linking veloxd_sched itself would (veloxd_sched
# already links veloxd_rpc, for EventHub).
target_link_libraries(veloxd_rpc
PUBLIC velox::proto velox::core veloxd_store veloxd_fs nlohmann_json::nlohmann_json
Threads::Threads
)
# --- veloxd the daemon binary -------------------------------------------------------
add_executable(veloxd src/main.cpp)
target_compile_features(veloxd PRIVATE cxx_std_23)
target_compile_options(veloxd PRIVATE -Wall -Wextra -Wpedantic -Werror)
target_link_libraries(veloxd PRIVATE veloxd_rpc veloxd_sched)
if(VELOX_BUILD_TESTS AND EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/tests/CMakeLists.txt)
add_subdirectory(tests)
endif()

Some files were not shown because too many files have changed in this diff Show More