core: rate/token_bucket + fold in DAEMON's engine-API review (stage 7)

rate/token_bucket.hpp — a lazily-refilled TokenBucket (starts full: burst
then throttle, IDM behaviour; rate 0 == unlimited; burst caps idle
accumulation) and RateLimiter, the global -> per-queue -> per-task
hierarchy (docs/04 §6). acquire(task, n) peeks every applicable level and
commits on all-or-none so a blocked attempt never leaks tokens at a level
that had them; held under one mutex so a concurrent detach can't dangle
the bucket it's using. vdm/ids.hpp gains QueueId.

Tests: burst/refill/cap/unlimited for the bucket; tightest-level-binds,
no-partial-consumption, detach-safety, and an 8-thread aggregate-rate
check for the hierarchy. Green under ASan/UBSan and TSan.

Engine-API review (DAEMON signed off, no sched/ or dispatch rewrite):
 - Engine::rate_limiter() accessor added (limiter.set -> set_global_limit).
 - Checksum::Algo gains sha512 to match the wire Checksum set.
 - DownloadSpec: DAEMON creates save_path's parent dir before start();
   missing dir -> Error::path_rejected (made explicit).
 - cancel(): documented to fire on_state(_, cancelled, nullopt) then
   on_finished(Err{canceled}), in that order; download.cancel ==
   cancel(false), download.remove == cancel(true).
 - engine-api-m1.md: the five open questions resolved with DAEMON's
   answers (probe_hint optional, single cancel flag, {restart,
   keep_partial, abort} is the whole set, per-task 4 Hz is fine,
   refresh_url restarts all segments after a validating re-probe).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HPPSGhiArbvQgwC2DNiURS
This commit is contained in:
2026-09-10 15:50:18 +04:00
co-authored by Claude Sonnet 5
parent d6cf1fe7dc
commit 3da4cd6e91
7 changed files with 414 additions and 26 deletions
+50 -22
View File
@@ -1,6 +1,18 @@
# CORE → DAEMON — the engine API for review (M1)
# CORE → DAEMON — the engine API (M1)
**Status: for review.** This is the `libveloxcore` public surface `veloxd` links and calls
**Status: reviewed and signed off** by DAEMON on 2026-09-10
(`daemon/docs/engine-api-review.md`). Nothing forced a `sched/` or dispatch rewrite. The
five open questions are resolved at the bottom; the review's four confirms are folded into
the headers (`sha512` added to `Checksum::Algo`; the parent-directory contract made
explicit; `cancel()` ordering documented; `rate_limiter()` accessor added).
Integration order: DAEMON wires this in after `daemon/src/sched/` lands (the scheduler is
what calls `start`/`pause`/`resume`/`cancel` and drives `set_task_order`); `sched/` builds
against these headers in parallel with CORE stage 8 and does not need the bodies.
---
This is the `libveloxcore` public surface `veloxd` links and calls
to actually run a download. It is the thing the AGENT-CORE brief asked for on day one and
that slipped: DAEMON has an RPC surface and a store and, until this is agreed, nothing to
call. Sketch headers: `core/include/vdm/engine.hpp`, `core/include/vdm/task/download.hpp`
@@ -109,9 +121,15 @@ task waiting on credentials.
- **`decide(Decision)`** — acts only while auto-paused for a decision. `restart` discards
the partial and re-downloads; `keep_partial` continues against what is on disk (the
user's stated risk); `abort``failed`. No-op otherwise.
- **`refresh_url(url, headers)`** — IDM's "Refresh Download Address": swap the URL on a
live or paused task without losing progress (a fresh signed URL). Maps to
`download.refreshUrl`.
- **`refresh_url(url, headers)`** — IDM's "Refresh Download Address": the engine re-probes
the new URL to validate it, then **restarts every segment on it** (the signed-URL-expiry
case the wire method exists for), keeping the bytes already on disk. Mirror rotation is
*not* this — that is `spec.mirrors` + the segmenter's requeue-to-a-different-host.
- **`on_decision_needed`** is only for the cases the engine cannot resolve itself. A
routine 416 / stale range is the engine's own re-probe + re-split loop; it escalates
`DecisionRequest{range_metadata_stale}` **only when that loop fails**, at which point
"retry the same range" is already exhausted — so `{restart, keep_partial, abort}` is the
whole choice set.
## 5. Progress
@@ -138,21 +156,31 @@ source until the stream ends.
---
## Open questions for DAEMON
## Resolved (DAEMON review, 2026-09-10)
1. **`DownloadSpec.probe_hint`** — do you want to pass the File-Info `ProbeResult` in, or
would you rather the engine always probe (one code path, ~1 extra round trip)? The
sketch supports both; picking one simplifies stage 8.
2. **`cancel(discard_partial)`** vs a separate `handle.remove()` — the wire has
`download.cancel` and `download.remove {deleteFile}` as two methods. One call with a
flag, or two?
3. **`on_decision_needed` granularity** — is `{restart, keep_partial, abort}` the right
choice set for `server_file_changed`, or do you also need "retry the same range once
more" as a distinct option for the stale-416 case?
4. **Progress cadence** — 4 Hz per task matches the wire. With 20 active tasks that is 80
callbacks/s on the engine's timer thread. Acceptable, or do you want a single
`on_progress_batch(span<Progress>)` so the engine coalesces across tasks too?
5. **`refresh_url` while `downloading`** — should an in-flight segment finish on the old
URL and only new/retried segments use the new one (less disruption), or should the
engine restart all segments on the new URL immediately (guaranteed consistency)? The
signed-URL-expiry case wants the latter; a mirror swap wants the former.
1. **`probe_hint` stays optional.** DAEMON has a `ProbeResult` only on the File-Info path;
capture-take, `velox add`, `addBatch` and restart have none. The engine probes when
it's absent.
2. **One `cancel(discard_partial)`.** `download.cancel` = `cancel(false)`;
`download.remove` = `cancel(true)` for a live task (plus DAEMON's row/file cleanup), or
pure DAEMON-side for an already-terminal one. No `handle.remove()`.
3. **`{restart, keep_partial, abort}` is the whole set.** The engine owns the routine
416 re-probe/re-split and only escalates `on_decision_needed` when that loop fails —
"retry the same range" is exhausted by then.
4. **Per-task 4 Hz is fine.** DAEMON coalesces across tasks for `event.task.progress`
regardless; `on_progress_batch` is a nice-to-have and must not block stage 8.
5. **`refresh_url` restarts all segments on the new URL** after a validating re-probe (the
signed-URL case). Mirror rotation is `spec.mirrors` + the segmenter, not this.
## Review confirms, folded in
- **(a)** `vdm::TaskId` is a cheap-copy hashable value; DAEMON never constructs one — it
only receives it from `start()` / callbacks and passes it back to `set_task_order()`
etc. ✔ (`vdm/ids.hpp`)
- **(b)** DAEMON `mkdir -p`s `save_path`'s parent before `start()`. The engine opens the
file and fails with `Error::path_rejected` if the directory is missing. ✔ (documented on
`DownloadSpec`)
- **(c)** `Checksum::Algo` now has `sha512`, matching the wire `Checksum` set. ✔
- **(d)** `cancel()` always fires `on_state(_, cancelled, nullopt)` then
`on_finished(Err{Error::canceled})` (note: the taxonomy value is `canceled`), in that
order. ✔ (documented on `DownloadHandle::cancel`)