core: post the engine API for DAEMON review (pre-stage-7)
The download entry point the AGENT-CORE brief asked for on day one and that slipped. DAEMON has an RPC surface and a store and, until this is agreed, nothing in velox::core to call. vdm/task/download.hpp — DownloadSpec (the resolved subset DAEMON hands in: absolute save_path, verbatim browser headers, requested segments/buffer, optional probe_hint / checksum / auth, allow_resume), EngineState (the CORE-owned subset of the wire TaskState), Progress / SegmentProgress, DownloadCallbacks (on_progress <=4 Hz, on_state for every transition incl. auto-pauses, on_auth_required, on_decision_needed, on_finished last), DownloadHandle (pause/resume/cancel — idempotent per the ADR 0013 signature — plus provide_auth / decide / refresh_url, and synchronous state()/progress() snapshots). vdm/engine.hpp — Engine: start(spec, callbacks) -> handle, segment_budget() (DAEMON's sched/ admission surface, ADR 0011), live connection.* setters, a standalone probe() on the pool outside the segment budget. core/docs/engine-api-m1.md — the review doc: field semantics, the state machine, threading/lifetime rules (which thread callbacks arrive on, what is legal from inside one, handle/engine lifetime), the shared-`paused` idempotency contract as a signature, and five open questions for DAEMON. Value types compile and are covered by api_compiles_test; Engine / DownloadHandle bodies land in stage 8, built against whatever DAEMON signs off here. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01HPPSGhiArbvQgwC2DNiURS
This commit is contained in:
Claude Sonnet 5
@@ -0,0 +1,158 @@
|
||||
# CORE → DAEMON — the engine API for review (M1)
|
||||
|
||||
**Status: for review.** This is the `libveloxcore` public surface `veloxd` links and calls
|
||||
to actually run a download. It is the thing the AGENT-CORE brief asked for on day one and
|
||||
that slipped: DAEMON has an RPC surface and a store and, until this is agreed, nothing to
|
||||
call. Sketch headers: `core/include/vdm/engine.hpp`, `core/include/vdm/task/download.hpp`
|
||||
(both compile now; `Engine` / `DownloadHandle` bodies land in CORE stage 8). Nothing here
|
||||
touches `contracts/` — DAEMON projects these callbacks onto `TaskSummary` / `TaskDetail` /
|
||||
`event.*`.
|
||||
|
||||
Please review the field semantics, the threading/lifetime rules, and the
|
||||
pause/resume/cancel contract, and raise anything that would force a `daemon/src/sched/` or
|
||||
RPC-dispatch rewrite later. Open questions are at the bottom.
|
||||
|
||||
---
|
||||
|
||||
## 1. `DownloadSpec` — what DAEMON hands the engine
|
||||
|
||||
DAEMON has already run the rules engine, picked the category folder, canonicalised the
|
||||
path and checked it against the allowed roots (`-32011` is DAEMON's error, raised before
|
||||
`start()`), and resolved the filename. The engine's spec is the concrete result.
|
||||
|
||||
| field | who fills it | notes |
|
||||
|---|---|---|
|
||||
| `url`, `mirrors` | DAEMON | `mirrors` are alternative URLs for the *same bytes*; the segmenter's requeue prefers a different host after 3 connection failures (docs/04 §3). |
|
||||
| `headers`, `cookies`, `referrer`, `user_agent` | DAEMON, verbatim from the capture | replayed on every segment request and on the probe. |
|
||||
| `save_path` | DAEMON | **absolute and final.** The engine never canonicalises or root-checks. `<save_path>.veloxpart` and `.veloxpart.meta` sit beside it; on success the part file is renamed in place. |
|
||||
| `segments` | user / `connection.maxSegmentsPerDownload` | requested upper bound 1–32; the engine lowers it to the per-host cap and to 1 for a non-resumable source. Effective value comes back in `Progress.effective_segments`. |
|
||||
| `buffer_bytes` | user / `connection.bufferBytes` | requested per-segment; silently reduced to fit `maxTotalBufferBytes` across all live segments. Effective value in `Progress.effective_buffer_bytes` → your `TaskDetail.effectiveBufferBytes`. |
|
||||
| `proxy`, `auth` | DAEMON | `auth` carries credentials only if known up front (Secret Service). Leave `scheme == none` to get an `on_auth_required` on a 401/407 instead. |
|
||||
| `checksum` | user (`download.add.checksum`) | `{algo, hex}`. Verified during `verifying`; a mismatch is a terminal `failed` with `Error::checksum_mismatch`. |
|
||||
| `probe_hint` | DAEMON | the `ProbeResult` you already got for the File Info dialog. Supplying it skips the engine's own probe — the task starts in `connecting`, not `probing`. The engine still revalidates with `If-Range` on resume. |
|
||||
| `allow_resume` | DAEMON | `true`: if a CRC-valid `.veloxpart.meta` sits beside `save_path`, resume from it. `false`: start fresh, overwrite. Your restart flow (ADR 0013 §5) sets this per task. |
|
||||
| `max_retries` | user / default 10 | per segment, before `failed` with `Error::max_retries_exhausted`. |
|
||||
|
||||
`start()` returns immediately. It never throws and never blocks on the network; a bad URL,
|
||||
DNS failure, or unwritable `save_path` is delivered through `on_finished`.
|
||||
|
||||
## 2. The state machine the engine drives
|
||||
|
||||
`EngineState` is the CORE-owned subset of the wire `TaskState` (ADR 0013 §1):
|
||||
|
||||
```
|
||||
probing ─▶ connecting ⇄ downloading ─▶ assembling ─▶ verifying ─▶ complete
|
||||
│ │ ▲ │ │ (M4 mux; a no-op rename in M1)
|
||||
│ │ └─ retry_wait ┘
|
||||
▼ ▼
|
||||
(any) ──────▶ paused ──(resume)──▶ connecting
|
||||
(any CORE state) ──────────────────▶ failed (terminal, engine-initiated)
|
||||
(any state, on cancel()) ──────────▶ cancelled (terminal, DAEMON-initiated)
|
||||
```
|
||||
|
||||
`new` and `queued` are yours; the engine never emits them. `start()` corresponds to your
|
||||
`queued → probing`. Every transition is reported through `on_state(from, to, error?)`,
|
||||
including the auto-pauses (§4) and the terminals. `previousState` on your
|
||||
`event.task.state` maps straight from the `from` argument.
|
||||
|
||||
## 3. Threading and lifetime
|
||||
|
||||
- **Callbacks run on an engine thread** — a transfer worker, the progress timer, or a
|
||||
dispatch thread — **never** the thread that called `start()` / `pause()` / etc.
|
||||
- **Per task, callbacks are serialised.** You will never get two callbacks for the same
|
||||
handle at once. Across tasks they run concurrently.
|
||||
- **A callback must not block.** It runs on a thread doing real transfer work; a slow
|
||||
callback stalls that work. Hand off to your own queue/loop.
|
||||
- **A callback must not re-enter the same handle synchronously** — no `pause()` from
|
||||
inside `on_state`, etc. Post it. (Calling into a *different* handle, or into
|
||||
`segment_budget()`, is fine.)
|
||||
- **`on_finished` is always the last callback** for a task. After it returns the engine
|
||||
makes no further callbacks for that handle and the handle's control methods are no-ops.
|
||||
- **The handle is copyable and thread-safe.** Dropping the last copy does **not** cancel —
|
||||
the task runs on. Call `cancel()` to stop it. (DAEMON holds the handle for the task's
|
||||
life anyway.)
|
||||
- **`Engine` must outlive every handle.** `~Engine()` cancels all running tasks and joins
|
||||
their workers before returning — expect it to block briefly.
|
||||
- **Logging**: the engine writes through `vdm::set_log_sink()` (a `core/util` global).
|
||||
Install your sink once at startup; the engine never opens a file itself.
|
||||
|
||||
## 4. `paused` is shared, and idempotency is the contract (ADR 0013 §2)
|
||||
|
||||
Both sides put a task in `paused`, for disjoint reasons:
|
||||
|
||||
- **DAEMON-initiated**: `handle.pause()` — user pause, a schedule window closing, a queue
|
||||
stop, `Queue.onComplete`, the admission governor reconciling a lowered
|
||||
`maxActiveSegments`.
|
||||
- **Engine auto-pause**: `on_auth_required` (401/407), `on_decision_needed`
|
||||
(`server_file_changed` / stale range), disk full. The engine transitions to `paused`
|
||||
on its own and fires `on_state(_, paused, ErrorInfo{...})` — the same path as any other
|
||||
transition. `ErrorInfo.code` present ⇒ engine-initiated; absent ⇒ you did it. That is
|
||||
the only discriminator, and it is what your `error`-on-`paused` widening (open item 3 of
|
||||
ADR 0013) carries on the wire.
|
||||
|
||||
**Idempotency, now a signature:**
|
||||
|
||||
| call | already in that state / terminal | otherwise |
|
||||
|---|---|---|
|
||||
| `pause()` | no-op, no error | stop new segment requests, flush + `fdatasync` in-flight buffers, write `.veloxpart.meta`, release the budget slots, `on_state(_, paused, nullopt)`. Bounded by the slowest in-flight flush. |
|
||||
| `resume()` | no-op if not `paused`; no-op if terminal | revalidate with `If-Range`, re-acquire budget slots, `paused → connecting`, resume from the sidecar offsets. |
|
||||
| `cancel(discard_partial)` | no-op if already terminal | stop everything, `on_state(_, cancelled, nullopt)`, `on_finished(Err{cancelled})`. `discard_partial` also unlinks `.veloxpart[.meta]` — wire this to `download.remove {deleteFile}`. |
|
||||
|
||||
`resume()` after an auto-pause for `auth_required` **without** a preceding
|
||||
`provide_auth()` is a no-op — the task stays paused. This is ADR 0013 §3's "resume must
|
||||
not cross reasons", enforced on CORE's side: the scheduler cannot accidentally un-pause a
|
||||
task waiting on credentials.
|
||||
|
||||
- **`provide_auth(user, pass, remember)`** — acts only while the task is auto-paused for
|
||||
auth; supplies the credential for the retry and resumes. `remember` asks *you* to
|
||||
persist to the Secret Service; the engine never stores it. No-op otherwise.
|
||||
- **`decide(Decision)`** — acts only while auto-paused for a decision. `restart` discards
|
||||
the partial and re-downloads; `keep_partial` continues against what is on disk (the
|
||||
user's stated risk); `abort` → `failed`. No-op otherwise.
|
||||
- **`refresh_url(url, headers)`** — IDM's "Refresh Download Address": swap the URL on a
|
||||
live or paused task without losing progress (a fresh signed URL). Maps to
|
||||
`download.refreshUrl`.
|
||||
|
||||
## 5. Progress
|
||||
|
||||
`on_progress` is coalesced to **≤ 4 Hz per task** inside the engine — the same cadence as
|
||||
`event.task.progress`, so your batcher can forward without re-throttling. It carries
|
||||
aggregate `downloaded` / `speed_bps` / `eta_seconds`, the effective segment count and
|
||||
buffer size, and a `SegmentProgress[]` (index, inclusive `[start,end]`, `completed`,
|
||||
per-segment speed, state) for the GUI's segment bars. `total` is absent for a chunked
|
||||
source until the stream ends.
|
||||
|
||||
`handle.state()` and `handle.progress()` are synchronous lock-guarded snapshots for
|
||||
`download.get` / `download.list` — call them any time, including from your RPC thread.
|
||||
|
||||
## 6. What the engine does NOT do
|
||||
|
||||
- No filename resolution, no category matching, no path canonicalisation, no allowed-root
|
||||
check — all DAEMON, before `start()`.
|
||||
- No queueing, scheduling, priority, or "when queue completes" — DAEMON, via
|
||||
`segment_budget().set_task_order()` and by choosing when to call `start()` / `pause()`.
|
||||
- No persistence beyond `.veloxpart.meta`. On a daemon restart the engine knows nothing;
|
||||
you reload from SQLite, rewrite CORE-owned states to `queued`, and re-`start()` with
|
||||
`allow_resume = true` (ADR 0013 §5).
|
||||
- No credential storage. Ever (`CLAUDE.md` §4).
|
||||
|
||||
---
|
||||
|
||||
## Open questions for DAEMON
|
||||
|
||||
1. **`DownloadSpec.probe_hint`** — do you want to pass the File-Info `ProbeResult` in, or
|
||||
would you rather the engine always probe (one code path, ~1 extra round trip)? The
|
||||
sketch supports both; picking one simplifies stage 8.
|
||||
2. **`cancel(discard_partial)`** vs a separate `handle.remove()` — the wire has
|
||||
`download.cancel` and `download.remove {deleteFile}` as two methods. One call with a
|
||||
flag, or two?
|
||||
3. **`on_decision_needed` granularity** — is `{restart, keep_partial, abort}` the right
|
||||
choice set for `server_file_changed`, or do you also need "retry the same range once
|
||||
more" as a distinct option for the stale-416 case?
|
||||
4. **Progress cadence** — 4 Hz per task matches the wire. With 20 active tasks that is 80
|
||||
callbacks/s on the engine's timer thread. Acceptable, or do you want a single
|
||||
`on_progress_batch(span<Progress>)` so the engine coalesces across tasks too?
|
||||
5. **`refresh_url` while `downloading`** — should an in-flight segment finish on the old
|
||||
URL and only new/retried segments use the new one (less disruption), or should the
|
||||
engine restart all segments on the new URL immediately (guaranteed consistency)? The
|
||||
signed-URL-expiry case wants the latter; a mirror swap wants the former.
|
||||
Reference in New Issue
Block a user