25c171f74295b341208db742970059133a5a6ed5
13
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  samiandClaude Sonnet 5
|
824fa481bb |
daemon: event.* fan-out (D5) + category.list/queue.list (D3) — GUI-ready
The two items aimed at pointing GUI at a real veloxd instead of mockd.
rpc/event_hub — per-subscription fan-out shared by both transports.
subscribe() registers a connection with no interest; set_filter()
(session.subscribe, replaces not adds) turns on event kinds and an
optional per-task id filter; publish() delivers a pre-built
notification to every matching subscriber. session.subscribe on both
UdsServer and WsServer now does the real thing — registers/updates a
subscription, tears it down in close_conn.
sched/scheduler — the on_engine_state hook now actually publishes:
- transition() is the one place a task's row changes state; it reads
the store's own prior row for previousState (authoritative
regardless of engine/scheduler timing), writes the error columns,
and — when a hub is supplied — publishes event.task.state with
{taskId, state, previousState, summary, error}. Wired into every
transition: scheduler-driven (admission -> probing, resume ->
connecting, pause) and engine-reported (on_engine_state).
- progress_snapshot(): one row per task the engine is tracking
(EnginePort::progress(), a new interface method backed by
DownloadHandle::progress()), plus a store side-effect
(Tasks::update_progress) so download.list/get stay current between
state transitions. Returns rows; does NOT publish itself — batching
into one array message is the caller's job, per the schema's
x-maxRateHz: 4 and AGENT-DAEMON.md item 5 ("one message per task per
tick burns a core"). main.cpp's 250 ms timerfd is that caller: one
event.task.progress per tick, only when there's something to say.
dispatcher::on_download_add now publishes event.task.added (schema:
"summary is always present so a client can insert the row without a
follow-up download.get").
store/categories, store/queues — the two D3 handlers GUI's panels
call. category.list projects the six seeded built-ins; queue.list
derives taskIds from tasks.queue_id/queue_position (Queue's own schema
note: a queue's stored row never carries membership, download.update
/ queue.reorder do).
Verified live end to end against tools/testserver: a subscribed client
sees event.task.added on add, then the full event.task.state sequence
(queued -> probing -> connecting -> downloading -> assembling ->
verifying -> complete) with correct previousState at every step, and
real category.list / queue.list results.
Tests: event_hub (filter-by-kind, filter-by-task-id, replace-not-add,
unsubscribe), store_categories_queues, plus new sched_scheduler cases
for event.task.state publishing and progress_snapshot's store
side-effect. 38 daemon/cli tests green; sched_scheduler / event_hub /
ws_server / uds_roundtrip TSan-clean.
deferrals.md: D5 mostly closed (event.task.removed and the
still-unpublished events wait on their owning D3 handlers); D3 down to
the remaining download.* verbs, rules/settings/limiter/schedule,
queue mutation, category mutation, grabber, media.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
|
||
|
samiandClaude Sonnet 5
|
08d7ee9263 |
daemon: wire the engine into veloxd — the vertical slice runs end to end
CORE stage 8 merged, so vdm::Engine is linkable. This closes D4a and narrows D4b: `velox add <url>` now actually downloads. - sched/engine_port_core.hpp — the real EnginePort: forwards to a live vdm::Engine, keeps the DownloadHandle per task for pause/resume/ cancel/provide_auth/decide/refresh_url, drives set_task_order / set_max_active_segments / set_host_segment_cap via engine.segment_budget(). CORE confirmed the admission model: DAEMON decides when to start(); the engine's own download_task calls register_task/set_want internally — DAEMON never touches per-task budget calls. EnginePort gains release(TaskId) so the port drops a handle when the task goes terminal. - rpc/event_loop — EventLoop::post(fn): thread-safe, runs fn on the loop thread next iteration. The marshaller for engine-thread callbacks. - main.cpp — constructs vdm::Engine + EnginePortCore + Scheduler (post_to_loop = loop.post). At startup: reconcile_after_restart() (ADR 0013 §5), reload_config(), tick(). A 1 s timerfd on the loop re-runs tick() (schedule windows, missed nudges); download.add nudges via dispatcher.set_on_mutation. End-to-end verified against tools/testserver: `velox add http://127.0.0.1:.../file/512K` -> task queued -> scheduler admits -> engine downloads 524288 bytes -> complete, file on disk. First byte-path all the way through the project. safepath-adversarial.md: re-verified per its own note — CORE landed O_NOFOLLOW on the target open (core/src/io/sparse_file.cpp), so the leaf-symlink TOCTOU is now closed; residual is down to one intermediate-dir gap (documented post-M1 chase). 36 daemon/cli tests green; scheduler + uds_roundtrip TSan-clean. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig |
||
|
samiandClaude Sonnet 5
|
d93c8e10a0 |
daemon: sched/scheduler — governor <-> store <-> engine, against an EnginePort seam
The Scheduler that D4 was waiting on. Built against CORE's engine
HEADERS (now in main); the real EnginePort and the veloxd wiring wait
for lane/core's stage-8 bodies to reach main (deferrals.md D4a/D4b) —
core/src/task/ is still .gitkeep there, so linking vdm::Engine now
would be an unresolved symbol.
- sched/engine_port — the abstract seam: start/pause/resume/cancel/
provide_auth/decide/refresh_url + the ADR 0011 admission config
(set_task_order / set_max_active_segments / set_host_segment_cap).
Keeps the Scheduler testable without a live engine and the daemon
unbound from the concrete vdm::Engine.
- sched/fake_engine_port — a recording impl for tests.
- sched/scheduler:
* owns the wire-UUID <-> vdm::TaskId map.
* tick(): snapshot queues (schedule window evaluated with an
injectable clock) + non-terminal tasks -> governor.evaluate ->
apply. to_start builds a vdm::task::DownloadSpec from the row and
calls EnginePort::start; to_resume -> resume(); to_pause ->
pause() + writes the pause_reason; priority_order -> set_task_order
over the mapped engine ids. `new` tasks are parked (startMode
manual) and skipped.
* on_engine_state(wire_id, state, err): projects an engine
transition onto the store row (state, pause_reason='auto' when an
error rides a paused transition per ADR 0013 §2, flattened error
columns) so the next tick sees ground truth. This is also the hook
event.task.state will fire from (D5).
* reconcile_after_restart(): CORE-owned states -> queued, paused
keeps its reason (ADR 0013 §5).
* reload_config(): reads connection.maxConcurrentDownloads /
maxActiveSegments + a daemon-local host-cap map, pushes caps to
the engine, updates the governor.
* Deps: injectable local-now clock and a post_to_loop marshaller
(engine callbacks arrive on engine threads; default runs inline
for tests).
Test veloxd.sched_scheduler (ASan+UBSan and TSan clean): admission +
ordering, a slot freeing on completion, queue-stop -> pause
(queue_stopped) then queue-restart -> resume (not a fresh start),
engine auto-pause -> pause_reason 'auto' + never auto-resumed,
reconcile_after_restart, reload_config caps push. 35 daemon/cli tests
green.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
|
||
|
samiandClaude Sonnet 5
|
3db0d01f9d |
daemon: download.add / download.list / download.get behind the store
VeloxDispatcher now takes a store::Db& and three handlers are real:
- download.list -> store::Tasks::list (filter / sort / paging in SQL) ->
to_summary per row. No more empty-table stub.
- download.add -> resolve saveDir (spec, else saveTo.defaultDir; ~ expanded)
and the leaf (spec.filename, else the URL's last segment percent-decoded,
else download.bin) -> fs::resolve_target against canonicalize_root'd
saveTo.allowedRoots. Any path-destination failure is -32011 with the
*original* saveDir in data.path. On success a TaskRow is inserted in
state `queued` (or `new` for startMode "manual") and {taskId, state}
returned. The scheduler that would then admit it is D4.
- download.get -> store::Tasks::get; a real -32010 + data.taskId for an
unknown id, else a TaskDetail (segmentDetail empty until the engine
segments the task, which the schema permits).
util/time.hpp: now_iso() factored out of ws_server.cpp.
main.cpp constructs the dispatcher with the opened db. The three
integration tests build an in-memory migrated db for it; velox.client
now drives the full slice through the CLI — add outside roots -> -32011
with data.path, add into an allowed root -> a task that download.list
shows and download.get details, unknown id -> -32010. Verified with the
real binaries: velox add persists, velox ls shows it, it survives a
daemon restart, /etc is refused.
ASan+UBSan and TSan clean; 34 daemon/cli tests green. deferrals.md:
D2 down to just download.probe; D3 down to categories/queues/rules/
settings/limiter/schedule.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
|
||
|
samiandClaude Sonnet 5
|
7514f5a4c4 |
daemon: correct an unverified cross-lane claim in safepath-adversarial.md
The residual section claimed the leaf/component TOCTOU is "closed in practice by CORE's O_NOFOLLOW open of the final file". Verified: it is not — core/src/io/sparse_file.cpp:77 opens O_WRONLY|O_CREAT|O_CLOEXEC, no O_NOFOLLOW, no O_EXCL. Requested the flags from CORE via PKG/QA. Doc now states the residual is currently OPEN, names the file:line and flags checked and the date, says what actually limits exposure today (0700 parent dirs), and flags this as the boundary where a reader stops checking. Step 5 reworded the same way. Re-verify the flags when the CORE change lands. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig |
||
|
samiandClaude Sonnet 5
|
ab479e7885 |
daemon: fs/safepath — the saveDir/filename path-traversal boundary (security)
veloxd is the one process that turns an untrusted string into a filesystem destination, and via capture.offer that string can come from a web page. CLAUDE.md §4 and the M1 DoD both name this. daemon/docs/safepath-adversarial.md is the spec, written before the code the way EXT did for shouldCapture: 21 rows — .. traversal (A1/A2), absolute-outside-roots (A3), prefix-match confusion (A4), symlink-out (A7), TOCTOU on a created tail (A8), NUL/control bytes in the leaf that CORE's fuzzer hit through Content-Disposition (A9/A10), degenerate and overlong leaves (A11/A13), overlong dir component (A14), symlinked root (A16), destination-is-a-file (A17), and the legitimate cases that must still pass — non-ASCII (A18), redundant "." (A19), trailing space/dot trimming (A20). fs/safepath.cpp: - sanitize_leaf: strip <0x20 and 0x7F, trim ws, strip trailing dots, reject ""/"."/".."/contains-'/', cap 255 UTF-8 bytes on a codepoint boundary. Mirrors core/src/net/content_disposition.cpp. - canonicalize_root: expand ~ and realpath each allowedRoots entry once, so a symlinked root resolves to its target. - resolve_target: reject relative saveDir and any ".." component lexically; if the dir exists, realpath + component-wise containment (a symlink that escapes is caught, one that stays inside passes); if a tail is missing, realpath+check the deepest existing ancestor then create the tail via an openat/mkdirat O_NOFOLLOW walk and re-derive the final path from the fd. Every failure is -32011 with data.path = the *original* saveDir (never the resolved path). Residual TOCTOU on a pre-existing intermediate dir is documented and closed by CORE's O_NOFOLLOW open of the file. veloxd_fs static lib; veloxd_rpc links it for the download.add wiring next. Test veloxd.safepath is the adversarial table, on a real temp tree. ASan+UBSan and TSan clean; 33 daemon/cli tests green. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig |
||
|
samiandClaude Sonnet 5
|
b010139421 |
daemon: sched/ — the concurrency governor + schedule-window evaluation (build step 4)
The scheduling brain, built against CORE's headers (vdm/engine.hpp,
vdm/segment/budget.hpp) — signatures only; the Engine bodies land in
CORE stage 8 and the Scheduler that wires governor <-> store <-> engine
<-> timer comes after that (daemon/docs/deferrals.md D4).
- sched/governor — a pure decision function. In: a snapshot of every
task's coarse RunState and every queue's state (schedule windows
pre-resolved). Out: {to_start, to_resume, to_pause, pause_reasons,
priority_order}. Enforces, all in TASK units per ADR 0011 §1:
connection.maxConcurrentDownloads; the min(that, maxActiveSegments)
clamp (§2); Queue.maxConcurrent; the per-host task cap (§4); and a
stopped queue / closed window runs nothing. Never touches a task
paused for `user` or CORE's `auto` (ADR 0013 §3) — only Schedule /
QueueStopped / AdmissionReconcile are auto-resumable. Deterministic:
main-list before queued-in-queue, then queue order, then FIFO, then
task_id.
- sched/schedule_window — window_open(Schedule, local tm): disabled =>
always open; `once` => date + time match; `periodic` => weekday in
daysOfWeek (empty = every day) + time in [start, stop); null start =>
midnight, null stop => end of day, stop < start => overnight window.
Pure; re-evaluated every tick, no cached instants.
- veloxd_sched static lib; veloxd links it (nothing calls it yet).
Tests (ASan+UBSan and TSan clean): veloxd.sched_window (10 window
cases incl. overnight, once, null bounds), veloxd.sched_governor
(global/clamp/per-queue/per-host caps, stop vs window pause reasons,
resume-only-governor-reasons, auth-pause untouched, admission
reconcile, determinism under shuffled input). 32 daemon/cli tests
green; full tree green.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
|
||
|
samiandClaude Sonnet 5
|
ea8fd7ca8f |
daemon: fold CORE's confirms into the engine-API review
CORE resolved all four (lane/core@3da4cd6): vdm::TaskId is hashable and DAEMON never constructs one; DAEMON mkdir -p's save_path's parent (engine -> Error::path_rejected if missing); sha512 added as the 4th Checksum::Algo so no -32602 at the RPC edge; on_state(cancelled) then on_finished(Err{Error::canceled}) -- note the one-L spelling in the error taxonomy. Also notes rate/token_bucket + Engine::rate_limiter() for the limiter.set wiring. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig |
||
|
samiandClaude Sonnet 5
|
207acb0c00 |
daemon: review CORE's engine API — sign off, answer the five questions
Review of core/docs/engine-api-m1.md (lane/core@d6cf1fe) in daemon/docs/engine-api-review.md. Sign off: nothing forces a sched/ or RPC-dispatch rewrite; the split matches ADR 0011 and ADR 0013. Answers: (1) keep probe_hint optional — DAEMON has a ProbeResult only on the File Info path; (2) one cancel(discard_partial), download.remove is cancel(true) + DAEMON-side row/file cleanup; (3) {restart, keep_partial, abort} suffices if the engine owns the mechanical 416 re-probe/re-split; (4) per-task 4 Hz progress is fine — DAEMON re-batches across tasks for event.task.progress anyway; (5) refresh_url restarts all segments on the new URL (the signed-URL case), mirror rotation is spec.mirrors not refresh_url. Four things to confirm, none blocking: vdm::TaskId copy/hash semantics and that DAEMON never constructs one; who mkdir -p's save_path's parent; sha512 (in the wire Checksum, not the engine enum) rejected at the RPC edge; on_finished(Err{cancelled}) code + ordering vs on_state(_, cancelled, _). Integration timing: wire after sched/ lands. sched/ builds against these signatures in parallel with CORE stage 8. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig |
||
|
samiandClaude Sonnet 5
|
4b279e8271 |
daemon: store/ — SQLite WAL schema + forward-only migrator (build step 3)
The daemon's persistent state. SQLite in WAL mode, foreign keys on,
5 s busy timeout so a writer waits rather than SQLITE_BUSY under the
RPC loop.
- store/sqlite — RAII Db/Stmt over the C API; errors returned as
DbResult<T> (std::expected), never thrown — the RPC loop must not
unwind. transaction() helper: BEGIN / fn / COMMIT, ROLLBACK on error.
- store/migrations/0001_initial.sql — the eight tables from the brief:
settings, categories, queues, tasks, segments, rules, history,
pairings. Notable choices:
* tasks columns project onto proto TaskSummary with no computation;
requested vs effective segments/buffer split per ADR 0010/0012;
pause_reason column per ADR 0013.
* segments end_byte is NOT constrained >= 0 so a whole-file
zero-length download is one row with end_byte = -1 (ADR 0010 B3a).
* pairings stores only token_sha256 — the plaintext token is
returned once from session.pair and never persisted (CLAUDE.md §4).
* indices on tasks(state), (category_id), (queue_id, queue_position),
(created_at), (completed_at) for the "1000 tasks, download.list
under 50 ms" DoD.
* six built-in categories + a Main queue seeded.
- store/migrations — runs every embedded migration past PRAGMA
user_version, each in its own transaction, forward-only. SQL files
are embedded at build time by cmake/embed_migrations.cmake.
Test veloxd.store_migrations (ASan+UBSan and TSan clean): fresh DB ->
head, all tables present, seed rows, FK cascade (segment orphan
rejected, task delete cascades), the end_byte=-1 zero-length case,
idempotent re-run, and forward-only from every released user_version.
Also: daemon/docs/proto-requests-m1.md — P1 marked landed on lane/proto
as 1.4.0 (HandlerError/HandlerResult), to be adopted in rpc/ once that
merges to main; P2 resolved.
Not linked into the running daemon yet — the store is wired to the
dispatcher when download.add/list/get get real bodies, next.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
|
||
|
samiandClaude Sonnet 5
|
e60d6669d8 |
daemon: rpc/ — Unix-socket transport + generated dispatch wiring (build step 1)
First real code in daemon/. veloxd now listens on
$XDG_RUNTIME_DIR/velox/velox.sock (0600, SO_PEERCRED same-UID check),
frames NDJSON, and routes every method through the generated
velox::proto::dispatch(). The CLI and GUI have a server to talk to.
Modules:
- rpc/ndjson.hpp — newline-delimited framing, 8 MiB frame cap, CRLF-
tolerant, partial-tail buffering. Header-only, tested.
- rpc/event_loop — single-threaded poll(2) reactor; never blocks the
loop. stop()/wake() are async-signal-safe (eventfd).
- rpc/runtime_dir — $XDG_RUNTIME_DIR/velox resolution, 0700, owner-checked;
refuses an insecure fallback rather than using /tmp.
- rpc/uds_server — listener + non-blocking per-conn read/write with
backpressure; handles session.hello (protocol-major
check -> -32001, sessionId, transport=uds) and
session.subscribe in the server layer; routes the
rest through dispatch().
- rpc/dispatcher — VeloxDispatcher : proto::Dispatcher, all 39 methods.
download.list answers an empty table; the rest return
"not implemented" (-> -32603) until the store lands.
- main.cpp — abstract-namespace single-instance lock, signal ->
clean shutdown, socket unlinked on exit.
Tests (ASan+UBSan and TSan clean):
- veloxd.ndjson — framing edge cases
- veloxd.uds_roundtrip — real socket: hello ok / version mismatch / empty
list / -32601 / -32700 / pipelined requests, and a
guard on the -32603 collapse documented in P1.
Known gap, filed not worked around: daemon/docs/proto-requests-m1.md P1 —
the generated Dispatcher has no error channel below -32603, so handlers
cannot yet return -32010/-32011/-32013 with their data payloads. The
server layer handles -32001/-32002/-32003 around dispatch(); genuine
in-handler errors collapse to -32603 until PROTO gives handlers a real
error return. Three error fixtures are non-conformant until then.
Not in this drop: rpc/ws_server (next; needs the store for hashed pairing
tokens), store/, sched/, cli/. WS reuses this event loop.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
|
||
|
samiandClaude Sonnet 5
|
67b7b75336 |
daemon: accept ADR 0011 with CORE's sign-off; land the engine API
CORE reviewed and accepted (core/docs/adr-0011-core-response.md, lane/core@7bf5cb5), with three amendments folded in: - yield (slot transfer at the next segment boundary) as the mechanism that satisfies min-1-before-seconds out of a full budget; steal stays slot-neutral as originally written. - "admission implies progress" is bounded-delay (min(next yield boundary, low_speed_secs) + connect_timeout), not immediate — widens the starvation-assertion window from 2s to ~low_speed_secs + connect_timeout (45s). - starved_tasks()/starved_since(TaskId) added to the accessor set; segments_active() and tasks_starved definitions pinned (a 'connecting' segment counts as held, not starved). All five open questions answered (min-1 buildable without inversion, probe pool size 4 outside the budget, drain-not-kill live-apply, ordered TaskId list for priority, 4Hz + starved-edge callback coalescing). Section 6 rewritten: connection.maxActiveSegments landed on the wire in PROTO's ADR 0012 while this was in flight, so the daemon-local stopgap is dropped. daemon/src/sched/ is unblocked. Both docs updated in the rebased vdm-daemon worktree against the frozen 1.0.0 contract. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig |
||
|
samiandClaude Sonnet 5
|
ecedfac903 |
daemon: propose ADR 0011 — admission control vs. the segment budget
Settles the open interface question in AGENT-DAEMON.md before sched/ is written: DAEMON's concurrency governor (global/per-queue/per-host, task units) and CORE's maxActiveSegments (segment units) are two governors on two axes with non-overlapping enforcement — each lane enforces exactly the ceilings counted in the units it owns, with one narrow task-unit clamp against maxActiveSegments. Records the fairness rule DAEMON needs from CORE (min-1-before-seconds) so admission implies progress even when one download could otherwise hold the entire segment budget. Companion daemon/docs/core-requests-m1.md is the concrete engine API ask (budget()/segments_active()/on_budget_changed, live-apply semantics for set_max_active_segments, set_host_segment_cap, probe pool sizing) plus one contract gap for PROTO (connection.maxActiveSegments missing from Settings.schema.json). Status: proposed, pending CORE sign-off on the five open items at the end of the ADR. daemon/src/sched/ does not land until that lands. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig |