b5c6e1f47ded9b121a9584133d17854948b40d18
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  samiandClaude Sonnet 5
|
4f6c0cc9d2 |
daemon: D3's remainder — rules.*, queue.reorder, schedule.*, limiter.*, download.update/refreshUrl
rules.list/rules.upsert: new store/rules.{hpp,cpp} (list in priority order; apply()
upserts+removes atomically, generating an id when absent, per the schema's own "never
leaves the table in a half-valid state"). Found and fixed along the way: migration
0001's rules table had no column for Rule.name at all — every rules.list/.upsert call
failed outright ("no such column: name"), unit tests included, since :memory: migrates
through the same path. Migration 0004 adds it.
queue.reorder: new store::Queues::reorder — taskIds must be an exact permutation of
the queue's current membership (compared as sorted sets) or nothing is written and
-32602 names the queue; a valid permutation rewrites every member's queue_position in
one transaction.
schedule.get/schedule.set: a thin wrapper over the queues.schedule column (already
read since D3b, never independently settable). nextRunAt is deliberately left unset —
computing it needs the same local-time, DST-aware window logic
sched/schedule_window.hpp's window_open() only has half of; called out rather than
approximated, and the field is optional.
limiter.get/limiter.set: backed by the same downloads.speedLimitEnabled/
downloads.speedLimitBps settings keys D9 already wired — one bag of truth, not two.
The new part is reaching the engine: EnginePort/TaskActionPort gain
set_global_speed_limit(bps) (0 = unlimited, TokenBucket's own convention), wired to
Engine::rate_limiter().set_global_limit(). Pushed live on every limiter.set *and* on
Scheduler::reload_config() so a limit from a previous run isn't silently unlimited
again after a restart. applyToRunning is accepted but has no lever to pull
differently — a single shared global bucket has no "next task only" variant. Also
found, not chased further: the schema's "globalBps:0 with enabled:true means 'stop
everything'" is the opposite of what TokenBucket does with rate_bps==0 (unlimited) —
a real discrepancy, but the schema says the GUI must not offer that combination.
download.update: "moving saveDir or filename moves the file on disk in the same
operation" — resolved and root-checked like download.add's destination, then the
.veloxpart/.veloxpart.meta pair (or the finished file, if complete) is moved via
rename, falling back to copy+remove across filesystems, only when the resolved
location actually differs. categoryId/queueId(appended to the new queue's run
order)/description/segments/bufferBytes/checksum apply through new
store::Tasks::apply_update.
download.refreshUrl: same async server-layer special-case as download.probe (a real
network round trip, same 30s deadline). Re-probes, flags contentChanged only when
size or validator are both known and actually differ, persists the new URL and probe
result, and swaps the URL on a live engine handle via a newly-widened
EnginePort::refresh_url (now takes headers too, matching DownloadHandle's real
signature — the seam had silently dropped them).
Found and documented, not fixed: the generated parser collapses "field absent" and
"field explicitly null" to the same nullopt for every optional<T> patch field
(DownloadUpdateParamsPatch, Settings) — both schemas document "an explicit null
clears the field" but neither handler can act on it because the wire distinction is
already gone by the time either sees the parsed struct. A generator-level gap
(PROTO's), not something to hand-route around locally.
Verified against real veloxd + tools/testserver: rules create/list, limiter.set
takes effect and reads back, schedule.set/get round-trips, queue.reorder against real
membership (and rejects a non-permutation), download.update renames+recategorizes a
task, download.refreshUrl swaps a paused task's URL and reports contentChanged
correctly. Full ctest: 55/55 (excluding the pre-existing, unrelated conformance
failure noted two commits back).
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GRDjHGgpYmMoPE2UFbe7pP
|
||
|
samiandClaude Sonnet 5
|
a967eca669 |
daemon: persist engine progress/probe to the store — segments:0 no longer leaks
Engine numbers never reached the store: download.get after a correctly-finished
download reported sizeBytes: null, downloadedBytes: 0, speedBps: 0, resumable:
false, segments: 0, segmentDetail: [] — segments: 0 breaks the frozen contract
(TaskSummary.segments is minimum:1, required).
- Scheduler::tick() now probes (EnginePort::probe) before every start(), persisting
sizeBytes/resumable/validators via Tasks::set_probe_result before a byte moves,
then starts the engine with that ProbeResult as probe_hint.
- Scheduler::persist_progress() (new) writes downloadedBytes/speedBps/segments/
segmentDetail from the engine's Progress. Called from progress_snapshot() (the
250ms tick) *and* once more from on_engine_state right before release()/unmap on
every terminal transition, so a task that finishes between two ticks — the common
case for anything small or fast — still leaves real numbers instead of the
pre-persistence defaults.
- TaskSummary.segments is sourced from segments.size() when the task has any
(matching what actually lands in segmentDetail, per the schema's "exactly
segments entries"), falling back to the engine's effective_segments (budget
slots *held*, not necessarily physical range count) only pre-segmentation.
- Tasks::set_final_bytes tops up on_finished's byte count as a last-resort
backstop.
- store/segments.{cpp,hpp}: read/write access to the segments table behind
TaskDetail.segmentDetail. Wired into daemon/CMakeLists.txt.
- migrations/0002: speed_bps on tasks and segments; fixes segments.state's CHECK
to include 'pending' (0001 omitted it, so a pre-connect snapshot could never be
written).
- store_migrations_test's forward-only loop faked "released version N" by setting
the user_version pragma alone, with no real schema underneath — never exercised
until 0002 existed. Fixed to actually build the db through migrations 1..N first.
Verified against real veloxd + tools/testserver (not just unit tests):
download.list/download.get correct immediately after completion and after a
daemon restart, with saveTo.allowedRoots pointed at an isolated dir.
Observed but not fixed (CORE, not this lane, noted in deferrals.md): Progress.
speed_bps reads back 0 for the whole lifetime of a live throttled download in the
same E2E check, despite downloadedBytes visibly advancing. DAEMON passes it
through unmodified; filed rather than worked around.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GRDjHGgpYmMoPE2UFbe7pP
|
||
|
samiandClaude Sonnet 5
|
824fa481bb |
daemon: event.* fan-out (D5) + category.list/queue.list (D3) — GUI-ready
The two items aimed at pointing GUI at a real veloxd instead of mockd.
rpc/event_hub — per-subscription fan-out shared by both transports.
subscribe() registers a connection with no interest; set_filter()
(session.subscribe, replaces not adds) turns on event kinds and an
optional per-task id filter; publish() delivers a pre-built
notification to every matching subscriber. session.subscribe on both
UdsServer and WsServer now does the real thing — registers/updates a
subscription, tears it down in close_conn.
sched/scheduler — the on_engine_state hook now actually publishes:
- transition() is the one place a task's row changes state; it reads
the store's own prior row for previousState (authoritative
regardless of engine/scheduler timing), writes the error columns,
and — when a hub is supplied — publishes event.task.state with
{taskId, state, previousState, summary, error}. Wired into every
transition: scheduler-driven (admission -> probing, resume ->
connecting, pause) and engine-reported (on_engine_state).
- progress_snapshot(): one row per task the engine is tracking
(EnginePort::progress(), a new interface method backed by
DownloadHandle::progress()), plus a store side-effect
(Tasks::update_progress) so download.list/get stay current between
state transitions. Returns rows; does NOT publish itself — batching
into one array message is the caller's job, per the schema's
x-maxRateHz: 4 and AGENT-DAEMON.md item 5 ("one message per task per
tick burns a core"). main.cpp's 250 ms timerfd is that caller: one
event.task.progress per tick, only when there's something to say.
dispatcher::on_download_add now publishes event.task.added (schema:
"summary is always present so a client can insert the row without a
follow-up download.get").
store/categories, store/queues — the two D3 handlers GUI's panels
call. category.list projects the six seeded built-ins; queue.list
derives taskIds from tasks.queue_id/queue_position (Queue's own schema
note: a queue's stored row never carries membership, download.update
/ queue.reorder do).
Verified live end to end against tools/testserver: a subscribed client
sees event.task.added on add, then the full event.task.state sequence
(queued -> probing -> connecting -> downloading -> assembling ->
verifying -> complete) with correct previousState at every step, and
real category.list / queue.list results.
Tests: event_hub (filter-by-kind, filter-by-task-id, replace-not-add,
unsubscribe), store_categories_queues, plus new sched_scheduler cases
for event.task.state publishing and progress_snapshot's store
side-effect. 38 daemon/cli tests green; sched_scheduler / event_hub /
ws_server / uds_roundtrip TSan-clean.
deferrals.md: D5 mostly closed (event.task.removed and the
still-unpublished events wait on their owning D3 handlers); D3 down to
the remaining download.* verbs, rules/settings/limiter/schedule,
queue mutation, category mutation, grabber, media.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
|
||
|
samiandClaude Sonnet 5
|
08d7ee9263 |
daemon: wire the engine into veloxd — the vertical slice runs end to end
CORE stage 8 merged, so vdm::Engine is linkable. This closes D4a and narrows D4b: `velox add <url>` now actually downloads. - sched/engine_port_core.hpp — the real EnginePort: forwards to a live vdm::Engine, keeps the DownloadHandle per task for pause/resume/ cancel/provide_auth/decide/refresh_url, drives set_task_order / set_max_active_segments / set_host_segment_cap via engine.segment_budget(). CORE confirmed the admission model: DAEMON decides when to start(); the engine's own download_task calls register_task/set_want internally — DAEMON never touches per-task budget calls. EnginePort gains release(TaskId) so the port drops a handle when the task goes terminal. - rpc/event_loop — EventLoop::post(fn): thread-safe, runs fn on the loop thread next iteration. The marshaller for engine-thread callbacks. - main.cpp — constructs vdm::Engine + EnginePortCore + Scheduler (post_to_loop = loop.post). At startup: reconcile_after_restart() (ADR 0013 §5), reload_config(), tick(). A 1 s timerfd on the loop re-runs tick() (schedule windows, missed nudges); download.add nudges via dispatcher.set_on_mutation. End-to-end verified against tools/testserver: `velox add http://127.0.0.1:.../file/512K` -> task queued -> scheduler admits -> engine downloads 524288 bytes -> complete, file on disk. First byte-path all the way through the project. safepath-adversarial.md: re-verified per its own note — CORE landed O_NOFOLLOW on the target open (core/src/io/sparse_file.cpp), so the leaf-symlink TOCTOU is now closed; residual is down to one intermediate-dir gap (documented post-M1 chase). 36 daemon/cli tests green; scheduler + uds_roundtrip TSan-clean. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig |
||
|
samiandClaude Sonnet 5
|
d93c8e10a0 |
daemon: sched/scheduler — governor <-> store <-> engine, against an EnginePort seam
The Scheduler that D4 was waiting on. Built against CORE's engine
HEADERS (now in main); the real EnginePort and the veloxd wiring wait
for lane/core's stage-8 bodies to reach main (deferrals.md D4a/D4b) —
core/src/task/ is still .gitkeep there, so linking vdm::Engine now
would be an unresolved symbol.
- sched/engine_port — the abstract seam: start/pause/resume/cancel/
provide_auth/decide/refresh_url + the ADR 0011 admission config
(set_task_order / set_max_active_segments / set_host_segment_cap).
Keeps the Scheduler testable without a live engine and the daemon
unbound from the concrete vdm::Engine.
- sched/fake_engine_port — a recording impl for tests.
- sched/scheduler:
* owns the wire-UUID <-> vdm::TaskId map.
* tick(): snapshot queues (schedule window evaluated with an
injectable clock) + non-terminal tasks -> governor.evaluate ->
apply. to_start builds a vdm::task::DownloadSpec from the row and
calls EnginePort::start; to_resume -> resume(); to_pause ->
pause() + writes the pause_reason; priority_order -> set_task_order
over the mapped engine ids. `new` tasks are parked (startMode
manual) and skipped.
* on_engine_state(wire_id, state, err): projects an engine
transition onto the store row (state, pause_reason='auto' when an
error rides a paused transition per ADR 0013 §2, flattened error
columns) so the next tick sees ground truth. This is also the hook
event.task.state will fire from (D5).
* reconcile_after_restart(): CORE-owned states -> queued, paused
keeps its reason (ADR 0013 §5).
* reload_config(): reads connection.maxConcurrentDownloads /
maxActiveSegments + a daemon-local host-cap map, pushes caps to
the engine, updates the governor.
* Deps: injectable local-now clock and a post_to_loop marshaller
(engine callbacks arrive on engine threads; default runs inline
for tests).
Test veloxd.sched_scheduler (ASan+UBSan and TSan clean): admission +
ordering, a slot freeing on completion, queue-stop -> pause
(queue_stopped) then queue-restart -> resume (not a fresh start),
engine auto-pause -> pause_reason 'auto' + never auto-resumed,
reconcile_after_restart, reload_config caps push. 35 daemon/cli tests
green.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upd9WhG9oppieig5nRDLig
|