b5c6e1f47ded9b121a9584133d17854948b40d18
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  samiandClaude Sonnet 5
|
322a20efa5 |
core: fix SegmentBudget over-admission and add a wait-list wakeup
Root cause of the M7 RSS gap (core/docs/m7-baseline.md: ~70 MB measured against a 60 MB target): SegmentBudget::confirm_slot() only checked a task's own held count against its own target -- never the engine-wide active_ sum. reallocate_locked()'s two-pass fairness allocation does bound sum(target) <= max_active_ at the moment it computes a plan, but that bound says nothing about sum(held): a task can be legitimately holding more than its own just-lowered target for a while (yield is deferred to a segment boundary, never mid-segment -- ADR 0011 A1), and another task's target can correctly rise to claim that capacity before the first task has physically released it. Both confirm_slot() calls could then succeed against their own, individually-correct targets while sum(held) exceeded max_active_ -- tools/bench heap-profile caught this directly: budget.active reading 56-86 against a total of 32. confirm_slot() now also checks active_ < max_active_, unconditionally, as a backstop that doesn't depend on any task's target bookkeeping being in sync with what every other task holds. That creates a liveness question the original design never answered: a task denied only by this new check has a target that's already correct, so it never changes again and reallocate_locked()'s plain "fire a callback when a task's target changes" mechanism never revisits it. Task gained a waiting_for_slot flag, set on exactly this denial; release_slot()/deregister_task() (the only two places that free real capacity) now hand a freed slot directly to the highest-priority waiting task via wake_one_waiter_locked(), if reallocate_locked()'s own plan didn't already produce a callback for anyone. download_task.cpp's fill_slots_locked() needed a matching fix: a woken task's stalled segments (SegState::stalled -- backed off mid-retry, its own release_slot() already called) have no live worker and never surface through Segmenter::assign_slot(), which only hands out unassigned or fresh ranges. fill_slots_locked() now restarts any stalled segment with no live worker directly (bounded by slot_target, same as its assign_slot() loop) before looking for new work; a segment it doesn't get to keeps its own scheduled retry_worker() timer as a second chance. Also fixes a real TSan-caught data race this work surfaced: SegWorker:: speed_bps was written only by its own segment's curl callback and, before Progress.speed_bps's polled-path fix, only ever read from that same thread -- safe without synchronization. snapshot_progress() reading it from whatever thread calls DownloadHandle::progress() broke that invariant (workers_mu's shared_lock protects the workers map's structure, not an individual SegWorker's fields). Now std::atomic<double> with relaxed ordering -- an informational EMA, nothing synchronizes real state on it -- rather than adding a lock to the write side. core/tests/segment/budget_test.cpp adds two tests reproducing the actual gap (budget_active_never_exceeds_max_active_segments_under_concurrent_load, budget_wait_list_wakes_a_task_whose_target_never_changed) plus a sanity baseline (budget_release_wakes_a_denied_waiter), and introduces AsyncFakeTask + TestTimer for the one existing test that drives the budget from multiple concurrent threads -- mirroring production's real dispatch (register_task()'s on_target lambda posts through host.schedule(), download_task.cpp, never a synchronous call) rather than adding reentrancy-guarding machinery to SegmentBudget itself to compensate for a synchronous test double being unlike production. See docs/adr/0017 for the full writeup, including what an earlier version of this fix got wrong chasing a same-thread reentrancy hazard that doesn't actually exist in production. core/docs/m7-baseline.md updated: the RSS number now clears the DoD line (45.41 MiB via heap-profile), root-caused rather than just re-measured. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Q3QrF7rCt21bkAjt9BCDFQ |
||
|
samiandClaude Sonnet 5
|
ef58796d22 |
core: fix Progress.speed_bps reading 0 on the polled path
DAEMON reported Progress.speed_bps reading 0 for the whole life of a live throttled download while downloaded bytes visibly advanced. DAEMON reads progress by polling DownloadHandle::progress() (engine_port_core.hpp), not the on_progress push callback. DownloadTaskState::snapshot_progress() -- the body behind progress() -- never set speed_bps, per-segment speed_bps, or eta_seconds at all; only the event-driven emit_progress_if_due() (which drives the on_progress callback) computed them, from the same live SegWorker::speed_bps EMA seg_data() maintains. snapshot_progress() now reads that same per-worker speed while building its segment list, so a segment with no live worker (idle, paused, complete, failed) correctly reports 0 and a segment with an active transfer reports its real EMA, matching emit_progress_if_due()'s math including the eta_seconds derivation. engine_polled_progress_reports_nonzero_speed reproduces the bug (fails without the fix, confirmed) by polling .progress() -- the same path DAEMON uses -- during a throttled download and asserting speed_bps > 0 once real progress has accumulated. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Q3QrF7rCt21bkAjt9BCDFQ |
||
|
samiandClaude Sonnet 5
|
ef816c21fb |
core: fix quiesce() racing an in-flight write callback (ASan use-after-free)
DownloadTaskState::quiesce() (Engine shutdown / ~Engine, via quiesce_task()) cancelled every live worker's transfer, then immediately cleared `workers` on the calling thread. transfer.cancel() only *requests* the HttpClient worker thread stop the transfer -- it does not wait for that to happen. If that thread was mid write-callback (seg_data -> WriteBuffer::append -> SparseFile::write_at), clearing the map destroyed the SegWorker (and its ring buffer) it was still writing through: a heap-use-after-free, caught by ASan via tools/bench alloc-check, which by design drops its Engine while a download is still active mid-sample. Every other exit path (verify/fail/auto_pause/demote, via begin_drain_locked) already gets this right: cancel, then let each worker remove and flush itself through seg_finished once HttpClient actually confirms the transfer stopped, on the correct thread. quiesce() now does the same instead of tearing the map down itself -- wait on a condition variable, notified from seg_finished right after it erases, until `workers` is empty. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Q3QrF7rCt21bkAjt9BCDFQ |
||
|
samiandClaude Sonnet 5
|
c99d1d9701 |
core: drain-aware hostile-mode handling (etag/416/lying-ranges/mismatch)
Resumes work left mid-session on the M1 hostile-mode matrix. download_task.cpp: - Replace the old cancel-then-clear teardown (cancel_all_transfers_locked / start_assembly_locked) with a single begin_drain_locked()/PendingAction mechanism: cancel every live worker, remember what to do (verify / fail / auto_pause / demote), and let whichever worker's seg_finished finds the worker map empty carry it out. Every sibling still flushes its buffer on the way out, so no buffered-but-unflushed tail is lost when a download finishes or fails while other segments are still mid-transfer. - A 200 where 206 was expected (wrong_status) now checks the response's ETag/Last-Modified against the probe's: a real mismatch asks the user (server_file_changed, "ask, never silently corrupt" -- docs/04 §5); a match means the server just stopped honouring Range for this connection, so demote to one segment and keep going without a round trip (docs/04 §7). - 416 mid-download (stale range metadata) now surfaces as a decision instead of retrying the same now-invalid range to exhaustion. - do_decide's abort path surfaces the actual reason a decision was asked for (last_error) instead of hardcoding server_file_changed, which was mislabeling a 416 abort. engine_test.cpp adds the four hostile modes where a bug means silent corruption rather than a visible failure: etag-changes, 416-always, lies-about-accept-ranges, content-length-mismatch. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Q3QrF7rCt21bkAjt9BCDFQ |
||
|
samiandClaude Sonnet 5
|
91636f8a4d |
core: add the download engine — task machine, DownloadHandle, Engine
Stage 8 of the CORE build order: the bodies behind the DownloadSpec / callback API reviewed in core/docs/engine-api-m1.md. Wires probe -> segment workers -> WriteBuffer -> SparseFile -> .veloxpart.meta -> retry/backoff -> SegmentBudget -> RateLimiter -> callbacks into one event-driven machine. - Engine (src/engine.cpp): owns HttpClient, Prober, SegmentBudget, RateLimiter and one timer jthread (min-heap of scheduled fns). start() builds a task and returns a DownloadHandle; ~Impl quiesces every task before joining the timer so no callback fires during teardown. - DownloadTaskState (src/task/download_task.cpp): one `mu` task lock; a shared_mutex over the worker map for the curl write path; callbacks collected under `mu` and fired after release via a separate deferred queue; weak_from_this() in every async hop. State machine over the CORE-owned EngineState subset, auto-pause on 401/407 and on a 200 where 206 was expected, validated resume via If-Range. - digest (src/task/digest.cpp): OpenSSL EVP hash_file() for the optional post-download checksum; links OpenSSL::Crypto PRIVATE. - Segmenter::release_segment(): hand a paused segment back to the pool unassigned so resume's assign_slot() picks it up instead of splitting a still-"assigned" range and orphaning its front half. - DownloadHandle now names the real control block (vdm::task:: DownloadTaskState, defined only in the engine TU) via a namespace-scope fwd decl and a public-but-effectively-engine-only ctor, replacing the nested State/friend pair. Every public signature is unchanged; DAEMON (vdm-79) confirmed sched/ names only the public API. Fixes found while building the end-to-end suite (tests/task/engine_test.cpp, 9 cases against tools/testserver, green under ASan/UBSan and TSan): - a dropped connection lost its unflushed WriteBuffer tail while advance() had already counted those bytes as done -> a retry resumed past an unwritten hole. Flush on the failure path. - when the byte counters hit total while other workers were still live, teardown dropped their buffered tails. Now: cancel them and let each worker's own seg_finished drain it (the `assembling` state), last one starts verification -- no cross-thread buffer access. - seg_head() let a 401 with credentials present abort before libcurl's resend; now it proceeds once and acts on the final status. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01HPPSGhiArbvQgwC2DNiURS |