0468b0176aeeefdc11b72908f9d6d8cc2c23a84f
2
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  samiandClaude Sonnet 5
|
322a20efa5 |
core: fix SegmentBudget over-admission and add a wait-list wakeup
Root cause of the M7 RSS gap (core/docs/m7-baseline.md: ~70 MB measured against a 60 MB target): SegmentBudget::confirm_slot() only checked a task's own held count against its own target -- never the engine-wide active_ sum. reallocate_locked()'s two-pass fairness allocation does bound sum(target) <= max_active_ at the moment it computes a plan, but that bound says nothing about sum(held): a task can be legitimately holding more than its own just-lowered target for a while (yield is deferred to a segment boundary, never mid-segment -- ADR 0011 A1), and another task's target can correctly rise to claim that capacity before the first task has physically released it. Both confirm_slot() calls could then succeed against their own, individually-correct targets while sum(held) exceeded max_active_ -- tools/bench heap-profile caught this directly: budget.active reading 56-86 against a total of 32. confirm_slot() now also checks active_ < max_active_, unconditionally, as a backstop that doesn't depend on any task's target bookkeeping being in sync with what every other task holds. That creates a liveness question the original design never answered: a task denied only by this new check has a target that's already correct, so it never changes again and reallocate_locked()'s plain "fire a callback when a task's target changes" mechanism never revisits it. Task gained a waiting_for_slot flag, set on exactly this denial; release_slot()/deregister_task() (the only two places that free real capacity) now hand a freed slot directly to the highest-priority waiting task via wake_one_waiter_locked(), if reallocate_locked()'s own plan didn't already produce a callback for anyone. download_task.cpp's fill_slots_locked() needed a matching fix: a woken task's stalled segments (SegState::stalled -- backed off mid-retry, its own release_slot() already called) have no live worker and never surface through Segmenter::assign_slot(), which only hands out unassigned or fresh ranges. fill_slots_locked() now restarts any stalled segment with no live worker directly (bounded by slot_target, same as its assign_slot() loop) before looking for new work; a segment it doesn't get to keeps its own scheduled retry_worker() timer as a second chance. Also fixes a real TSan-caught data race this work surfaced: SegWorker:: speed_bps was written only by its own segment's curl callback and, before Progress.speed_bps's polled-path fix, only ever read from that same thread -- safe without synchronization. snapshot_progress() reading it from whatever thread calls DownloadHandle::progress() broke that invariant (workers_mu's shared_lock protects the workers map's structure, not an individual SegWorker's fields). Now std::atomic<double> with relaxed ordering -- an informational EMA, nothing synchronizes real state on it -- rather than adding a lock to the write side. core/tests/segment/budget_test.cpp adds two tests reproducing the actual gap (budget_active_never_exceeds_max_active_segments_under_concurrent_load, budget_wait_list_wakes_a_task_whose_target_never_changed) plus a sanity baseline (budget_release_wakes_a_denied_waiter), and introduces AsyncFakeTask + TestTimer for the one existing test that drives the budget from multiple concurrent threads -- mirroring production's real dispatch (register_task()'s on_target lambda posts through host.schedule(), download_task.cpp, never a synchronous call) rather than adding reentrancy-guarding machinery to SegmentBudget itself to compensate for a synchronous test double being unlike production. See docs/adr/0017 for the full writeup, including what an earlier version of this fix got wrong chasing a same-thread reentrancy hazard that doesn't actually exist in production. core/docs/m7-baseline.md updated: the RSS number now clears the DoD line (45.41 MiB via heap-profile), root-caused rather than just re-measured. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Q3QrF7rCt21bkAjt9BCDFQ |
||
|
samiandClaude Sonnet 5
|
b60d4e6f5b |
core: add tools/bench (throughput/load/alloc-check) and record the M7 baseline
Three subcommands in one binary, driving vdm::Engine directly (docs/04 §8): - throughput: a single download against a fast local origin (support/local_server.hpp, busybox httpd), reporting Mbps/CPU%/RSS. Gates on --require-mbps/--max-cpu-pct only when passed, so the ctest smoke registration stays a correctness check, not a hardware-dependent perf gate -- the real 1-Gbit-link sign-off is a manual/CI job (see the file's header comment). - load: N concurrent tasks against tools/testserver's `throttled` mode (support/testserver_client.hpp), reporting peak RSS via getrusage(). Paced externally rather than through the engine's own rate::RateLimiter or busybox: the limiter's pause/resume path allocates on every throttle event (would contaminate alloc-check's measurement) and under heavy segment contention was found to starve individual tasks indefinitely (see docs/adr/0016, added here); busybox couldn't sustain the DoD's ~160 concurrent connections (20 tasks * default_segments=8) reliably. The ctest registration runs at reduced concurrency under sanitizer presets -- see the CMakeLists.txt comment and the ADR's postscript. - alloc-check: operator new/delete overridden process-wide, sampling the allocation count across a steady mid-transfer window against a paced tools/testserver origin. Caught a real bug in the same change (see the http_client.cpp commit) and, by dropping its Engine mid-download to end cleanly, also surfaced the quiesce() use-after-free (see that commit). core/docs/m7-baseline.md records actual measured numbers against the M1/M7 DoD lines, including where they don't clear yet (RSS ~70 MB vs a 60 MB target; throughput/CPU only measured on loopback, no 1 Gbit link available here) rather than rounding them away. docs/adr/0016 documents a rate::RateLimiter fairness gap found building the load subcommand: a single shared TokenBucket under heavy segment contention has no fairness ordering across its peek/commit race and can starve a waiter well past what its configured rate implies. Filed as a follow-up (it's a core/src/rate design question, not a tools/bench one) rather than fixed here, along with a related TSan-only load-test straggler that could not be root-caused in this environment. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Q3QrF7rCt21bkAjt9BCDFQ |