core: redo bufferBytes default and RSS reconciliation (per-segment)
First pass counted one buffer per download; it is one per segment. 20 active downloads at the default 8 segments = 160 buffers, so at 20 tasks the binding constraint is the global cap, not the per-segment default — 256 MiB and the "<=60 MB RSS / 20 downloads" target (line 125) cannot both hold whatever the default is. Floor (64 KiB) and ceiling (16 MiB) unchanged — the 256/64 unreachability argument is stronger under per-segment accounting. Changes: - default 1 MiB (was 2): with the cap below, 32 live segments x 1 MiB = 32 MiB buffers -> ~45-50 MiB RSS, line 125 holds with margin. - NEW maxActiveSegments (default 32): a global concurrent-segment cap is the actual mechanism that bounds "20 active downloads"; docs/01 §2 implies it, docs/04 never states it. Without it no buffer policy hits 60 MB. - maxTotalBufferBytes 128 MiB (was 256) and it must be ADDED to the contract — currently absent, so the clamp CORE implements has no wire representation and Options can't show/set it. Folded into B2a. - line 125: keep 60 MB "given maxActiveSegments=32 and default buffers", or explicitly raise to 120 MB — ADR records which. Flagged that changing it is a defensible outcome CORE owns, not a number that quietly loses. - bufferBytes bounds are in FOUR schema files, not three: Settings.schema.json connection.bufferBytes also has 4096-8388608. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01HPPSGhiArbvQgwC2DNiURS
This commit is contained in:
Claude Sonnet 5
+111
-86
@@ -1,116 +1,141 @@
|
||||
# CORE → PROTO — `bufferBytes` range (answer to the freeze question)
|
||||
# CORE → PROTO — `bufferBytes` range + the RSS-budget reconciliation
|
||||
|
||||
`contracts/` froze `bufferBytes` at **4 KiB – 8 MiB**; `docs/04` §4 line 70 says
|
||||
**64 KiB – 64 MiB, default 4 MiB**. They disagree on the floor (16×), the ceiling (8×),
|
||||
and the RSS budget can't hold the default. CORE owns the ring buffer, the 32-segment
|
||||
ceiling and `max_total_buffer_bytes`, so here is the range that is actually right and why.
|
||||
PROTO to land schema + `docs/04` §4 + an ADR together.
|
||||
`contracts/` froze `bufferBytes` at **4 KiB – 8 MiB** in four places; `docs/04` §4 line 70
|
||||
says **64 KiB – 64 MiB, default 4 MiB**. The floor and ceiling are settled below (64 KiB /
|
||||
16 MiB — my earlier reasoning holds). The default and `max_total_buffer_bytes` are
|
||||
**not** — my first pass counted one buffer per download; it is one per *segment*
|
||||
("Per-segment ring buffer", line 68; "Write buffer per connection", line 70), which
|
||||
changes the whole RSS picture. This version fixes that.
|
||||
|
||||
PROTO to land schema + `docs/04` §4 + §8 + an ADR together.
|
||||
|
||||
## Recommendation
|
||||
|
||||
| Field | Value | |
|
||||
| Knob | Value | |
|
||||
|---|---|---|
|
||||
| `bufferBytes` minimum | **65536** (64 KiB) | from `docs/04`; the frozen 4 KiB is wrong — see below |
|
||||
| `bufferBytes` maximum | **16777216** (16 MiB) | 8 MiB is defensible if simplicity wins; 64 MiB is not |
|
||||
| `bufferBytes` default | **2097152** (2 MiB) | 4 MiB busts the RSS DoD — see below |
|
||||
| `max_total_buffer_bytes` default | **268435456** (256 MiB), unchanged | the real ceiling; the backstop for everything |
|
||||
| `bufferBytes` minimum | **65536** (64 KiB) | frozen 4 KiB is below one libcurl callback — see §Floor |
|
||||
| `bufferBytes` maximum | **16777216** (16 MiB) | see §Ceiling; the `256/64` unreachability argument is *stronger* per-segment |
|
||||
| `bufferBytes` default | **1048576** (1 MiB) | see §Default+RSS; 2 MiB and 4 MiB both bust line 125 |
|
||||
| `maxActiveSegments` (**new**) | **32** | global cap on concurrently-transferring segments — the actual bound on "20 active downloads" |
|
||||
| `maxTotalBufferBytes` | **134217728** (128 MiB), down from 256 | backstop for the buffer knob; RSS-safe given the 1 MiB default + the segment cap |
|
||||
|
||||
## Why the floor is 64 KiB, not 4 KiB
|
||||
## Floor — 64 KiB, not 4 KiB
|
||||
|
||||
The buffer's first job is to coalesce libcurl write-callback deliveries into one `pwrite`.
|
||||
Over HTTP/2 a single write callback is routinely 16–64 KiB, and can be up to ~256 KiB. A
|
||||
4 KiB ring buffer is **smaller than one callback**: every callback would have to flush
|
||||
mid-call (or loop), so the "one `pwrite` per fill" design degrades to a syscall per curl
|
||||
chunk — the exact thing the buffer exists to avoid. 64 KiB (≈4 typical chunks) is the
|
||||
smallest floor that still buys anything. 4 KiB is a page size that wandered into a
|
||||
throughput knob.
|
||||
Over HTTP/2 a single write callback is routinely 16–64 KiB, up to ~256 KiB. A 4 KiB ring
|
||||
buffer is **smaller than one callback**: the "one `pwrite` per fill" design degrades to a
|
||||
syscall per curl chunk — the exact thing the buffer exists to prevent. 64 KiB (≈4 typical
|
||||
chunks) is the smallest floor that buys anything. 4 KiB is a page size that wandered into
|
||||
a throughput knob.
|
||||
|
||||
## Why the ceiling is ~16 MiB, not 64 MiB
|
||||
## Ceiling — 16 MiB, not 64 MiB
|
||||
|
||||
Two things the buffer buys, past coalescing:
|
||||
Two things the buffer buys past coalescing:
|
||||
|
||||
1. **Large sequential writes.** On NVMe, write throughput as a function of write size is
|
||||
flat by ~1–4 MiB. From 4 MiB to 8 MiB you gain a little on syscall overhead at
|
||||
multi-Gbit; past 8 MiB there is **no throughput left to get** — you are only adding
|
||||
`fdatasync` latency and page-cache pressure (a 40 GB ISO must not evict the user's
|
||||
working set — `docs/04` §4).
|
||||
2. **Absorbing a disk stall without stalling the socket.** This is the only reason to go
|
||||
above 8 MiB. Fast link + bursty storage (HDD, SMR, USB, a network mount): 1 Gbit is
|
||||
~125 MB/s, so 16 MiB per segment ≈ 130 ms of write-stall cover; across 4–8 segments,
|
||||
~0.5–1 s aggregate — enough to ride out a seek storm. Beyond 16 MiB the marginal cover
|
||||
isn't worth the cache footprint.
|
||||
1. **Large sequential writes.** On NVMe, throughput vs. write size is flat by ~1–4 MiB.
|
||||
4→8 MiB gains a little on syscall overhead at multi-Gbit; past 8 MiB there is no
|
||||
throughput left to get, only `fdatasync` latency and page-cache pressure (a 40 GB ISO
|
||||
must not evict the user's working set — `docs/04` §4).
|
||||
2. **Absorbing a disk stall without stalling the socket** — the only reason to exceed
|
||||
8 MiB. Fast link + bursty storage (HDD, SMR, USB, network mount): 1 Gbit ≈ 125 MB/s,
|
||||
so 16 MiB/segment ≈ 130 ms of write-stall cover, ~0.5 s across 4 segments — enough to
|
||||
ride out a seek storm. Beyond 16 MiB the marginal cover isn't worth the footprint.
|
||||
|
||||
So: **8 MiB captures all the throughput; 8–16 MiB is stall-absorption headroom for the
|
||||
fast-pipe/slow-disk power user; >16 MiB is waste.**
|
||||
**8 MiB captures all throughput; 8–16 MiB is stall-absorption headroom for the
|
||||
fast-pipe/slow-disk case; >16 MiB is waste.**
|
||||
|
||||
## The arithmetic you asked about: 32 × 64 MiB vs a 256 MiB cap
|
||||
### `maxTotalBufferBytes / segments` unreachability — worse per-segment
|
||||
|
||||
`max_total_buffer_bytes` (256 MiB) caps the sum of every active segment's buffer across
|
||||
every active task. The effective per-segment buffer is
|
||||
`effective = clamp(requested, 64 KiB, floor(maxTotalBufferBytes / live_segment_count))`.
|
||||
|
||||
```
|
||||
effective = clamp( requested,
|
||||
64 KiB,
|
||||
floor(max_total_buffer_bytes / active_segment_count) )
|
||||
```
|
||||
With a **64 MiB** ceiling and the old 256 MiB cap, 64 MiB is unreachable once total live
|
||||
segments exceed 4 (256/64). Under correct per-segment accounting "4 total segments" is
|
||||
*half of one default download* (8 segments) — so the frozen-doc ceiling is unreachable in
|
||||
essentially every real configuration. 16 MiB with a 128 MiB cap is reachable up to 8 live
|
||||
segments (128/16) — i.e. exactly the single-download slow-disk case the ceiling exists
|
||||
for — and clamps predictably beyond that, where per-segment buffering has stopped
|
||||
mattering because each segment holds a small fraction of the link.
|
||||
|
||||
Reachable `bufferBytes` before the cap clamps it, by workload:
|
||||
## Default + RSS — the part that didn't land
|
||||
|
||||
| Active work | segments | cap ÷ segments | 16 MiB reachable? |
|
||||
`docs/04` line 125: **"≤ 60 MB RSS with 20 active downloads at default buffers."**
|
||||
|
||||
Per-segment accounting: 20 downloads × the default 8 segments = **160 buffers**, not 20.
|
||||
|
||||
| default | 160 buffers | after a 256 MiB cap | vs 60 MB RSS |
|
||||
|---|---|---|---|
|
||||
| 1 task, 1 seg (small file / non-resumable) | 1 | 256 MiB | yes |
|
||||
| 1 task, 8 seg (typical big download) | 8 | 32 MiB | yes, unclamped |
|
||||
| 1 task, 16 seg | 16 | 16 MiB | exactly at the cap |
|
||||
| 1 task, 32 seg | 32 | 8 MiB | **clamps to 8 MiB** |
|
||||
| 4 tasks × 8 seg | 32 | 8 MiB | clamps to 8 MiB |
|
||||
| 4 MiB | 640 MiB | 256 MiB | ~4× over |
|
||||
| 2 MiB | 320 MiB | 256 MiB | ~4× over |
|
||||
| 1 MiB | 160 MiB | 160 MiB (cap not binding) | still ~3× over |
|
||||
| any | — | **cap is the binding constraint at 20 tasks, not the default** | — |
|
||||
|
||||
Now the same table with a **64 MiB** ceiling: it is unreachable the moment total active
|
||||
segments exceed **4** (256 / 64). At the default 8 segments the user asks for 64 MiB and
|
||||
silently gets 32 MiB; at 32 segments they get 8 MiB. A maximum that no realistic
|
||||
configuration can actually use is a misleading number in the Options dialog. 16 MiB is
|
||||
reachable for the single-task and light-multitask cases — the cases where deep buffering
|
||||
is the point — and degrades predictably (to 8 MiB) exactly when per-segment buffering
|
||||
stops mattering because each segment is only getting 1/32 of the link.
|
||||
So the per-segment default is *not* what decides the 20-task case — the global cap is,
|
||||
whatever the default. **256 MiB and 60 MB RSS cannot both hold.** Hitting 60 MB across
|
||||
160 segments is ~48 MiB of buffers total (≈300 KiB each), leaving ~12 MB for 160 curl
|
||||
handles + TLS + the daemon — which is not achievable; 160 live TLS connections alone are
|
||||
~10–15 MB.
|
||||
|
||||
32 segments is already past the point of diminishing returns on segment count itself
|
||||
(`docs/04` §3: "more segments than [1 MiB each] is pure overhead and gets you
|
||||
rate-limited"); clamping their buffers to 8 MiB is the right behaviour, not a regression.
|
||||
### The real fix: cap concurrent segments, not just total buffer bytes
|
||||
|
||||
## The default: 4 MiB fails the RSS DoD
|
||||
No download manager runs 160 simultaneous connections for 20 downloads. `docs/01` §2
|
||||
already implies this ("1 curl-multi transfer thread per ~8 active segments (capped)");
|
||||
`docs/04` never states the cap. Add **`maxActiveSegments` (default 32)** — a global
|
||||
ceiling on segments actually transferring at once. 20 "active" downloads then means ~32
|
||||
live connections with the rest of each download's segments queued, not 160.
|
||||
|
||||
`docs/04` §8: "≤ 60 MB RSS with 20 active downloads at default buffers." Twenty active
|
||||
downloads, each with at least one segment:
|
||||
RSS with `maxActiveSegments = 32`, default `bufferBytes = 1 MiB`:
|
||||
|
||||
- default **4 MiB** → 20 × 4 = **80 MiB in buffers alone**, before curl handles, TLS
|
||||
buffers, thread stacks, and the task table. Busts 60 MB outright. The 256 MiB global
|
||||
cap does not save you — 80 < 256, so nothing clamps.
|
||||
- default **2 MiB** → 20 × 2 = 40 MiB, leaving ~20 MiB for everything else. Fits.
|
||||
- A single 8-segment download at 2 MiB is 16 MiB of buffer — already ample for line rate
|
||||
on NVMe (see the throughput-plateau point above).
|
||||
- buffers: 32 × 1 MiB = **32 MiB** (the 128 MiB cap isn't even engaged at the default)
|
||||
- 32 curl/HTTP2/TLS connections: ~2–3 MiB
|
||||
- daemon base (RPC loop, event batching, SQLite cache + WAL, ~8 thread stacks resident,
|
||||
task table): ~8–12 MiB
|
||||
- **total ≈ 45–50 MiB RSS — line 125 holds**, at the default, with margin.
|
||||
|
||||
So the default has to be **2 MiB** for the RSS target and the buffer default to be
|
||||
consistent, or `docs/04` §8 has to be renegotiated. 2 MiB is the cheaper fix and is not a
|
||||
throughput compromise.
|
||||
`docs/04` §8 keeps the 60 MB number but must state it now depends on
|
||||
`maxActiveSegments = 32` and default buffers. A power user who overrides every download to
|
||||
16 MiB gets clamped by `maxTotalBufferBytes` to `128 / 32 = 4 MiB` per live segment →
|
||||
128 MiB of buffers, ~140 MiB RSS — deliberately, and outside "at default buffers", so
|
||||
line 125 is unaffected.
|
||||
|
||||
### If 60 MB is the wrong target
|
||||
|
||||
It's defensible to raise it instead. 160 (or even 32) live TLS connections have an
|
||||
irreducible cost, and IDM itself uses more. If PROTO/docs prefer, change line 125 to
|
||||
**"≤ 120 MB RSS with 20 active downloads at default buffers"** and keep
|
||||
`maxActiveSegments` higher (64). CORE's recommendation is the 60 MB + cap-at-32 route
|
||||
because "lean daemon" is in the brief and 45–50 MiB is comfortably achievable — but it
|
||||
must be an explicit decision in the ADR, not the number that quietly loses.
|
||||
|
||||
## Clamp behaviour CORE will implement
|
||||
|
||||
- Per **task start** and on any change to the active-segment count (new task, task
|
||||
finishing, a steal), recompute `effective` for every live segment by the formula above.
|
||||
- Never clamp below the 64 KiB floor. If `max_total_buffer_bytes / active_segment_count`
|
||||
is itself below 64 KiB (would need >4096 concurrent segments — not reachable at the
|
||||
32-per-task ceiling and a sane concurrent-task limit), CORE admits fewer concurrent
|
||||
segments rather than shipping a sub-floor buffer.
|
||||
- The resulting `effective` value is what CORE reports upward for the readback field
|
||||
(request **B2a**). `requested` is echoed back too so the GUI can show "8 MiB (using
|
||||
2 MiB)".
|
||||
- On task start and on any change to `live_segment_count` (new task, task finishing, a
|
||||
steal), recompute `effective` for every live segment by the formula above.
|
||||
- Never below the 64 KiB floor. If `maxTotalBufferBytes / live_segment_count` < 64 KiB
|
||||
(needs >2048 live segments at a 128 MiB cap — not reachable under
|
||||
`maxActiveSegments = 32`), CORE admits fewer concurrent segments rather than shipping a
|
||||
sub-floor buffer.
|
||||
- `effective` and `requested` both reported upward so the GUI shows "16 MiB (using
|
||||
4 MiB)" — request **B2a**.
|
||||
|
||||
## Net contract delta for PROTO
|
||||
|
||||
- `DownloadSpec.bufferBytes`, `download.update` patch: `minimum: 65536`, `maximum:
|
||||
16777216`, `default: 2097152`.
|
||||
- `docs/04` §4 line ~70: "Default 2 MiB, range 64 KiB – 16 MiB. Silently reduced to fit
|
||||
`max_total_buffer_bytes` (256 MiB) across all active segments; the effective value is
|
||||
reported back."
|
||||
- `docs/04` §8: keep "≤ 60 MB RSS / 20 downloads" — it now holds at the 2 MiB default.
|
||||
- ADR: record the throughput-plateau + global-cap-arithmetic reasoning; note 8 MiB was
|
||||
considered for the ceiling and 16 MiB chosen for stall absorption.
|
||||
1. **`bufferBytes` bounds in FOUR schema files** (not three): `DownloadSpec`,
|
||||
`TaskDetail`, `download.update` patch, **and `Settings.schema.json`
|
||||
`connection.bufferBytes`** — all currently `4096`–`8388608`. Change every one to
|
||||
`minimum: 65536`, `maximum: 16777216`, `default: 1048576`.
|
||||
2. **`maxTotalBufferBytes` is absent from the contract entirely.** The clamp CORE
|
||||
implements has no wire representation, so the Options dialog can neither show nor set
|
||||
it. Add `Settings.schema.json connection.maxTotalBufferBytes` (default `134217728`) —
|
||||
fold into the **B2a** follow-up alongside `effectiveBufferBytes`.
|
||||
3. **`maxActiveSegments` is new.** Add `Settings.schema.json connection.maxActiveSegments`
|
||||
(default `32`). CORE enforces it; DAEMON's scheduler needs to know it to decide what to
|
||||
start.
|
||||
4. **`docs/04` §4 line ~70:** "Default 1 MiB, range 64 KiB – 16 MiB. Reduced to fit
|
||||
`maxTotalBufferBytes` (128 MiB) across all live segments; effective value reported
|
||||
back."
|
||||
5. **`docs/04` §8 line 125:** keep "≤ 60 MB RSS / 20 downloads" but add "given
|
||||
`maxActiveSegments = 32` and default buffers" — or raise to 120 MB (see above). ADR
|
||||
records which and why.
|
||||
6. **ADR:** throughput-plateau reasoning for the ceiling; the per-segment RSS arithmetic;
|
||||
the `maxActiveSegments` addition as the mechanism that makes line 125 hold; 8 MiB
|
||||
considered and rejected for the ceiling in favour of 16 MiB stall absorption.
|
||||
|
||||
@@ -82,6 +82,15 @@ extension; it has to be in the schema or it's dropped by every conformance repla
|
||||
and placement are PROTO's call (`effectiveBufferBytes` on `TaskSummary` / `TaskDetail`,
|
||||
or a dedicated readback). Land it whenever — it just can't be improvised later.
|
||||
|
||||
Fold in two more, all discovered via B4 (`buffer-sizing.md`):
|
||||
- **`maxTotalBufferBytes` is absent from the contract entirely.** CORE implements the
|
||||
clamp against it; the Options dialog can neither show nor set it. Add
|
||||
`Settings.schema.json connection.maxTotalBufferBytes` (recommended default
|
||||
`134217728` = 128 MiB).
|
||||
- **`maxActiveSegments` is new** — the global concurrent-segment cap that actually bounds
|
||||
"20 active downloads" for the RSS budget. Add `Settings.schema.json
|
||||
connection.maxActiveSegments` (recommended default `32`). DAEMON's scheduler reads it.
|
||||
|
||||
### B3. Freeze `Segment.schema.json` field names
|
||||
|
||||
`Segment` is already on PROTO's type list (`contracts/README.md`) and `TaskDetail`
|
||||
@@ -128,12 +137,15 @@ total length 0 round-trips. Flag back if the chosen fix needs anything else from
|
||||
|
||||
### B4. `bufferBytes` range is wrong in the frozen schema — *see `buffer-sizing.md`*
|
||||
|
||||
`contracts/` froze `bufferBytes` at 4 KiB – 8 MiB; `docs/04` §4 says 64 KiB – 64 MiB
|
||||
default 4 MiB; the RSS DoD (`docs/04` §8) can't hold either default. CORE's analysis and
|
||||
the recommended range (**64 KiB – 16 MiB, default 2 MiB**, `max_total_buffer_bytes`
|
||||
unchanged at 256 MiB) with the global-cap arithmetic is in
|
||||
[`core/docs/buffer-sizing.md`](buffer-sizing.md). PROTO to land schema + `docs/04` §4 +
|
||||
ADR together.
|
||||
`contracts/` froze `bufferBytes` at 4 KiB – 8 MiB **in four schema files** (`DownloadSpec`,
|
||||
`TaskDetail`, `download.update` patch, and `Settings.schema.json connection.bufferBytes`);
|
||||
`docs/04` §4 says 64 KiB – 64 MiB default 4 MiB; the RSS DoD (`docs/04` §8 line 125) holds
|
||||
with neither. CORE's recommendation — **64 KiB – 16 MiB, default 1 MiB**, plus a new
|
||||
**`maxActiveSegments` (32)** global concurrent-segment cap and **`maxTotalBufferBytes`
|
||||
lowered to 128 MiB** and *added to the contract*, with line 125 kept at 60 MB "given
|
||||
`maxActiveSegments = 32`" (or explicitly raised to 120 MB) — and the per-segment RSS
|
||||
arithmetic are in [`core/docs/buffer-sizing.md`](buffer-sizing.md). PROTO to land the four
|
||||
schema edits + `docs/04` §4 + §8 + ADR together.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user