merge: util layer and buffer-sizing analysis

core/docs/buffer-sizing.md

@@ -0,0 +1,141 @@
+# CORE → PROTO — `bufferBytes` range + the RSS-budget reconciliation
+
+`contracts/` froze `bufferBytes` at **4 KiB – 8 MiB** in four places; `docs/04` §4 line 70
+says **64 KiB – 64 MiB, default 4 MiB**. The floor and ceiling are settled below (64 KiB /
+16 MiB — my earlier reasoning holds). The default and `max_total_buffer_bytes` are
+**not** — my first pass counted one buffer per download; it is one per *segment*
+("Per-segment ring buffer", line 68; "Write buffer per connection", line 70), which
+changes the whole RSS picture. This version fixes that.
+
+PROTO to land schema + `docs/04` §4 + §8 + an ADR together.
+
+## Recommendation
+
+| Knob | Value | |
+|---|---|---|
+| `bufferBytes` minimum | **65536** (64 KiB) | frozen 4 KiB is below one libcurl callback — see §Floor |
+| `bufferBytes` maximum | **16777216** (16 MiB) | see §Ceiling; the `256/64` unreachability argument is *stronger* per-segment |
+| `bufferBytes` default | **1048576** (1 MiB) | see §Default+RSS; 2 MiB and 4 MiB both bust line 125 |
+| `maxActiveSegments` (**new**) | **32** | global cap on concurrently-transferring segments — the actual bound on "20 active downloads" |
+| `maxTotalBufferBytes` | **134217728** (128 MiB), down from 256 | backstop for the buffer knob; RSS-safe given the 1 MiB default + the segment cap |
+
+## Floor — 64 KiB, not 4 KiB
+
+The buffer's first job is to coalesce libcurl write-callback deliveries into one `pwrite`.
+Over HTTP/2 a single write callback is routinely 16–64 KiB, up to ~256 KiB. A 4 KiB ring
+buffer is **smaller than one callback**: the "one `pwrite` per fill" design degrades to a
+syscall per curl chunk — the exact thing the buffer exists to prevent. 64 KiB (≈4 typical
+chunks) is the smallest floor that buys anything. 4 KiB is a page size that wandered into
+a throughput knob.
+
+## Ceiling — 16 MiB, not 64 MiB
+
+Two things the buffer buys past coalescing:
+
+1. **Large sequential writes.** On NVMe, throughput vs. write size is flat by ~1–4 MiB.
+   4→8 MiB gains a little on syscall overhead at multi-Gbit; past 8 MiB there is no
+   throughput left to get, only `fdatasync` latency and page-cache pressure (a 40 GB ISO
+   must not evict the user's working set — `docs/04` §4).
+2. **Absorbing a disk stall without stalling the socket** — the only reason to exceed
+   8 MiB. Fast link + bursty storage (HDD, SMR, USB, network mount): 1 Gbit ≈ 125 MB/s,
+   so 16 MiB/segment ≈ 130 ms of write-stall cover, ~0.5 s across 4 segments — enough to
+   ride out a seek storm. Beyond 16 MiB the marginal cover isn't worth the footprint.
+
+**8 MiB captures all throughput; 8–16 MiB is stall-absorption headroom for the
+fast-pipe/slow-disk case; >16 MiB is waste.**
+
+### `maxTotalBufferBytes / segments` unreachability — worse per-segment
+
+`effective = clamp(requested, 64 KiB, floor(maxTotalBufferBytes / live_segment_count))`.
+
+With a **64 MiB** ceiling and the old 256 MiB cap, 64 MiB is unreachable once total live
+segments exceed 4 (256/64). Under correct per-segment accounting "4 total segments" is
+*half of one default download* (8 segments) — so the frozen-doc ceiling is unreachable in
+essentially every real configuration. 16 MiB with a 128 MiB cap is reachable up to 8 live
+segments (128/16) — i.e. exactly the single-download slow-disk case the ceiling exists
+for — and clamps predictably beyond that, where per-segment buffering has stopped
+mattering because each segment holds a small fraction of the link.
+
+## Default + RSS — the part that didn't land
+
+`docs/04` line 125: **"≤ 60 MB RSS with 20 active downloads at default buffers."**
+
+Per-segment accounting: 20 downloads × the default 8 segments = **160 buffers**, not 20.
+
+| default | 160 buffers | after a 256 MiB cap | vs 60 MB RSS |
+|---|---|---|---|
+| 4 MiB | 640 MiB | 256 MiB | ~4× over |
+| 2 MiB | 320 MiB | 256 MiB | ~4× over |
+| 1 MiB | 160 MiB | 160 MiB (cap not binding) | still ~3× over |
+| any | — | **cap is the binding constraint at 20 tasks, not the default** | — |
+
+So the per-segment default is *not* what decides the 20-task case — the global cap is,
+whatever the default. **256 MiB and 60 MB RSS cannot both hold.** Hitting 60 MB across
+160 segments is ~48 MiB of buffers total (≈300 KiB each), leaving ~12 MB for 160 curl
+handles + TLS + the daemon — which is not achievable; 160 live TLS connections alone are
+~10–15 MB.
+
+### The real fix: cap concurrent segments, not just total buffer bytes
+
+No download manager runs 160 simultaneous connections for 20 downloads. `docs/01` §2
+already implies this ("1 curl-multi transfer thread per ~8 active segments (capped)");
+`docs/04` never states the cap. Add **`maxActiveSegments` (default 32)** — a global
+ceiling on segments actually transferring at once. 20 "active" downloads then means ~32
+live connections with the rest of each download's segments queued, not 160.
+
+RSS with `maxActiveSegments = 32`, default `bufferBytes = 1 MiB`:
+
+- buffers: 32 × 1 MiB = **32 MiB** (the 128 MiB cap isn't even engaged at the default)
+- 32 curl/HTTP2/TLS connections: ~2–3 MiB
+- daemon base (RPC loop, event batching, SQLite cache + WAL, ~8 thread stacks resident,
+  task table): ~8–12 MiB
+- **total ≈ 45–50 MiB RSS — line 125 holds**, at the default, with margin.
+
+`docs/04` §8 keeps the 60 MB number but must state it now depends on
+`maxActiveSegments = 32` and default buffers. A power user who overrides every download to
+16 MiB gets clamped by `maxTotalBufferBytes` to `128 / 32 = 4 MiB` per live segment →
+128 MiB of buffers, ~140 MiB RSS — deliberately, and outside "at default buffers", so
+line 125 is unaffected.
+
+### If 60 MB is the wrong target
+
+It's defensible to raise it instead. 160 (or even 32) live TLS connections have an
+irreducible cost, and IDM itself uses more. If PROTO/docs prefer, change line 125 to
+**"≤ 120 MB RSS with 20 active downloads at default buffers"** and keep
+`maxActiveSegments` higher (64). CORE's recommendation is the 60 MB + cap-at-32 route
+because "lean daemon" is in the brief and 45–50 MiB is comfortably achievable — but it
+must be an explicit decision in the ADR, not the number that quietly loses.
+
+## Clamp behaviour CORE will implement
+
+- On task start and on any change to `live_segment_count` (new task, task finishing, a
+  steal), recompute `effective` for every live segment by the formula above.
+- Never below the 64 KiB floor. If `maxTotalBufferBytes / live_segment_count` < 64 KiB
+  (needs >2048 live segments at a 128 MiB cap — not reachable under
+  `maxActiveSegments = 32`), CORE admits fewer concurrent segments rather than shipping a
+  sub-floor buffer.
+- `effective` and `requested` both reported upward so the GUI shows "16 MiB (using
+  4 MiB)" — request **B2a**.
+
+## Net contract delta for PROTO
+
+1. **`bufferBytes` bounds in FOUR schema files** (not three): `DownloadSpec`,
+   `TaskDetail`, `download.update` patch, **and `Settings.schema.json`
+   `connection.bufferBytes`** — all currently `4096`–`8388608`. Change every one to
+   `minimum: 65536`, `maximum: 16777216`, `default: 1048576`.
+2. **`maxTotalBufferBytes` is absent from the contract entirely.** The clamp CORE
+   implements has no wire representation, so the Options dialog can neither show nor set
+   it. Add `Settings.schema.json connection.maxTotalBufferBytes` (default `134217728`) —
+   fold into the **B2a** follow-up alongside `effectiveBufferBytes`.
+3. **`maxActiveSegments` is new.** Add `Settings.schema.json connection.maxActiveSegments`
+   (default `32`). CORE enforces it; DAEMON's scheduler needs to know it to decide what to
+   start.
+4. **`docs/04` §4 line ~70:** "Default 1 MiB, range 64 KiB – 16 MiB. Reduced to fit
+   `maxTotalBufferBytes` (128 MiB) across all live segments; effective value reported
+   back."
+5. **`docs/04` §8 line 125:** keep "≤ 60 MB RSS / 20 downloads" but add "given
+   `maxActiveSegments = 32` and default buffers" — or raise to 120 MB (see above). ADR
+   records which and why.
+6. **ADR:** throughput-plateau reasoning for the ceiling; the per-segment RSS arithmetic;
+   the `maxActiveSegments` addition as the mechanism that makes line 125 hold; 8 MiB
+   considered and rejected for the ceiling in favour of 16 MiB stall absorption.

core/docs/proto-requests-m1.md

@@ -82,6 +82,15 @@ extension; it has to be in the schema or it's dropped by every conformance repla
 and placement are PROTO's call (`effectiveBufferBytes` on `TaskSummary` / `TaskDetail`,
 or a dedicated readback). Land it whenever — it just can't be improvised later.
 
+Fold in two more, all discovered via B4 (`buffer-sizing.md`):
+- **`maxTotalBufferBytes` is absent from the contract entirely.** CORE implements the
+  clamp against it; the Options dialog can neither show nor set it. Add
+  `Settings.schema.json connection.maxTotalBufferBytes` (recommended default
+  `134217728` = 128 MiB).
+- **`maxActiveSegments` is new** — the global concurrent-segment cap that actually bounds
+  "20 active downloads" for the RSS budget. Add `Settings.schema.json
+  connection.maxActiveSegments` (recommended default `32`). DAEMON's scheduler reads it.
+
 ### B3. Freeze `Segment.schema.json` field names
 
 `Segment` is already on PROTO's type list (`contracts/README.md`) and `TaskDetail`
@@ -93,16 +102,50 @@ CORE will emit per segment:
 | CORE field | Type | Note |
 |---|---|---|
 | `index` | int ≥ 0 | **`event.task.progress` currently says `i`.** Pick one name for both. |
-| `start` | int ≥ 0 | absolute byte offset, inclusive |
-| `end` | int ≥ 0 | absolute byte offset, **exclusive** — range is `[start, end)` |
+| `startByte` | int ≥ 0 | absolute byte offset, inclusive |
+| `endByte` | int | absolute byte offset, **inclusive** — range is `[startByte, endByte]` (see below) |
 | `completed` | int ≥ 0 | bytes written in this range so far |
 | `speedBps` | int ≥ 0 | current per-segment rate |
 | `state` | enum | `connecting` \| `downloading` \| `stalled` \| `complete` \| `failed` |
 
-Asks: (a) reconcile `i` vs `index` — one spelling in both the `Segment` type and the
-`event.task.progress` payload; (b) confirm the half-open `[start, end)` convention in the
-schema `description` so DAEMON and GUI don't off-by-one the last byte; (c) confirm the
-segment `state` enum values.
+**Resolved by PROTO at freeze:** `endByte` is **inclusive** (matches HTTP `Range`
+semantics — `Range: bytes=start-end` is inclusive — and removes an off-by-one at the curl
+boundary). CORE designs stage 6 (segmenter/stealer) against inclusive. The earlier
+`[start, end)` ask is withdrawn.
+
+Still open: (a) reconcile `i` vs `index`; (b) confirm the segment `state` enum values;
+(c) **empty-segment representation** — see B3a.
+
+### B3a. Zero-length segment must be representable — *PROTO is fixing; CORE's requirement*
+
+With `endByte` inclusive and `minimum: 0`, a zero-length segment (`endByte = startByte -
+1`) at offset 0 is `endByte = -1`, which the schema forbids. The one case CORE actually
+needs: a **whole-file zero-length download** (`Content-Length: 0`) — one segment, length
+0. It is a valid HTTP response and the daemon/GUI must be able to hold it.
+
+CORE will **not** produce empty segments mid-download: the `min_segment_bytes` floor
+(1 MiB, `docs/04` §3) means the segmenter never splits below 1 MiB and the stealer only
+takes a half-range if it is ≥ that floor. So B3a is purely about the degenerate
+whole-file case.
+
+Preference: encode segments as `startByte` + `length` (+ `completed`) rather than an
+inclusive `endByte` with a `startByte - 1` sentinel — `length: 0` is then the natural
+representation and there is no negative value to allow. If `endByte` inclusive stays,
+then a 0-byte task needs an explicit encoding (an `empty`/`length` field, or permitting
+`endByte = startByte - 1` with `minimum: -1`) — any of those work for CORE as long as
+total length 0 round-trips. Flag back if the chosen fix needs anything else from CORE.
+
+### B4. `bufferBytes` range is wrong in the frozen schema — *see `buffer-sizing.md`*
+
+`contracts/` froze `bufferBytes` at 4 KiB – 8 MiB **in four schema files** (`DownloadSpec`,
+`TaskDetail`, `download.update` patch, and `Settings.schema.json connection.bufferBytes`);
+`docs/04` §4 says 64 KiB – 64 MiB default 4 MiB; the RSS DoD (`docs/04` §8 line 125) holds
+with neither. CORE's recommendation — **64 KiB – 16 MiB, default 1 MiB**, plus a new
+**`maxActiveSegments` (32)** global concurrent-segment cap and **`maxTotalBufferBytes`
+lowered to 128 MiB** and *added to the contract*, with line 125 kept at 60 MB "given
+`maxActiveSegments = 32`" (or explicitly raised to 120 MB) — and the per-segment RSS
+arithmetic are in [`core/docs/buffer-sizing.md`](buffer-sizing.md). PROTO to land the four
+schema edits + `docs/04` §4 + §8 + ADR together.
 
 ---