samiandClaude Sonnet 5
60363a7142
Minor bump on 1.0.0, per core/docs/buffer-sizing.md. B4 — bufferBytes bounds corrected in all four locations (DownloadSpec, TaskDetail, download.update's patch, Settings.connection.bufferBytes): was 4 KiB-8 MiB with no stated default, now 64 KiB-16 MiB with a 1 MiB default. 64 KiB because 4 KiB is smaller than one libcurl HTTP/2 write-callback delivery; 16 MiB because throughput from write size is flat past ~1-4 MiB and past 16 MiB there is stall-cover left to buy but no memory left to spend it on; 1 MiB default because it is the only candidate for which docs/04's 60 MB RSS target actually holds once buffers are counted per segment, not per download. Two new settings keys: connection.maxTotalBufferBytes (128 MiB default) and connection.maxActiveSegments (32 default). Without them CORE's clamp — reduce every live segment's buffer to fit the global cap — has no wire configuration surface, and "20 active downloads" has no meaning distinct from 160 live TLS connections. B2a — TaskDetail.effectiveBufferBytes: what a segment is actually using right now, after the clamp. Placed on TaskDetail next to bufferBytes, following the requested/effective pattern ADR 0010 already established for segments. The download.get fixture now demonstrates a real clamp (16 MiB requested, 4 MiB effective) rather than a case where the cap happens not to bind. docs/04-engine-design.md §4 and §8 updated in the same change per CORE's request and CLAUDE.md rule 5: the RSS target is now stated as conditional on maxActiveSegments = 32, and the old 4 MiB/64 MiB/256 MiB numbers are corrected to match the schema. ADR 0012 records the reasoning and explicitly keeps the 60 MB target over CORE's offered 120 MB alternative, with the arithmetic that makes 60 MB achievable with margin. Numbered 0012 rather than 0011: DAEMON is independently drafting ADR 0011 (admission control / segment budget split) in a peer session at time of writing, so 0011 was reserved to avoid a collision. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_012fgjnqFCS5h5L7gZTZo3rV
138 lines
7.2 KiB
Markdown
138 lines
7.2 KiB
Markdown
# 04 — Download engine design (`libveloxcore`)
|
||
|
||
Owner: lane **CORE**. This is the part that has to be genuinely fast.
|
||
|
||
## 1. Task lifecycle
|
||
|
||
```
|
||
NEW → PROBING → QUEUED → CONNECTING → DOWNLOADING ⇄ PAUSED
|
||
│ │
|
||
│ └→ RETRY_WAIT → CONNECTING
|
||
▼
|
||
ASSEMBLING → VERIFYING → COMPLETE
|
||
│
|
||
└→ FAILED | CANCELLED
|
||
```
|
||
|
||
`ASSEMBLING` is normally a no-op rename (see §4). It exists as a real state only for the
|
||
HLS/DASH path, which must mux.
|
||
|
||
## 2. Probe (`download.probe`)
|
||
|
||
Feeds IDM's "Download File Info" dialog, so it must be fast and never partially download.
|
||
|
||
1. `HEAD` with the browser's headers/cookies/referrer/UA verbatim.
|
||
2. If `HEAD` is refused (405/403 — common), fall back to `GET` with `Range: bytes=0-0` and
|
||
abort after headers.
|
||
3. Extract: final URL after redirects, `Content-Length`, `Content-Type`,
|
||
`Content-Disposition` filename (RFC 5987/6266, UTF-8 and legacy), `Accept-Ranges`,
|
||
`ETag`, `Last-Modified`.
|
||
4. **Resumability is proven, not assumed:** treat as resumable only if the range request
|
||
returned `206` *and* `Content-Range` matches. Servers lie about `Accept-Ranges`.
|
||
5. Filename resolution order: explicit user name → `Content-Disposition` → last path
|
||
segment (percent-decoded) → `Content-Type` extension → `download.bin`.
|
||
Then sanitize: strip `/` `\0`, control chars, trailing dots/spaces, cap at 200 bytes
|
||
*of UTF-8* (never split a codepoint), reject `.` and `..`.
|
||
|
||
## 3. Segmentation — the part IDM is actually famous for
|
||
|
||
Static N-way splitting is not what makes IDM feel fast; **dynamic segment stealing** is.
|
||
|
||
```
|
||
Initial: [====seg0====][====seg1====][====seg2====][====seg3====]
|
||
seg1 is on a fast mirror and finishes early:
|
||
[==seg0==....][ done ][==seg2==....][==seg3==....]
|
||
▲ largest remaining tail
|
||
Steal: seg1 restarts on the second half of seg2's remaining range:
|
||
[==seg0==....][ done ][=seg2=][==seg1'==][==seg3==....]
|
||
```
|
||
|
||
Rules:
|
||
- Default 8 segments, user-configurable 1–32, clamped per-host by settings (some hosts
|
||
ban >4 connections; keep a per-host override table).
|
||
- Never split below `min_segment_bytes` (default 1 MiB) — more segments than that is pure
|
||
overhead and gets you rate-limited.
|
||
- On finish, a worker steals the **second half of the largest remaining range**, atomically
|
||
under the task lock, and only if that half is ≥ `min_segment_bytes`.
|
||
- A segment that fails 3× with a connection error is not retried on the same host if a
|
||
mirror exists; the range is returned to the pool and re-split.
|
||
- Non-resumable servers → exactly 1 segment, and the UI must say so
|
||
(IDM's "Resume capability: No").
|
||
|
||
## 4. Disk I/O — the buffer setting you asked for
|
||
|
||
One file, opened once, `O_WRONLY`. Each segment `pwrite()`s at its own absolute offset, so
|
||
**there is no reassembly pass and no second write of the whole file.**
|
||
|
||
- `posix_fallocate()` the full size up front → contiguous extents, no ENOSPC surprise at
|
||
99 %, no fragmentation.
|
||
- Per-segment ring buffer, size = **`buffer_bytes`** (the user-visible "Buffer size"
|
||
setting, `connection.bufferBytes` on the wire). Default **1 MiB**, range **64 KiB –
|
||
16 MiB**. Curl's write callback appends; the buffer is flushed with a single `pwrite`
|
||
when full or when the segment ends.
|
||
*This is the single biggest throughput knob and it is exposed in the UI: Options →
|
||
Downloads → "Write buffer per connection".*
|
||
- Global cap `max_total_buffer_bytes` (`connection.maxTotalBufferBytes`, default
|
||
**128 MiB**) so a burst of large downloads with a large per-segment buffer can't OOM the
|
||
box. Combined with `max_active_segments` (`connection.maxActiveSegments`, default
|
||
**32**) — the ceiling on segments actually transferring at once, across every task, not
|
||
per download — every live segment's buffer is reduced to fit
|
||
`max_total_buffer_bytes / live_segment_count` (capped by `max_active_segments`), never
|
||
below the 64 KiB floor. The requested and effective values are both reported back to
|
||
the UI (`TaskDetail.bufferBytes` / `.effectiveBufferBytes`) so it can show, for example,
|
||
"16 MiB (using 4 MiB)". See `docs/adr/0012-buffer-and-segment-budget.md` for the
|
||
reasoning behind these numbers, including why 4 MiB / 64 MiB / 256 MiB (this section's
|
||
earlier draft) does not hold ≤ 60 MB RSS once buffers are counted per segment rather
|
||
than per download.
|
||
- `posix_fadvise(POSIX_FADV_DONTNEED)` on written ranges — do not let a 40 GB ISO evict
|
||
the user's entire page cache.
|
||
- `fdatasync()` on a timer (default 5 s) and on pause, **not** per write.
|
||
- `io_uring` is a **post-1.0 optimization**, gated behind a benchmark in `tools/bench/`
|
||
that must show ≥10 % improvement on NVMe. Do not start there.
|
||
|
||
## 5. Resume metadata — `<name>.veloxpart.meta`
|
||
|
||
Written next to the part file so a download survives a daemon crash, a reboot, *and* a
|
||
database loss. Little-endian, versioned, `fdatasync`'d on every segment-boundary update:
|
||
|
||
```
|
||
magic "VDMP" | u16 version | u16 flags
|
||
u64 total_size | u64 downloaded
|
||
url_set (original + effective + mirrors, length-prefixed UTF-8)
|
||
etag | last_modified | content_type
|
||
u32 segment_count
|
||
per segment: u64 start, u64 end, u64 completed
|
||
sha256_partial_state (optional, for streaming hash)
|
||
crc32 of the whole record
|
||
```
|
||
|
||
On resume, revalidate with `If-Range: <etag or last-modified>`. If the server answers
|
||
`200` instead of `206`, the file changed underneath us: surface it as
|
||
"File on server has changed — restart download?" rather than silently corrupting the file.
|
||
This is the single most common way download managers produce broken files. Do not get it wrong.
|
||
|
||
## 6. Rate limiting
|
||
|
||
Hierarchical token buckets: global → per-queue → per-task. Refill on a 100 ms tick;
|
||
throttle by delaying curl reads (`CURLOPT_MAX_RECV_SPEED_LARGE` as a coarse floor, plus
|
||
our own pause/unpause via `curl_easy_pause` for precision). "Speed Limiter" in the UI
|
||
toggles between Full speed / a saved limit, exactly as IDM does.
|
||
|
||
## 7. Failure policy
|
||
|
||
| Case | Behaviour |
|
||
|---|---|
|
||
| 5xx / timeout / reset | Exponential backoff 1 s→2→4→8→…→cap 60 s, jitter ±20 %, `max_retries` default 10 |
|
||
| 401/407 | Emit `event.auth.required`, pause task, GUI shows credential dialog, store in secret service (never in SQLite) |
|
||
| 403 after redirect | Retry once with the original referrer; many CDNs require it |
|
||
| 416 | Metadata is stale → re-probe, re-split |
|
||
| Disk full | Pause all, one notification, do not spin |
|
||
| Server drops range support mid-download | Demote to 1 segment, keep what's on disk, continue |
|
||
|
||
## 8. Performance targets (M7 gate, `tools/bench/`)
|
||
|
||
- Saturate a 1 Gbit link with ≤ 8 % of one core.
|
||
- ≤ 60 MB RSS with 20 active downloads at default buffers, **given `max_active_segments = 32`** — without that cap, 20 downloads × 8 segments each is 160 live buffers even at the 1 MiB default, and the number does not hold. See `docs/adr/0012-buffer-and-segment-budget.md`.
|
||
- 10 000-row task list: RPC `download.list` under 50 ms, GUI scroll at 60 fps.
|
||
- No allocation in the curl write callback hot path (ring buffer is preallocated).
|