# 04 — Download engine design (`libveloxcore`) Owner: lane **CORE**. This is the part that has to be genuinely fast. ## 1. Task lifecycle ``` NEW → PROBING → QUEUED → CONNECTING → DOWNLOADING ⇄ PAUSED │ │ │ └→ RETRY_WAIT → CONNECTING ▼ ASSEMBLING → VERIFYING → COMPLETE │ └→ FAILED | CANCELLED ``` `ASSEMBLING` is normally a no-op rename (see §4). It exists as a real state only for the HLS/DASH path, which must mux. ## 2. Probe (`download.probe`) Feeds IDM's "Download File Info" dialog, so it must be fast and never partially download. 1. `HEAD` with the browser's headers/cookies/referrer/UA verbatim. 2. If `HEAD` is refused (405/403 — common), fall back to `GET` with `Range: bytes=0-0` and abort after headers. 3. Extract: final URL after redirects, `Content-Length`, `Content-Type`, `Content-Disposition` filename (RFC 5987/6266, UTF-8 and legacy), `Accept-Ranges`, `ETag`, `Last-Modified`. 4. **Resumability is proven, not assumed:** treat as resumable only if the range request returned `206` *and* `Content-Range` matches. Servers lie about `Accept-Ranges`. 5. Filename resolution order: explicit user name → `Content-Disposition` → last path segment (percent-decoded) → `Content-Type` extension → `download.bin`. Then sanitize: strip `/` `\0`, control chars, trailing dots/spaces, cap at 200 bytes *of UTF-8* (never split a codepoint), reject `.` and `..`. ## 3. Segmentation — the part IDM is actually famous for Static N-way splitting is not what makes IDM feel fast; **dynamic segment stealing** is. ``` Initial: [====seg0====][====seg1====][====seg2====][====seg3====] seg1 is on a fast mirror and finishes early: [==seg0==....][ done ][==seg2==....][==seg3==....] ▲ largest remaining tail Steal: seg1 restarts on the second half of seg2's remaining range: [==seg0==....][ done ][=seg2=][==seg1'==][==seg3==....] ``` Rules: - Default 8 segments, user-configurable 1–32, clamped per-host by settings (some hosts ban >4 connections; keep a per-host override table). - Never split below `min_segment_bytes` (default 1 MiB) — more segments than that is pure overhead and gets you rate-limited. - On finish, a worker steals the **second half of the largest remaining range**, atomically under the task lock, and only if that half is ≥ `min_segment_bytes`. - A segment that fails 3× with a connection error is not retried on the same host if a mirror exists; the range is returned to the pool and re-split. - Non-resumable servers → exactly 1 segment, and the UI must say so (IDM's "Resume capability: No"). ## 4. Disk I/O — the buffer setting you asked for One file, opened once, `O_WRONLY | O_NOFOLLOW`. Each segment `pwrite()`s at its own absolute offset, so **there is no reassembly pass and no second write of the whole file.** `O_NOFOLLOW` on the part-file open: DAEMON canonicalises the save path and checks it against the allowed roots before `start()`, but the final component could be swapped for a symlink in the window between that check and our open. A symlinked leaf is rejected here (`ELOOP` → `Error::path_rejected`), not followed — it closes the TOCTOU residual that `daemon/docs/safepath-adversarial.md` accepts on those grounds. - `posix_fallocate()` the full size up front → contiguous extents, no ENOSPC surprise at 99 %, no fragmentation. - Per-segment ring buffer, size = **`buffer_bytes`** (the user-visible "Buffer size" setting, `connection.bufferBytes` on the wire). Default **1 MiB**, range **64 KiB – 16 MiB**. Curl's write callback appends; the buffer is flushed with a single `pwrite` when full or when the segment ends. *This is the single biggest throughput knob and it is exposed in the UI: Options → Downloads → "Write buffer per connection".* - Global cap `max_total_buffer_bytes` (`connection.maxTotalBufferBytes`, default **128 MiB**) so a burst of large downloads with a large per-segment buffer can't OOM the box. Combined with `max_active_segments` (`connection.maxActiveSegments`, default **32**) — the ceiling on segments actually transferring at once, across every task, not per download — every live segment's buffer is reduced to fit `max_total_buffer_bytes / live_segment_count` (capped by `max_active_segments`), never below the 64 KiB floor. The requested and effective values are both reported back to the UI (`TaskDetail.bufferBytes` / `.effectiveBufferBytes`) so it can show, for example, "16 MiB (using 4 MiB)". See `docs/adr/0012-buffer-and-segment-budget.md` for the reasoning behind these numbers, including why 4 MiB / 64 MiB / 256 MiB (this section's earlier draft) does not hold ≤ 60 MB RSS once buffers are counted per segment rather than per download. - `posix_fadvise(POSIX_FADV_DONTNEED)` on written ranges — do not let a 40 GB ISO evict the user's entire page cache. - `fdatasync()` on a timer (default 5 s) and on pause, **not** per write. - `io_uring` is a **post-1.0 optimization**, gated behind a benchmark in `tools/bench/` that must show ≥10 % improvement on NVMe. Do not start there. ## 5. Resume metadata — `.veloxpart.meta` Written next to the part file so a download survives a daemon crash, a reboot, *and* a database loss. Little-endian, versioned, `fdatasync`'d on every segment-boundary update: ``` magic "VDMP" | u16 version | u16 flags u64 total_size | u64 downloaded url_set (original + effective + mirrors, length-prefixed UTF-8) etag | last_modified | content_type u32 segment_count per segment: u64 start, u64 end, u64 completed sha256_partial_state (optional, for streaming hash) crc32 of the whole record ``` On resume, revalidate with `If-Range: `. If the server answers `200` instead of `206`, the file changed underneath us: surface it as "File on server has changed — restart download?" rather than silently corrupting the file. This is the single most common way download managers produce broken files. Do not get it wrong. ## 6. Rate limiting Hierarchical token buckets: global → per-queue → per-task. Refill on a 100 ms tick; throttle by delaying curl reads (`CURLOPT_MAX_RECV_SPEED_LARGE` as a coarse floor, plus our own pause/unpause via `curl_easy_pause` for precision). "Speed Limiter" in the UI toggles between Full speed / a saved limit, exactly as IDM does. ## 7. Failure policy | Case | Behaviour | |---|---| | 5xx / timeout / reset | Exponential backoff 1 s→2→4→8→…→cap 60 s, jitter ±20 %, `max_retries` default 10 | | 401/407 | Emit `event.auth.required`, pause task, GUI shows credential dialog, store in secret service (never in SQLite) | | 403 after redirect | Retry once with the original referrer; many CDNs require it | | 416 | Metadata is stale → re-probe, re-split | | Disk full | Pause all, one notification, do not spin | | Server drops range support mid-download | Demote to 1 segment, keep what's on disk, continue | ## 8. Performance targets (M7 gate, `tools/bench/`) - Saturate a 1 Gbit link with ≤ 8 % of one core. - ≤ 60 MB RSS with 20 active downloads at default buffers, **given `max_active_segments = 32`** — without that cap, 20 downloads × 8 segments each is 160 live buffers even at the 1 MiB default, and the number does not hold. See `docs/adr/0012-buffer-and-segment-budget.md`. - 10 000-row task list: RPC `download.list` under 50 ms, GUI scroll at 60 fps. - No allocation in the curl write callback hot path (ring buffer is preallocated).