# 04 — Download engine design (`libveloxcore`) Owner: lane **CORE**. This is the part that has to be genuinely fast. ## 1. Task lifecycle ``` NEW → PROBING → QUEUED → CONNECTING → DOWNLOADING ⇄ PAUSED │ │ │ └→ RETRY_WAIT → CONNECTING ▼ ASSEMBLING → VERIFYING → COMPLETE │ └→ FAILED | CANCELLED ``` `ASSEMBLING` is normally a no-op rename (see §4). It exists as a real state only for the HLS/DASH path, which must mux. ## 2. Probe (`download.probe`) Feeds IDM's "Download File Info" dialog, so it must be fast and never partially download. 1. `HEAD` with the browser's headers/cookies/referrer/UA verbatim. 2. If `HEAD` is refused (405/403 — common), fall back to `GET` with `Range: bytes=0-0` and abort after headers. 3. Extract: final URL after redirects, `Content-Length`, `Content-Type`, `Content-Disposition` filename (RFC 5987/6266, UTF-8 and legacy), `Accept-Ranges`, `ETag`, `Last-Modified`. 4. **Resumability is proven, not assumed:** treat as resumable only if the range request returned `206` *and* `Content-Range` matches. Servers lie about `Accept-Ranges`. 5. Filename resolution order: explicit user name → `Content-Disposition` → last path segment (percent-decoded) → `Content-Type` extension → `download.bin`. Then sanitize: strip `/` `\0`, control chars, trailing dots/spaces, cap at 200 bytes *of UTF-8* (never split a codepoint), reject `.` and `..`. ## 3. Segmentation — the part IDM is actually famous for Static N-way splitting is not what makes IDM feel fast; **dynamic segment stealing** is. ``` Initial: [====seg0====][====seg1====][====seg2====][====seg3====] seg1 is on a fast mirror and finishes early: [==seg0==....][ done ][==seg2==....][==seg3==....] ▲ largest remaining tail Steal: seg1 restarts on the second half of seg2's remaining range: [==seg0==....][ done ][=seg2=][==seg1'==][==seg3==....] ``` Rules: - Default 8 segments, user-configurable 1–32, clamped per-host by settings (some hosts ban >4 connections; keep a per-host override table). - Never split below `min_segment_bytes` (default 1 MiB) — more segments than that is pure overhead and gets you rate-limited. - On finish, a worker steals the **second half of the largest remaining range**, atomically under the task lock, and only if that half is ≥ `min_segment_bytes`. - A segment that fails 3× with a connection error is not retried on the same host if a mirror exists; the range is returned to the pool and re-split. - Non-resumable servers → exactly 1 segment, and the UI must say so (IDM's "Resume capability: No"). ## 4. Disk I/O — the buffer setting you asked for One file, opened once, `O_WRONLY`. Each segment `pwrite()`s at its own absolute offset, so **there is no reassembly pass and no second write of the whole file.** - `posix_fallocate()` the full size up front → contiguous extents, no ENOSPC surprise at 99 %, no fragmentation. - Per-segment ring buffer, size = **`buffer_bytes`** (the user-visible "Buffer size" setting). Default 4 MiB, range 64 KiB – 64 MiB. Curl's write callback appends; the buffer is flushed with a single `pwrite` when full or when the segment ends. *This is the single biggest throughput knob and it is exposed in the UI: Options → Downloads → "Write buffer per connection".* - Global cap `max_total_buffer_bytes` (default 256 MiB) so 32 segments × 64 MiB can't OOM the box. The per-segment value is silently reduced to fit and the effective value is reported back to the UI. - `posix_fadvise(POSIX_FADV_DONTNEED)` on written ranges — do not let a 40 GB ISO evict the user's entire page cache. - `fdatasync()` on a timer (default 5 s) and on pause, **not** per write. - `io_uring` is a **post-1.0 optimization**, gated behind a benchmark in `tools/bench/` that must show ≥10 % improvement on NVMe. Do not start there. ## 5. Resume metadata — `.veloxpart.meta` Written next to the part file so a download survives a daemon crash, a reboot, *and* a database loss. Little-endian, versioned, `fdatasync`'d on every segment-boundary update: ``` magic "VDMP" | u16 version | u16 flags u64 total_size | u64 downloaded url_set (original + effective + mirrors, length-prefixed UTF-8) etag | last_modified | content_type u32 segment_count per segment: u64 start, u64 end, u64 completed sha256_partial_state (optional, for streaming hash) crc32 of the whole record ``` On resume, revalidate with `If-Range: `. If the server answers `200` instead of `206`, the file changed underneath us: surface it as "File on server has changed — restart download?" rather than silently corrupting the file. This is the single most common way download managers produce broken files. Do not get it wrong. ## 6. Rate limiting Hierarchical token buckets: global → per-queue → per-task. Refill on a 100 ms tick; throttle by delaying curl reads (`CURLOPT_MAX_RECV_SPEED_LARGE` as a coarse floor, plus our own pause/unpause via `curl_easy_pause` for precision). "Speed Limiter" in the UI toggles between Full speed / a saved limit, exactly as IDM does. ## 7. Failure policy | Case | Behaviour | |---|---| | 5xx / timeout / reset | Exponential backoff 1 s→2→4→8→…→cap 60 s, jitter ±20 %, `max_retries` default 10 | | 401/407 | Emit `event.auth.required`, pause task, GUI shows credential dialog, store in secret service (never in SQLite) | | 403 after redirect | Retry once with the original referrer; many CDNs require it | | 416 | Metadata is stale → re-probe, re-split | | Disk full | Pause all, one notification, do not spin | | Server drops range support mid-download | Demote to 1 segment, keep what's on disk, continue | ## 8. Performance targets (M7 gate, `tools/bench/`) - Saturate a 1 Gbit link with ≤ 8 % of one core. - ≤ 60 MB RSS with 20 active downloads at default buffers. - 10 000-row task list: RPC `download.list` under 50 ms, GUI scroll at 60 fps. - No allocation in the curl write callback hot path (ring buffer is preallocated).