vdm/docs/04-engine-design.md

# 04 — Download engine design (`libveloxcore`)

Owner: lane **CORE**. This is the part that has to be genuinely fast.

## 1. Task lifecycle

```
NEW → PROBING → QUEUED → CONNECTING → DOWNLOADING ⇄ PAUSED
                                          │  │
                                          │  └→ RETRY_WAIT → CONNECTING
                                          ▼
                                     ASSEMBLING → VERIFYING → COMPLETE
                                          │
                                          └→ FAILED  |  CANCELLED
```

`ASSEMBLING` is normally a no-op rename (see §4). It exists as a real state only for the
HLS/DASH path, which must mux.

## 2. Probe (`download.probe`)

Feeds IDM's "Download File Info" dialog, so it must be fast and never partially download.

1. `HEAD` with the browser's headers/cookies/referrer/UA verbatim.
2. If `HEAD` is refused (405/403 — common), fall back to `GET` with `Range: bytes=0-0` and
   abort after headers.
3. Extract: final URL after redirects, `Content-Length`, `Content-Type`,
   `Content-Disposition` filename (RFC 5987/6266, UTF-8 and legacy), `Accept-Ranges`,
   `ETag`, `Last-Modified`.
4. **Resumability is proven, not assumed:** treat as resumable only if the range request
   returned `206` *and* `Content-Range` matches. Servers lie about `Accept-Ranges`.
5. Filename resolution order: explicit user name → `Content-Disposition` → last path
   segment (percent-decoded) → `Content-Type` extension → `download.bin`.
   Then sanitize: strip `/` `\0`, control chars, trailing dots/spaces, cap at 200 bytes
   *of UTF-8* (never split a codepoint), reject `.` and `..`.

## 3. Segmentation — the part IDM is actually famous for

Static N-way splitting is not what makes IDM feel fast; **dynamic segment stealing** is.

```
Initial:   [====seg0====][====seg1====][====seg2====][====seg3====]
seg1 is on a fast mirror and finishes early:
           [==seg0==....][    done    ][==seg2==....][==seg3==....]
                                        ▲ largest remaining tail
Steal:     seg1 restarts on the second half of seg2's remaining range:
           [==seg0==....][ done ][=seg2=][==seg1'==][==seg3==....]
```

Rules:
- Default 8 segments, user-configurable 1–32, clamped per-host by settings (some hosts
  ban >4 connections; keep a per-host override table).
- Never split below `min_segment_bytes` (default 1 MiB) — more segments than that is pure
  overhead and gets you rate-limited.
- On finish, a worker steals the **second half of the largest remaining range**, atomically
  under the task lock, and only if that half is ≥ `min_segment_bytes`.
- A segment that fails 3× with a connection error is not retried on the same host if a
  mirror exists; the range is returned to the pool and re-split.
- Non-resumable servers → exactly 1 segment, and the UI must say so
  (IDM's "Resume capability: No").

## 4. Disk I/O — the buffer setting you asked for

One file, opened once, `O_WRONLY`. Each segment `pwrite()`s at its own absolute offset, so
**there is no reassembly pass and no second write of the whole file.**

- `posix_fallocate()` the full size up front → contiguous extents, no ENOSPC surprise at
  99 %, no fragmentation.
- Per-segment ring buffer, size = **`buffer_bytes`** (the user-visible "Buffer size"
  setting). Default 4 MiB, range 64 KiB – 64 MiB. Curl's write callback appends; the
  buffer is flushed with a single `pwrite` when full or when the segment ends.
  *This is the single biggest throughput knob and it is exposed in the UI: Options →
  Downloads → "Write buffer per connection".*
- Global cap `max_total_buffer_bytes` (default 256 MiB) so 32 segments × 64 MiB can't OOM
  the box. The per-segment value is silently reduced to fit and the effective value is
  reported back to the UI.
- `posix_fadvise(POSIX_FADV_DONTNEED)` on written ranges — do not let a 40 GB ISO evict
  the user's entire page cache.
- `fdatasync()` on a timer (default 5 s) and on pause, **not** per write.
- `io_uring` is a **post-1.0 optimization**, gated behind a benchmark in `tools/bench/`
  that must show ≥10 % improvement on NVMe. Do not start there.

## 5. Resume metadata — `<name>.veloxpart.meta`

Written next to the part file so a download survives a daemon crash, a reboot, *and* a
database loss. Little-endian, versioned, `fdatasync`'d on every segment-boundary update:

```
magic "VDMP" | u16 version | u16 flags
u64 total_size | u64 downloaded
url_set (original + effective + mirrors, length-prefixed UTF-8)
etag | last_modified | content_type
u32 segment_count
  per segment: u64 start, u64 end, u64 completed
sha256_partial_state (optional, for streaming hash)
crc32 of the whole record
```

On resume, revalidate with `If-Range: <etag or last-modified>`. If the server answers
`200` instead of `206`, the file changed underneath us: surface it as
"File on server has changed — restart download?" rather than silently corrupting the file.
This is the single most common way download managers produce broken files. Do not get it wrong.

## 6. Rate limiting

Hierarchical token buckets: global → per-queue → per-task. Refill on a 100 ms tick;
throttle by delaying curl reads (`CURLOPT_MAX_RECV_SPEED_LARGE` as a coarse floor, plus
our own pause/unpause via `curl_easy_pause` for precision). "Speed Limiter" in the UI
toggles between Full speed / a saved limit, exactly as IDM does.

## 7. Failure policy

| Case | Behaviour |
|---|---|
| 5xx / timeout / reset | Exponential backoff 1 s→2→4→8→…→cap 60 s, jitter ±20 %, `max_retries` default 10 |
| 401/407 | Emit `event.auth.required`, pause task, GUI shows credential dialog, store in secret service (never in SQLite) |
| 403 after redirect | Retry once with the original referrer; many CDNs require it |
| 416 | Metadata is stale → re-probe, re-split |
| Disk full | Pause all, one notification, do not spin |
| Server drops range support mid-download | Demote to 1 segment, keep what's on disk, continue |

## 8. Performance targets (M7 gate, `tools/bench/`)

- Saturate a 1 Gbit link with ≤ 8 % of one core.
- ≤ 60 MB RSS with 20 active downloads at default buffers.
- 10 000-row task list: RPC `download.list` under 50 ms, GUI scroll at 60 fps.
- No allocation in the curl write callback hot path (ring buffer is preallocated).