vdm/docs/04-engine-design.md

04 — Download engine design (libveloxcore)

Owner: lane CORE. This is the part that has to be genuinely fast.

1. Task lifecycle

NEW → PROBING → QUEUED → CONNECTING → DOWNLOADING ⇄ PAUSED
                                          │  │
                                          │  └→ RETRY_WAIT → CONNECTING
                                          ▼
                                     ASSEMBLING → VERIFYING → COMPLETE
                                          │
                                          └→ FAILED  |  CANCELLED

ASSEMBLING is normally a no-op rename (see §4). It exists as a real state only for the HLS/DASH path, which must mux.

2. Probe (download.probe)

Feeds IDM's "Download File Info" dialog, so it must be fast and never partially download.

1. HEAD with the browser's headers/cookies/referrer/UA verbatim.
2. If HEAD is refused (405/403 — common), fall back to GET with Range: bytes=0-0 and abort after headers.
3. Extract: final URL after redirects, Content-Length, Content-Type, Content-Disposition filename (RFC 5987/6266, UTF-8 and legacy), Accept-Ranges, ETag, Last-Modified.
4. Resumability is proven, not assumed: treat as resumable only if the range request returned 206 and Content-Range matches. Servers lie about Accept-Ranges.
5. Filename resolution order: explicit user name → Content-Disposition → last path segment (percent-decoded) → Content-Type extension → download.bin. Then sanitize: strip / \0, control chars, trailing dots/spaces, cap at 200 bytes of UTF-8 (never split a codepoint), reject . and ...

3. Segmentation — the part IDM is actually famous for

Static N-way splitting is not what makes IDM feel fast; dynamic segment stealing is.

Initial:   [====seg0====][====seg1====][====seg2====][====seg3====]
seg1 is on a fast mirror and finishes early:
           [==seg0==....][    done    ][==seg2==....][==seg3==....]
                                        ▲ largest remaining tail
Steal:     seg1 restarts on the second half of seg2's remaining range:
           [==seg0==....][ done ][=seg2=][==seg1'==][==seg3==....]

Rules:

• Default 8 segments, user-configurable 1–32, clamped per-host by settings (some hosts ban >4 connections; keep a per-host override table).
• Never split below min_segment_bytes (default 1 MiB) — more segments than that is pure overhead and gets you rate-limited.
• On finish, a worker steals the second half of the largest remaining range, atomically under the task lock, and only if that half is ≥ min_segment_bytes.
• A segment that fails 3× with a connection error is not retried on the same host if a mirror exists; the range is returned to the pool and re-split.
• Non-resumable servers → exactly 1 segment, and the UI must say so (IDM's "Resume capability: No").

4. Disk I/O — the buffer setting you asked for

One file, opened once, O_WRONLY. Each segment pwrite()s at its own absolute offset, so there is no reassembly pass and no second write of the whole file.

• posix_fallocate() the full size up front → contiguous extents, no ENOSPC surprise at 99 %, no fragmentation.
• Per-segment ring buffer, size = buffer_bytes (the user-visible "Buffer size" setting, connection.bufferBytes on the wire). Default 1 MiB, range 64 KiB – 16 MiB. Curl's write callback appends; the buffer is flushed with a single pwrite when full or when the segment ends. This is the single biggest throughput knob and it is exposed in the UI: Options → Downloads → "Write buffer per connection".
• Global cap max_total_buffer_bytes (connection.maxTotalBufferBytes, default 128 MiB) so a burst of large downloads with a large per-segment buffer can't OOM the box. Combined with max_active_segments (connection.maxActiveSegments, default 32) — the ceiling on segments actually transferring at once, across every task, not per download — every live segment's buffer is reduced to fit max_total_buffer_bytes / live_segment_count (capped by max_active_segments), never below the 64 KiB floor. The requested and effective values are both reported back to the UI (TaskDetail.bufferBytes / .effectiveBufferBytes) so it can show, for example, "16 MiB (using 4 MiB)". See docs/adr/0012-buffer-and-segment-budget.md for the reasoning behind these numbers, including why 4 MiB / 64 MiB / 256 MiB (this section's earlier draft) does not hold ≤ 60 MB RSS once buffers are counted per segment rather than per download.
• posix_fadvise(POSIX_FADV_DONTNEED) on written ranges — do not let a 40 GB ISO evict the user's entire page cache.
• fdatasync() on a timer (default 5 s) and on pause, not per write.
• io_uring is a post-1.0 optimization, gated behind a benchmark in tools/bench/ that must show ≥10 % improvement on NVMe. Do not start there.

5. Resume metadata — <name>.veloxpart.meta

Written next to the part file so a download survives a daemon crash, a reboot, and a database loss. Little-endian, versioned, fdatasync'd on every segment-boundary update:

magic "VDMP" | u16 version | u16 flags
u64 total_size | u64 downloaded
url_set (original + effective + mirrors, length-prefixed UTF-8)
etag | last_modified | content_type
u32 segment_count
  per segment: u64 start, u64 end, u64 completed
sha256_partial_state (optional, for streaming hash)
crc32 of the whole record

On resume, revalidate with If-Range: <etag or last-modified>. If the server answers 200 instead of 206, the file changed underneath us: surface it as "File on server has changed — restart download?" rather than silently corrupting the file. This is the single most common way download managers produce broken files. Do not get it wrong.

6. Rate limiting

Hierarchical token buckets: global → per-queue → per-task. Refill on a 100 ms tick; throttle by delaying curl reads (CURLOPT_MAX_RECV_SPEED_LARGE as a coarse floor, plus our own pause/unpause via curl_easy_pause for precision). "Speed Limiter" in the UI toggles between Full speed / a saved limit, exactly as IDM does.

7. Failure policy

Case	Behaviour
5xx / timeout / reset	Exponential backoff 1 s→2→4→8→…→cap 60 s, jitter ±20 %, max_retries default 10
401/407	Emit event.auth.required, pause task, GUI shows credential dialog, store in secret service (never in SQLite)
403 after redirect	Retry once with the original referrer; many CDNs require it
416	Metadata is stale → re-probe, re-split
Disk full	Pause all, one notification, do not spin
Server drops range support mid-download	Demote to 1 segment, keep what's on disk, continue

8. Performance targets (M7 gate, tools/bench/)

• Saturate a 1 Gbit link with ≤ 8 % of one core.
• ≤ 60 MB RSS with 20 active downloads at default buffers, given max_active_segments = 32 — without that cap, 20 downloads × 8 segments each is 160 live buffers even at the 1 MiB default, and the number does not hold. See docs/adr/0012-buffer-and-segment-budget.md.
• 10 000-row task list: RPC download.list under 50 ms, GUI scroll at 60 fps.
• No allocation in the curl write callback hot path (ring buffer is preallocated).