scaffold: project structure, wire contract, roadmap and agent briefs
Lays out Velox Download Manager (IDM-class download manager for Ubuntu 26.04) as a monorepo ready for parallel lane development. No implementation code by design. - docs/: architecture, roadmap M0-M7, IDM-parity GUI spec, engine design, Firefox extension spec, risks/spikes, packaging - contracts/: wire-contract skeleton (JSON Schema + fixture templates) — the single synchronization point between lanes - docs/agents/: one brief per lane (PROTO, CORE, DAEMON, GUI, EXT, PKG/QA) with owned directories, build order and definition of done - CLAUDE.md: rules of engagement — lane ownership, layering, non-negotiables - CMake scaffolding with dev/tsan/release/ci presets Two environment findings shape the design: Firefox here is the Mozilla snap (native-messaging risk, so the extension carries a loopback-WebSocket fallback), and Wayland forbids passive clipboard monitoring (so clipboard capture is explicit-action-first). Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Claude Opus 5
@@ -0,0 +1,127 @@
|
||||
# 04 — Download engine design (`libveloxcore`)
|
||||
|
||||
Owner: lane **CORE**. This is the part that has to be genuinely fast.
|
||||
|
||||
## 1. Task lifecycle
|
||||
|
||||
```
|
||||
NEW → PROBING → QUEUED → CONNECTING → DOWNLOADING ⇄ PAUSED
|
||||
│ │
|
||||
│ └→ RETRY_WAIT → CONNECTING
|
||||
▼
|
||||
ASSEMBLING → VERIFYING → COMPLETE
|
||||
│
|
||||
└→ FAILED | CANCELLED
|
||||
```
|
||||
|
||||
`ASSEMBLING` is normally a no-op rename (see §4). It exists as a real state only for the
|
||||
HLS/DASH path, which must mux.
|
||||
|
||||
## 2. Probe (`download.probe`)
|
||||
|
||||
Feeds IDM's "Download File Info" dialog, so it must be fast and never partially download.
|
||||
|
||||
1. `HEAD` with the browser's headers/cookies/referrer/UA verbatim.
|
||||
2. If `HEAD` is refused (405/403 — common), fall back to `GET` with `Range: bytes=0-0` and
|
||||
abort after headers.
|
||||
3. Extract: final URL after redirects, `Content-Length`, `Content-Type`,
|
||||
`Content-Disposition` filename (RFC 5987/6266, UTF-8 and legacy), `Accept-Ranges`,
|
||||
`ETag`, `Last-Modified`.
|
||||
4. **Resumability is proven, not assumed:** treat as resumable only if the range request
|
||||
returned `206` *and* `Content-Range` matches. Servers lie about `Accept-Ranges`.
|
||||
5. Filename resolution order: explicit user name → `Content-Disposition` → last path
|
||||
segment (percent-decoded) → `Content-Type` extension → `download.bin`.
|
||||
Then sanitize: strip `/` `\0`, control chars, trailing dots/spaces, cap at 200 bytes
|
||||
*of UTF-8* (never split a codepoint), reject `.` and `..`.
|
||||
|
||||
## 3. Segmentation — the part IDM is actually famous for
|
||||
|
||||
Static N-way splitting is not what makes IDM feel fast; **dynamic segment stealing** is.
|
||||
|
||||
```
|
||||
Initial: [====seg0====][====seg1====][====seg2====][====seg3====]
|
||||
seg1 is on a fast mirror and finishes early:
|
||||
[==seg0==....][ done ][==seg2==....][==seg3==....]
|
||||
▲ largest remaining tail
|
||||
Steal: seg1 restarts on the second half of seg2's remaining range:
|
||||
[==seg0==....][ done ][=seg2=][==seg1'==][==seg3==....]
|
||||
```
|
||||
|
||||
Rules:
|
||||
- Default 8 segments, user-configurable 1–32, clamped per-host by settings (some hosts
|
||||
ban >4 connections; keep a per-host override table).
|
||||
- Never split below `min_segment_bytes` (default 1 MiB) — more segments than that is pure
|
||||
overhead and gets you rate-limited.
|
||||
- On finish, a worker steals the **second half of the largest remaining range**, atomically
|
||||
under the task lock, and only if that half is ≥ `min_segment_bytes`.
|
||||
- A segment that fails 3× with a connection error is not retried on the same host if a
|
||||
mirror exists; the range is returned to the pool and re-split.
|
||||
- Non-resumable servers → exactly 1 segment, and the UI must say so
|
||||
(IDM's "Resume capability: No").
|
||||
|
||||
## 4. Disk I/O — the buffer setting you asked for
|
||||
|
||||
One file, opened once, `O_WRONLY`. Each segment `pwrite()`s at its own absolute offset, so
|
||||
**there is no reassembly pass and no second write of the whole file.**
|
||||
|
||||
- `posix_fallocate()` the full size up front → contiguous extents, no ENOSPC surprise at
|
||||
99 %, no fragmentation.
|
||||
- Per-segment ring buffer, size = **`buffer_bytes`** (the user-visible "Buffer size"
|
||||
setting). Default 4 MiB, range 64 KiB – 64 MiB. Curl's write callback appends; the
|
||||
buffer is flushed with a single `pwrite` when full or when the segment ends.
|
||||
*This is the single biggest throughput knob and it is exposed in the UI: Options →
|
||||
Downloads → "Write buffer per connection".*
|
||||
- Global cap `max_total_buffer_bytes` (default 256 MiB) so 32 segments × 64 MiB can't OOM
|
||||
the box. The per-segment value is silently reduced to fit and the effective value is
|
||||
reported back to the UI.
|
||||
- `posix_fadvise(POSIX_FADV_DONTNEED)` on written ranges — do not let a 40 GB ISO evict
|
||||
the user's entire page cache.
|
||||
- `fdatasync()` on a timer (default 5 s) and on pause, **not** per write.
|
||||
- `io_uring` is a **post-1.0 optimization**, gated behind a benchmark in `tools/bench/`
|
||||
that must show ≥10 % improvement on NVMe. Do not start there.
|
||||
|
||||
## 5. Resume metadata — `<name>.veloxpart.meta`
|
||||
|
||||
Written next to the part file so a download survives a daemon crash, a reboot, *and* a
|
||||
database loss. Little-endian, versioned, `fdatasync`'d on every segment-boundary update:
|
||||
|
||||
```
|
||||
magic "VDMP" | u16 version | u16 flags
|
||||
u64 total_size | u64 downloaded
|
||||
url_set (original + effective + mirrors, length-prefixed UTF-8)
|
||||
etag | last_modified | content_type
|
||||
u32 segment_count
|
||||
per segment: u64 start, u64 end, u64 completed
|
||||
sha256_partial_state (optional, for streaming hash)
|
||||
crc32 of the whole record
|
||||
```
|
||||
|
||||
On resume, revalidate with `If-Range: <etag or last-modified>`. If the server answers
|
||||
`200` instead of `206`, the file changed underneath us: surface it as
|
||||
"File on server has changed — restart download?" rather than silently corrupting the file.
|
||||
This is the single most common way download managers produce broken files. Do not get it wrong.
|
||||
|
||||
## 6. Rate limiting
|
||||
|
||||
Hierarchical token buckets: global → per-queue → per-task. Refill on a 100 ms tick;
|
||||
throttle by delaying curl reads (`CURLOPT_MAX_RECV_SPEED_LARGE` as a coarse floor, plus
|
||||
our own pause/unpause via `curl_easy_pause` for precision). "Speed Limiter" in the UI
|
||||
toggles between Full speed / a saved limit, exactly as IDM does.
|
||||
|
||||
## 7. Failure policy
|
||||
|
||||
| Case | Behaviour |
|
||||
|---|---|
|
||||
| 5xx / timeout / reset | Exponential backoff 1 s→2→4→8→…→cap 60 s, jitter ±20 %, `max_retries` default 10 |
|
||||
| 401/407 | Emit `event.auth.required`, pause task, GUI shows credential dialog, store in secret service (never in SQLite) |
|
||||
| 403 after redirect | Retry once with the original referrer; many CDNs require it |
|
||||
| 416 | Metadata is stale → re-probe, re-split |
|
||||
| Disk full | Pause all, one notification, do not spin |
|
||||
| Server drops range support mid-download | Demote to 1 segment, keep what's on disk, continue |
|
||||
|
||||
## 8. Performance targets (M7 gate, `tools/bench/`)
|
||||
|
||||
- Saturate a 1 Gbit link with ≤ 8 % of one core.
|
||||
- ≤ 60 MB RSS with 20 active downloads at default buffers.
|
||||
- 10 000-row task list: RPC `download.list` under 50 ms, GUI scroll at 60 fps.
|
||||
- No allocation in the curl write callback hot path (ring buffer is preallocated).
|
||||
Reference in New Issue
Block a user