commit 8bb683b09dde98f152f64c919820eca77dbb37b6 Author: sami Date: Wed Sep 9 18:21:11 2026 +0400 scaffold: project structure, wire contract, roadmap and agent briefs Lays out Velox Download Manager (IDM-class download manager for Ubuntu 26.04) as a monorepo ready for parallel lane development. No implementation code by design. - docs/: architecture, roadmap M0-M7, IDM-parity GUI spec, engine design, Firefox extension spec, risks/spikes, packaging - contracts/: wire-contract skeleton (JSON Schema + fixture templates) — the single synchronization point between lanes - docs/agents/: one brief per lane (PROTO, CORE, DAEMON, GUI, EXT, PKG/QA) with owned directories, build order and definition of done - CLAUDE.md: rules of engagement — lane ownership, layering, non-negotiables - CMake scaffolding with dev/tsan/release/ci presets Two environment findings shape the design: Firefox here is the Mozilla snap (native-messaging risk, so the extension carries a loopback-WebSocket fallback), and Wayland forbids passive clipboard monitoring (so clipboard capture is explicit-action-first). Co-Authored-By: Claude Opus 5 diff --git a/.github/workflows/.gitkeep b/.github/workflows/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..26bd4ab --- /dev/null +++ b/.gitignore @@ -0,0 +1,15 @@ +build/ +.cache/ +compile_commands.json +node_modules/ +dist/ +web-ext-artifacts/ +*.o +*.so +*.a +*.log +.venv/ +__pycache__/ +.DS_Store +*.veloxpart +*.veloxpart.meta diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..d6ce8df --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,70 @@ +# Rules of engagement — agents working in this repo + +Read this before touching anything. Then read your lane brief in [docs/agents/](docs/agents/). + +## 1. Stay in your lane + +| Lane | Owns | Never writes | +|---|---|---| +| PROTO | `contracts/`, `tools/mockd/`, `tests/conformance/` | everything else | +| CORE | `core/`, `tools/bench/`, `tools/fuzz/` | `daemon/`, `gui/`, `extension/`, `contracts/` | +| DAEMON | `daemon/`, `cli/`, `nmhost/`, `packaging/nativehost/` | `core/`, `gui/`, `extension/`, `contracts/` | +| GUI | `gui/` | everything else | +| EXT | `extension/` | everything else | +| PKG/QA | `packaging/`, `.github/`, `tools/testserver/`, `tests/integration/`, `tests/e2e/`, root build files | any lane's feature code | + +If your task seems to require editing another lane's files, that is a signal the interface +is wrong. **File the request; don't reach across.** + +## 2. `contracts/` is sacred + +- Only PROTO commits there. +- Generated code (`core/generated/`, `extension/src/shared/protocol/`) is committed and + **must never be hand-edited**. Fix the schema, regenerate. +- Need a new field? Open a `contracts/`-only PR: schema + fixtures + regenerated code + + `VERSION` bump. Optional field or new method = minor; rename/remove/retype = major + ADR. +- Working around a wrong contract locally is the single failure mode most likely to sink + this project. Don't. + +## 3. Layering + +``` +core → no JSON, no SQL, no Qt, no RPC. Ever. +daemon → depends on core. No Qt. +gui / ext → zero download logic. They render state and forward user intent. +nmhost → a dumb pipe. Under 300 lines. No logic. +``` + +A grep for `curl|pwrite|sqlite` in `gui/` must come back empty. Same for download logic in +`extension/`. + +## 4. Non-negotiable behaviours + +- **Capture fails open.** Daemon down, slow, or erroring → Firefox downloads normally. + Never swallow a user's download. `capture.offer` answers within 750 ms or the extension + gives up. +- **Resume is validated.** `If-Range` with ETag/Last-Modified; a `200` where `206` was + expected means the file changed — ask the user, never silently corrupt. +- **Never bind beyond `127.0.0.1`.** The WS transport is token-authenticated, origin-checked, + and rate-limited. +- **Secrets go to the Secret Service**, never SQLite, never logs. +- **Paths are canonicalized** and checked against allowed roots before any write. +- **No allocation in the transfer hot path.** + +## 5. Definition of done, everywhere + +Code + tests + docs updated in the same change. A feature with no test does not exist. If +you change observable behaviour, update the doc in `docs/` that describes it in the same PR. + +## 6. Style + +- C++23, `-Wall -Wextra -Werror`, clang-format (config at root), clang-tidy clean. +- TypeScript strict mode, ESLint, no `any` on protocol boundaries. +- Commit messages: `lane: imperative summary` (e.g. `core: add dynamic segment stealing`). +- One logical change per commit. Rebase onto `main`; no merge commits. + +## 7. When you're unsure + +Ask in the PR rather than guessing at the interface. A day of clarification is cheaper than +an M2 integration rewrite. And record real decisions as an ADR in `docs/adr/` — the next +agent to touch this will have none of your context. diff --git a/CMakeLists.txt b/CMakeLists.txt new file mode 100644 index 0000000..c93d480 --- /dev/null +++ b/CMakeLists.txt @@ -0,0 +1,48 @@ +# Velox Download Manager — top-level build +# SCAFFOLDING ONLY. Lane PKG/QA owns this file; each add_subdirectory() is enabled by the +# owning lane when it lands its first target. Nothing here builds yet by design. + +cmake_minimum_required(VERSION 3.28) + +project(velox + VERSION 0.1.0 + DESCRIPTION "IDM-class download manager for Linux" + LANGUAGES CXX) + +set(CMAKE_CXX_STANDARD 23) +set(CMAKE_CXX_STANDARD_REQUIRED ON) +set(CMAKE_CXX_EXTENSIONS OFF) +set(CMAKE_EXPORT_COMPILE_COMMANDS ON) + +option(VELOX_BUILD_GUI "Build the Qt 6 GUI client" ON) +option(VELOX_BUILD_TESTS "Build unit and integration tests" ON) +option(VELOX_BUILD_FUZZ "Build libFuzzer targets (clang only)" OFF) +option(VELOX_ENABLE_MEDIA "Build the HLS/DASH media grabber (M4)" OFF) + +add_compile_options(-Wall -Wextra -Wpedantic) +if(CMAKE_BUILD_TYPE STREQUAL "Release" OR CMAKE_BUILD_TYPE STREQUAL "RelWithDebInfo") + add_compile_options(-Werror) +endif() + +# --- Dependencies (see README bootstrap; none are vendored) -------------------- +# find_package(CURL 8.0 REQUIRED) +# find_package(SQLite3 REQUIRED) +# find_package(nlohmann_json 3.11 REQUIRED) +# find_package(OpenSSL REQUIRED) +# if(VELOX_BUILD_GUI) +# find_package(Qt6 6.6 REQUIRED COMPONENTS Widgets Svg Network LinguistTools) +# qt_standard_project_setup() +# endif() + +# --- Targets: each lane enables its own line ---------------------------------- +# add_subdirectory(core) # lane CORE → libveloxcore +# add_subdirectory(daemon) # lane DAEMON → veloxd +# add_subdirectory(cli) # lane DAEMON → velox +# add_subdirectory(nmhost) # lane DAEMON → velox-nmhost +# if(VELOX_BUILD_GUI) +# add_subdirectory(gui) # lane GUI → velox-gui +# endif() +# if(VELOX_BUILD_TESTS) +# enable_testing() +# add_subdirectory(tests) +# endif() diff --git a/CMakePresets.json b/CMakePresets.json new file mode 100644 index 0000000..ae78247 --- /dev/null +++ b/CMakePresets.json @@ -0,0 +1,63 @@ +{ + "version": 6, + "cmakeMinimumRequired": { "major": 3, "minor": 28, "patch": 0 }, + "configurePresets": [ + { + "name": "base", + "hidden": true, + "generator": "Ninja", + "binaryDir": "${sourceDir}/build/${presetName}", + "cacheVariables": { "CMAKE_EXPORT_COMPILE_COMMANDS": "ON" } + }, + { + "name": "dev", + "inherits": "base", + "displayName": "Debug + ASan/UBSan (default for all lanes)", + "cacheVariables": { + "CMAKE_BUILD_TYPE": "Debug", + "CMAKE_CXX_FLAGS": "-fsanitize=address,undefined -fno-omit-frame-pointer -g", + "VELOX_BUILD_TESTS": "ON" + } + }, + { + "name": "tsan", + "inherits": "base", + "displayName": "Debug + ThreadSanitizer (required for CORE and DAEMON)", + "cacheVariables": { + "CMAKE_BUILD_TYPE": "Debug", + "CMAKE_CXX_FLAGS": "-fsanitize=thread -fno-omit-frame-pointer -g", + "VELOX_BUILD_TESTS": "ON" + } + }, + { + "name": "release", + "inherits": "base", + "displayName": "RelWithDebInfo + LTO (benchmarks and packages)", + "cacheVariables": { + "CMAKE_BUILD_TYPE": "RelWithDebInfo", + "CMAKE_INTERPROCEDURAL_OPTIMIZATION": "ON", + "VELOX_BUILD_TESTS": "OFF" + } + }, + { + "name": "ci", + "inherits": "dev", + "displayName": "CI build", + "cacheVariables": { "VELOX_BUILD_FUZZ": "ON" } + } + ], + "buildPresets": [ + { "name": "dev", "configurePreset": "dev" }, + { "name": "tsan", "configurePreset": "tsan" }, + { "name": "release", "configurePreset": "release" }, + { "name": "ci", "configurePreset": "ci" } + ], + "testPresets": [ + { + "name": "dev", + "configurePreset": "dev", + "output": { "outputOnFailure": true }, + "execution": { "noTestsAction": "error", "stopOnFailure": false } + } + ] +} diff --git a/README.md b/README.md new file mode 100644 index 0000000..37ee49f --- /dev/null +++ b/README.md @@ -0,0 +1,114 @@ +# Velox Download Manager (VDM) + +An IDM-class download manager for Ubuntu 26.04 LTS: multi-segment accelerated HTTP(S) +downloading, resume, categories and automatic file distribution, queues and scheduler, +speed limiter, a Firefox extension that captures downloads automatically, and clipboard +link capture. + +> **Name is a placeholder.** Binaries are `veloxd`, `velox-gui`, `velox`, `velox-nmhost`. +> Rename before first release if you want something else — do it in M0, never later. + +--- + +## Status + +**Phase M0 — scaffolding.** No implementation code exists yet. This repository currently +contains the architecture, the wire contract, the roadmap, and one brief per build lane +so that several agents can work in parallel without colliding. + +Start here: + +| Document | What it answers | +|---|---| +| [docs/01-architecture.md](docs/01-architecture.md) | Process model, why four binaries, framework choices and why | +| [docs/02-roadmap.md](docs/02-roadmap.md) | Milestones M0–M7, what runs in parallel, exit gates | +| [docs/03-gui-spec.md](docs/03-gui-spec.md) | IDM-parity UI: every window, dialog, column, menu | +| [docs/04-engine-design.md](docs/04-engine-design.md) | Segmentation, resume, buffers, rate limiting, disk I/O | +| [docs/05-extension-spec.md](docs/05-extension-spec.md) | Firefox capture, transports, pairing, media grabber | +| [docs/06-risks-and-spikes.md](docs/06-risks-and-spikes.md) | Snap Firefox, Wayland clipboard, and the other landmines | +| [docs/07-packaging.md](docs/07-packaging.md) | .deb/PPA, Flatpak, AMO signing, install layout | +| [contracts/README.md](contracts/README.md) | **The interface.** Both sides build against this | +| [CLAUDE.md](CLAUDE.md) | Rules of engagement for agents working in this repo | + +Per-lane briefs live in [docs/agents/](docs/agents/) — one per agent, each with an owned +directory list, a definition of done, and the files it must never touch. + +--- + +## Architecture in one picture + +``` + ┌────────────────────┐ native messaging (stdio JSON) ┌──────────────┐ + │ Firefox extension │◄───────────── or ──────────────►│ velox-nmhost │ + │ (MV3, TS) │ loopback WS 127.0.0.1 + token └──────┬───────┘ + └────────────────────┘ │ + │ + ┌────────────────────┐ ▼ + │ velox-gui (Qt 6) │◄──── JSON-RPC 2.0 over Unix socket ──► ┌─────────────┐ + └────────────────────┘ $XDG_RUNTIME_DIR/velox/velox.sock │ veloxd │ + │ (daemon) │ + ┌────────────────────┐ │ │ + │ velox (CLI) │◄───────────────────────────────────────┤ libveloxcore│ + └────────────────────┘ └──────┬──────┘ + │ + SQLite + sparse files +``` + +The daemon owns all state and all sockets. The GUI is a *view* — closing it does not stop +a download. The extension never touches the disk; it hands URL + headers + cookies to the +daemon and gets a task id back. + +--- + +## Repository layout + +``` +vdm/ +├── contracts/ ⭐ Wire contract: JSON Schema, fixtures, codegen. Frozen per version. +├── core/ C++23 libveloxcore — engine. No UI, no RPC, no SQL. +├── daemon/ C++23 veloxd — RPC server, scheduler, queues, SQLite store. +├── gui/ C++23 velox-gui — Qt 6 Widgets, IDM-parity UI. +├── cli/ C++23 velox — scriptable client. +├── nmhost/ C++23 velox-nmhost — Firefox native-messaging bridge (thin pipe). +├── extension/ TS Firefox MV3 WebExtension. +├── tools/ +│ ├── mockd/ TS mock daemon — lets GUI + extension work before veloxd exists. +│ ├── testserver/ Deliberately hostile HTTP server (no Range, flaky, redirects, auth). +│ ├── bench/ Throughput and CPU benchmarks. +│ └── fuzz/ libFuzzer targets for parsers. +├── tests/ +│ ├── conformance/ Protocol suite. Every lane must pass it. Gate for merging. +│ ├── integration/ veloxd + testserver. +│ └── e2e/ Playwright: real Firefox + real daemon + real file on disk. +├── packaging/ debian/, flatpak/, appimage/, native-host manifests. +└── docs/ Everything above, plus adr/ and agents/. +``` + +## Toolchain bootstrap + +Surveyed on this machine 2026-09-09 — **most of it is already installed**: + +| Present | Version | +|---|---| +| git · cmake · ninja · g++ · gdb | 2.53.0 · 4.2.3 · — · 15.2.0 (C++23 ready) | +| qt6-base-dev · qt6-tools-dev · qt6-tools-dev-tools | ✓ | +| libcurl4-openssl-dev · libsqlite3-dev · nlohmann-json3-dev · libssl-dev | ✓ | +| libavformat-dev · libavcodec-dev · ffmpeg | ✓ | +| clang-format · clang-tidy · python3 · pkg-config | ✓ | + +**Only these four are missing:** + +```bash +sudo apt update && sudo apt install -y \ + libqt6svg6-dev \ # GUI: SVG icon rendering + libsecret-1-dev \ # DAEMON: Secret Service for site logins + nodejs npm \ # EXT + PROTO: extension build, mockd, conformance runner + clang # optional: libFuzzer targets in M7 +``` + +Node in the 26.04 archive may lag; if the extension toolchain needs 22+, use `nvm`. + +Verify with `cmake --preset dev && cmake --build --preset dev` once lane CORE lands its +first target. Note **CMake 4.2.3** is installed — newer than the `3.28` floor in +`CMakeLists.txt`, and it hard-errors on `cmake_minimum_required` below 3.5, so no +dependency may ship a pre-3.5 CMake file. diff --git a/cli/README.md b/cli/README.md new file mode 100644 index 0000000..d04e007 --- /dev/null +++ b/cli/README.md @@ -0,0 +1 @@ +Owner: lane DAEMON. Scriptable client over the same JSON-RPC. diff --git a/cli/src/.gitkeep b/cli/src/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/contracts/README.md b/contracts/README.md new file mode 100644 index 0000000..86b786e --- /dev/null +++ b/contracts/README.md @@ -0,0 +1,114 @@ +# contracts/ — the wire contract + +**This directory is the interface between every lane.** Owner: agent **PROTO**. +Nobody else commits here. Everybody else *generates from* here. + +``` +contracts/ +├── VERSION # protocol semver, e.g. 1.0.0 +├── openrpc.json # human-readable API doc (generated from schema/) +├── schema/ +│ ├── envelope.schema.json # JSON-RPC 2.0 envelope + our error codes +│ ├── types/ # Task, Segment, Category, Queue, Settings, CaptureOffer… +│ ├── methods/ # one file per method: params + result +│ └── events/ # one file per server→client notification +├── fixtures/ # golden request/response pairs, replayed by conformance +└── codegen/ + ├── gen_cpp.py # → core/generated/ (structs + to_json/from_json) + └── gen_ts.py # → extension/src/shared/protocol/ (types + client) +``` + +## Rules + +1. **Generated code is committed.** No lane may be blocked because it can't run Python. +2. **Hand-editing generated files is a merge blocker.** Fix the schema and regenerate. +3. **Every method needs at least one fixture** — a success case and, where meaningful, an + error case. A method with no fixture is not done. +4. **Versioning:** adding an optional field or a new method → minor bump. Removing, + renaming, retyping, or changing a default → **major** bump and a written migration note + in `docs/adr/`. `session.hello` rejects a major mismatch with error `-32001` and a + message the GUI renders as "Velox needs updating". +5. **Changes arrive as a PR to `contracts/` alone**, containing: schema edit + fixtures + + regenerated code + `VERSION` bump. Lanes rebase onto it. This is the only synchronization + point in the whole project — keep it cheap and frequent rather than big and rare. + +## Transport framing + +| Client | Transport | Framing | +|---|---|---| +| GUI, CLI | `$XDG_RUNTIME_DIR/velox/velox.sock` | newline-delimited JSON (NDJSON) | +| nmhost ← Firefox | stdio | 4-byte little-endian length prefix (Firefox's format) | +| nmhost → daemon | same Unix socket | NDJSON | +| Extension (fallback) | `ws://127.0.0.1:520xx` | one JSON message per WS text frame | + +All four carry **the same JSON-RPC 2.0 payloads**. The framing differences stop at the +transport layer; no method behaves differently depending on how it arrived — except that +methods marked `"privileged": true` in the schema are refused over the WebSocket transport. + +## Method surface (v1.0.0 target — expand only via PR) + +### Session +| Method | Params → Result | +|---|---| +| `session.hello` | `{clientType, clientName, protocolVersion, token?}` → `{daemonVersion, protocolVersion, capabilities[], sessionId}` | +| `session.pair` | `{clientName, extensionId}` → `{token, expiresAt}` *(WS only; triggers user prompt)* | +| `session.subscribe` | `{events[]}` → `{ok}` | + +### Downloads +| Method | Params → Result | +|---|---| +| `download.probe` | `{url, headers?, cookies?, referrer?, userAgent?}` → `{filename, sizeBytes?, mime, resumable, effectiveUrl, suggestedCategoryId}` | +| `download.add` | `{url, headers?, cookies?, referrer?, userAgent?, filename?, saveDir?, categoryId?, segments?, bufferBytes?, startMode:"now"\|"later"\|"queue", queueId?, description?, checksum?}` → `{taskId, state}` | +| `download.addBatch` | `{items[], defaults}` → `{taskIds[]}` | +| `download.list` | `{filter?, sort?, offset?, limit?}` → `{total, items: TaskSummary[]}` | +| `download.get` | `{taskId}` → `TaskDetail` (includes `segments[]`) | +| `download.start` \| `.pause` \| `.resume` \| `.cancel` | `{taskIds[]}` → `{updated[]}` | +| `download.remove` | `{taskIds[], deleteFile:bool}` → `{removed[]}` | +| `download.update` | `{taskId, patch:{filename?, saveDir?, categoryId?, queueId?, description?, segments?, bufferBytes?}}` → `TaskSummary` | +| `download.refreshUrl` | `{taskId, url, headers?}` → `{ok}` *(IDM's "Refresh Download Address")* | + +### Organisation +`category.list` · `category.upsert` · `category.remove` · `queue.list` · `queue.upsert` · +`queue.start` · `queue.stop` · `queue.reorder` · `rules.list` · `rules.upsert` · +`schedule.get` · `schedule.set` + +### Settings & limits +`settings.get {keys?}` · `settings.set {values}` · `limiter.get` · `limiter.set {globalBps?, enabled}` + +### Browser integration +| Method | Notes | +|---|---| +| `capture.offer` | `{url, method, headers, cookies, contentType?, contentLength?, contentDisposition?, tabUrl, filename?}` → `{action:"take"\|"ignore", taskId?, reason?}` — **must answer within 750 ms**; the extension gives up and lets Firefox handle it otherwise | +| `capture.getRules` | Extension mirrors the daemon's monitored types so the two never disagree | +| `media.listVariants` | `{manifestUrl, headers}` → `{variants:[{id,resolution,bitrate,codec,sizeEstimate}]}` | +| `media.addVariant` | `{manifestUrl, variantId, ...addParams}` → `{taskId}` | + +### Grabber +`grabber.start {startUrl, depth, includePatterns[], excludePatterns[], fileTypes[]}` → +`{jobId}`; `grabber.status {jobId}`; `grabber.harvest {jobId, select[]}` → `{taskIds[]}` + +### Events (server → client notifications) +| Event | Payload | +|---|---| +| `event.task.added` / `.removed` | `{taskId, summary?}` | +| `event.task.state` | `{taskId, state, error?}` | +| `event.task.progress` | **Batched array**, emitted at ≤4 Hz: `[{taskId, downloaded, speedBps, etaSec, segments:[{i,completed,speedBps}]}]` | +| `event.speed.global` | `{downBps, activeCount}` | +| `event.auth.required` | `{taskId, host, realm, scheme}` | +| `event.notify` | `{level, title, body, taskId?}` | +| `event.settings.changed` | `{keys[]}` | +| `event.grabber.progress` | `{jobId, found, crawled, done}` | + +## Error codes + +| Code | Meaning | +|---|---| +| `-32600/-32601/-32602/-32603` | Standard JSON-RPC | +| `-32001` | Protocol major version mismatch | +| `-32002` | Not paired / invalid token | +| `-32003` | Method not permitted on this transport | +| `-32010` | Task not found | +| `-32011` | Invalid destination path (outside allowed roots, or not writable) | +| `-32012` | Disk full | +| `-32013` | Probe failed (with `data.httpStatus`) | +| `-32014` | Rate limited (pairing brute-force lockout) | diff --git a/contracts/VERSION b/contracts/VERSION new file mode 100644 index 0000000..67dbaea --- /dev/null +++ b/contracts/VERSION @@ -0,0 +1 @@ +1.0.0-draft diff --git a/contracts/fixtures/capture.offer.take.json b/contracts/fixtures/capture.offer.take.json new file mode 100644 index 0000000..ff3b13a --- /dev/null +++ b/contracts/fixtures/capture.offer.take.json @@ -0,0 +1,40 @@ +{ + "name": "capture.offer — attachment on a monitored type is taken", + "description": "Golden fixture. tests/conformance replays this against the real daemon AND the TS client. If either side drifts, this goes red before the lanes ever integrate.", + "request": { + "jsonrpc": "2.0", + "id": 42, + "method": "capture.offer", + "params": { + "url": "https://releases.ubuntu.com/26.04/ubuntu-26.04-desktop-amd64.iso", + "method": "GET", + "tabUrl": "https://releases.ubuntu.com/26.04/", + "headers": { + "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:154.0) Gecko/20100101 Firefox/154.0", + "Referer": "https://releases.ubuntu.com/26.04/", + "Accept": "*/*" + }, + "cookies": [], + "contentType": "application/octet-stream", + "contentLength": 6228541440, + "contentDisposition": "attachment; filename=\"ubuntu-26.04-desktop-amd64.iso\"", + "filename": "ubuntu-26.04-desktop-amd64.iso", + "origin": "moz-extension://11111111-2222-3333-4444-555555555555" + } + }, + "response": { + "jsonrpc": "2.0", + "id": 42, + "result": { + "action": "take", + "taskId": "$uuid", + "reason": null + } + }, + "assertions": [ + "response arrives within 750 ms", + "a task exists afterwards with state in [queued, connecting, downloading]", + "the task's saveDir resolves to the 'Programs' category folder for .iso", + "the Referer and User-Agent from params.headers are replayed on the daemon's own request" + ] +} diff --git a/contracts/schema/methods/capture.offer.schema.json b/contracts/schema/methods/capture.offer.schema.json new file mode 100644 index 0000000..fd9a9ce --- /dev/null +++ b/contracts/schema/methods/capture.offer.schema.json @@ -0,0 +1,62 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://velox.dev/schema/methods/capture.offer.schema.json", + "title": "capture.offer", + "description": "Firefox offers an intercepted response to the daemon. The daemon MUST reply within 750 ms; the extension abandons the offer and lets Firefox download normally on timeout. TEMPLATE — lane PROTO owns the final shape.", + "x-privileged": false, + "x-transports": ["uds", "ws"], + "x-deadlineMs": 750, + "type": "object", + "properties": { + "params": { + "type": "object", + "additionalProperties": false, + "required": ["url", "method", "tabUrl"], + "properties": { + "url": { "type": "string", "format": "uri" }, + "method": { "type": "string", "enum": ["GET", "POST"] }, + "tabUrl": { "type": "string", "format": "uri" }, + "headers": { + "type": "object", + "description": "Request headers Firefox was about to send, verbatim. Needed for signed-URL and referrer-gated CDNs.", + "additionalProperties": { "type": "string" } + }, + "cookies": { + "type": "array", + "description": "Cookies for the URL, so authenticated downloads work outside the browser.", + "items": { + "type": "object", + "required": ["name", "value"], + "properties": { + "name": { "type": "string" }, + "value": { "type": "string" }, + "domain": { "type": "string" }, + "path": { "type": "string" } + } + } + }, + "contentType": { "type": ["string", "null"] }, + "contentLength": { "type": ["integer", "null"], "minimum": 0 }, + "contentDisposition": { "type": ["string", "null"] }, + "filename": { "type": ["string", "null"], "description": "Extension's best guess; the daemon may override" }, + "userAgent": { "type": ["string", "null"] }, + "referrer": { "type": ["string", "null"] }, + "origin": { "type": "string", "description": "moz-extension://… — the daemon verifies this on the WS transport" } + } + }, + "result": { + "type": "object", + "additionalProperties": false, + "required": ["action"], + "properties": { + "action": { "type": "string", "enum": ["take", "ignore"] }, + "taskId": { "type": ["string", "null"], "format": "uuid" }, + "reason": { + "type": ["string", "null"], + "enum": ["excluded_host", "type_not_monitored", "below_min_size", "duplicate", + "capture_disabled", "user_declined", null] + } + } + } + } +} diff --git a/contracts/schema/types/TaskSummary.schema.json b/contracts/schema/types/TaskSummary.schema.json new file mode 100644 index 0000000..9544087 --- /dev/null +++ b/contracts/schema/types/TaskSummary.schema.json @@ -0,0 +1,44 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://velox.dev/schema/types/TaskSummary.schema.json", + "title": "TaskSummary", + "description": "One row of the main download list. Everything the GUI table needs, and nothing more. TEMPLATE — lane PROTO owns the final shape.", + "type": "object", + "additionalProperties": false, + "required": ["taskId", "filename", "state", "createdAt"], + "properties": { + "taskId": { "type": "string", "format": "uuid" }, + "filename": { "type": "string", "maxLength": 255 }, + "saveDir": { "type": "string" }, + "url": { "type": "string", "format": "uri" }, + "effectiveUrl": { "type": "string", "format": "uri" }, + "sizeBytes": { "type": ["integer", "null"], "minimum": 0, "description": "null when the server did not report a length" }, + "downloadedBytes": { "type": "integer", "minimum": 0 }, + "state": { + "type": "string", + "enum": ["new", "probing", "queued", "connecting", "downloading", "paused", + "retry_wait", "assembling", "verifying", "complete", "failed", "cancelled"] + }, + "speedBps": { "type": "integer", "minimum": 0 }, + "etaSeconds": { "type": ["integer", "null"], "minimum": 0 }, + "resumable": { "type": "boolean" }, + "segments": { "type": "integer", "minimum": 1, "maximum": 32 }, + "categoryId": { "type": ["string", "null"] }, + "queueId": { "type": ["string", "null"] }, + "queuePosition":{ "type": ["integer", "null"], "minimum": 0, "description": "the Q column" }, + "description": { "type": "string", "maxLength": 1024 }, + "createdAt": { "type": "string", "format": "date-time" }, + "lastTryAt": { "type": ["string", "null"], "format": "date-time" }, + "completedAt": { "type": ["string", "null"], "format": "date-time" }, + "error": { + "type": ["object", "null"], + "additionalProperties": false, + "properties": { + "code": { "type": "integer" }, + "message": { "type": "string" }, + "httpStatus": { "type": ["integer", "null"] }, + "retryable": { "type": "boolean" } + } + } + } +} diff --git a/core/README.md b/core/README.md new file mode 100644 index 0000000..e1a24f5 --- /dev/null +++ b/core/README.md @@ -0,0 +1,2 @@ +Owner: lane CORE. See ../docs/agents/AGENT-CORE.md and ../docs/04-engine-design.md. +No JSON, no SQL, no Qt, no RPC in this tree. diff --git a/core/include/vdm/io/.gitkeep b/core/include/vdm/io/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/core/include/vdm/media/.gitkeep b/core/include/vdm/media/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/core/include/vdm/meta/.gitkeep b/core/include/vdm/meta/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/core/include/vdm/net/.gitkeep b/core/include/vdm/net/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/core/include/vdm/rate/.gitkeep b/core/include/vdm/rate/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/core/include/vdm/rules/.gitkeep b/core/include/vdm/rules/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/core/include/vdm/segment/.gitkeep b/core/include/vdm/segment/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/core/include/vdm/task/.gitkeep b/core/include/vdm/task/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/core/include/vdm/util/.gitkeep b/core/include/vdm/util/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/core/src/io/.gitkeep b/core/src/io/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/core/src/media/.gitkeep b/core/src/media/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/core/src/meta/.gitkeep b/core/src/meta/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/core/src/net/.gitkeep b/core/src/net/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/core/src/rate/.gitkeep b/core/src/rate/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/core/src/rules/.gitkeep b/core/src/rules/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/core/src/segment/.gitkeep b/core/src/segment/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/core/src/task/.gitkeep b/core/src/task/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/core/src/util/.gitkeep b/core/src/util/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/core/tests/.gitkeep b/core/tests/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/daemon/README.md b/daemon/README.md new file mode 100644 index 0000000..4708eac --- /dev/null +++ b/daemon/README.md @@ -0,0 +1,2 @@ +Owner: lane DAEMON. See ../docs/agents/AGENT-DAEMON.md. +Owns all state: RPC, scheduler, queues, SQLite, pairing. diff --git a/daemon/src/integration/.gitkeep b/daemon/src/integration/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/daemon/src/rpc/.gitkeep b/daemon/src/rpc/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/daemon/src/sched/.gitkeep b/daemon/src/sched/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/daemon/src/store/migrations/.gitkeep b/daemon/src/store/migrations/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/docs/01-architecture.md b/docs/01-architecture.md new file mode 100644 index 0000000..37bbd8a --- /dev/null +++ b/docs/01-architecture.md @@ -0,0 +1,100 @@ +# 01 — Architecture + +## 1. Decision summary + +| Concern | Decision | Why this and not the alternative | +|---|---|---| +| Engine language | **C++23** (`libveloxcore`) | You asked for speed and it is the right call here: segment scheduling, a 64 KiB–8 MiB write path, and 32 concurrent sockets are exactly where a GC and a per-chunk allocation tax show up. Rust would be an equally good engine choice, but Qt's C++ API means C++ removes an entire FFI boundary from the GUI lane. | +| HTTP stack | **libcurl** (multi interface, `curl_multi_poll`) | Writing an HTTP/1.1+2 client on raw epoll costs weeks and buys nothing: curl already does connection reuse, HTTP/2 multiplexing, TLS, proxies, SOCKS5, Digest/NTLM auth, redirect and cookie semantics. One `curl_multi` handle per worker thread, N easy handles = N segments. | +| GUI toolkit | **Qt 6 Widgets** (not QML, not GTK4) | IDM's UI is a dense data grid, a toolbar of big icons, a tree, and ~12 tabbed dialogs. `QTreeView` + a custom `QAbstractItemModel` renders 100k rows without breaking a sweat; QML would need that grid hand-built. GTK4/libadwaita fights you the moment you want a non-GNOME look, and gtkmm's tree/column story is worse. Qt also gives you `QSystemTrayIcon`, drag-and-drop, a global clipboard API, and `QSS` theming to get the IDM look. LGPLv3 dynamic linking is fine for an open-source app. | +| Process model | **Daemon + thin clients** | Downloads must survive closing the window, and the extension must work when no window is open. One owner of state also kills every "GUI and extension disagree" bug class. | +| Wire protocol | **JSON-RPC 2.0** over Unix domain socket (+ loopback WebSocket for the extension fallback) | One protocol for GUI, CLI, and extension = one contract to keep in sync, and it is trivially mockable so three lanes can build in parallel. D-Bus is more idiomatic on Linux but is painful to speak from a WebExtension and adds a second schema. A D-Bus *shim* can be added in M5 for desktop integration only. | +| Persistence | **SQLite** (WAL) | Task list, segment bitmaps, categories, rules, queues, history. Crash-safe, zero admin, one file. | +| Extension | **MV3 WebExtension, TypeScript** | Firefox MV3 still allows blocking `webRequest`, which is what makes true "intercept before the browser downloads it" possible — the thing Chrome MV3 took away. | +| Build | **CMake ≥ 3.28 + Ninja + presets** | Presets mean every agent and CI run the identical configure line. | + +## 2. Processes + +### `veloxd` — the daemon +Owns everything: task list, scheduler, queues, disk, sockets, settings. + +- Single instance, enforced by an abstract-namespace lock socket. +- Started by systemd **user** unit `velox.service`, socket-activated by `velox.socket`. + Also auto-started by the GUI or nmhost if not running (`systemctl --user start velox`). +- Listens on: + - `$XDG_RUNTIME_DIR/velox/velox.sock` (mode 0600) — GUI, CLI, nmhost. Peer credentials + checked via `SO_PEERCRED`; same-UID only. No token needed, the socket permission *is* + the authorization. + - `127.0.0.1:` — loopback + WebSocket for the extension fallback transport. **Token-authenticated + pairing + prompt.** See `docs/05-extension-spec.md` §4. +- Threads: 1 RPC/event loop, 1 curl-multi transfer thread per ~8 active segments + (capped), 1 disk writer thread per active task, 1 timer thread for the scheduler. + Never block the RPC loop on disk or DNS. + +### `velox-gui` — Qt 6 client +Stateless view. On start: connect → `session.hello` → `download.list` → subscribe to +`event.*`. On daemon restart: exponential-backoff reconnect with a banner, never lose the +window. Everything it displays comes from the daemon; it holds no download state of its own. + +### `velox` — CLI +Same RPC, scriptable: `velox add --dir ~/ISOs --segments 16`, `velox ls`, +`velox pause `. Falls out nearly free once the generated client exists, and it is the +fastest way to test the daemon before the GUI is ready. + +### `velox-nmhost` — native-messaging bridge +Deliberately trivial and boring: reads Firefox's 4-byte-length-prefixed JSON from stdin, +writes it to the Unix socket, pumps replies back. **No business logic — ever.** Under 300 +lines. It exists only because Firefox's native messaging speaks stdio and the daemon +speaks sockets. If a feature needs logic, it belongs in the daemon. + +## 3. The layering rule (enforced in review) + +``` +libveloxcore → knows nothing about RPC, JSON, SQL, or Qt. + Input: a DownloadSpec. Output: bytes on disk + callbacks. +veloxd → knows RPC, SQL, scheduling. Depends on core. No Qt. +velox-gui → knows Qt and the generated client. Zero engine code. +extension → knows the browser and the generated TS client. Zero download logic. +``` + +If the GUI ever needs to know what a "segment steal" is, the layering has been violated. +The daemon's job is to project the engine's state into the contract's `TaskDetail` type, +and that is the only shape the GUI ever sees. + +## 4. How three lanes build at the same time without breaking each other + +This is the part that makes parallel agents work, so it is a hard process, not a habit: + +1. **`contracts/` is the single source of truth** and is owned by exactly one lane (PROTO). + JSON Schema for every type, method, and event, plus an OpenRPC document. +2. **Codegen, not hand-written types.** `contracts/codegen/gen_cpp.py` emits + `libveloxproto` (structs + (de)serialization); `gen_ts.py` emits `@velox/protocol`. + Generated files are committed so nobody is blocked on running the generator. Hand-editing + a generated file is a merge-blocking offence. +3. **Golden fixtures.** Every method has request/response JSON examples in + `contracts/fixtures/`. `tests/conformance/` replays them against *both* the C++ server + and the TS client. Green fixtures = the two lanes are compatible, without them ever + having run against each other. +4. **`tools/mockd`** — a TS mock daemon that serves the fixtures and fakes progress events. + The GUI and extension lanes develop against it from day one and never wait for `veloxd`. +5. **Protocol changes are a PR to `contracts/` only**, with a `VERSION` bump and updated + fixtures. Adding an optional field = minor. Removing/renaming/retyping = major, and + `session.hello` refuses a mismatched major with a clear error the GUI shows as + "Update Velox". + +## 5. Data locations + +| What | Path | +|---|---| +| Config | `~/.config/velox/settings.json` | +| Database | `~/.local/share/velox/velox.db` | +| Logs | `~/.local/state/velox/velox.log` (rotated, 5×2 MiB) | +| Runtime sockets | `$XDG_RUNTIME_DIR/velox/` | +| Temp/partial data | configurable; default `~/.local/share/velox/temp/` | +| Default download root | `~/Downloads/` with category subfolders | + +Partial files: the target file is created **sparse and preallocated at final size** in the +destination directory as `.veloxpart`, with `.veloxpart.meta` beside it. On +completion the part file is renamed in place — no copy, no second full-size write, and a +half-finished download is never mistaken for a real file by other apps. diff --git a/docs/02-roadmap.md b/docs/02-roadmap.md new file mode 100644 index 0000000..bb10af5 --- /dev/null +++ b/docs/02-roadmap.md @@ -0,0 +1,141 @@ +# 02 — Roadmap + +Eight milestones. **M0 is the only serialized one** — after the contract freeze, four +lanes run in parallel and only re-synchronize at integration gates. + +Durations are given in "lane-weeks" for a focused agent. Treat them as sequencing weight, +not a delivery promise. + +--- + +## M0 — Foundations & contract freeze *(everything is blocked on this; keep it short)* + +| # | Task | Lane | +|---|---|---| +| 0.1 | Toolchain bootstrap (`apt` list in README), `git init`, branch policy | PKG | +| 0.2 | CMake presets, top-level targets, clang-format/tidy, ASan/UBSan/TSan CI | PKG | +| 0.3 | **Write every schema in `contracts/schema/`** for the v1 method surface | PROTO | +| 0.4 | `gen_cpp.py` + `gen_ts.py`, generated code committed | PROTO | +| 0.5 | Golden fixtures for every method; `tests/conformance/` runner (C++ + TS) | PROTO | +| 0.6 | `tools/mockd` serving fixtures + synthetic progress events | PROTO | +| 0.7 | `tools/testserver` — a *hostile* HTTP server: no-Range, flaky, redirect chains, 401, slow-loris, changing ETag | QA | +| 0.8 | Decide product name + icon set licence; ADR for each M0 decision | PKG | + +**Exit gate:** `tools/mockd` answers every fixture; the generated C++ and TS clients both +round-trip every fixture; CI is green on an empty repo. **Freeze `VERSION` at 1.0.0.** + +--- + +## M1 — Four lanes in parallel *(the long stretch)* + +### 1A · CORE — engine ⏱ 4–5 +Probe → segmentation → dynamic stealing → sparse preallocation → ring-buffered `pwrite` +→ `.veloxpart.meta` resume → token-bucket limiter → retry policy. +**DoD:** downloads a 5 GB file at line rate; kill `-9` at 60 % and resume finishes with a +byte-identical SHA-256; every `tools/testserver` hostile mode handled; ASan/TSan clean; +`tools/bench` numbers recorded as the baseline. + +### 1B · DAEMON — RPC + state ⏱ 3–4 +Unix socket + WS listeners, JSON-RPC dispatcher, subscriptions and event batching, SQLite +schema + migrations, queues, scheduler, settings, pairing/token store, systemd user units. +**DoD:** passes the full conformance suite as a *server*; `velox` CLI can add/list/pause/ +resume; survives daemon restart with all tasks intact. + +### 1C · GUI — Qt shell against `mockd` ⏱ 4–5 +Main window, model/delegate table, category tree, Add-URL + File-Info dialogs, progress +dialog with segment bars and speed graph, Options tabs wired to `settings.*`, tray, drop +target, QSS theming. +**DoD:** every screen in `docs/03-gui-spec.md` exists and is driven **entirely** by +`mockd`; 10 000 synthetic rows scroll at 60 fps; zero download logic in the GUI tree. + +### 1D · EXT — extension against `mockd` ⏱ 3–4 +Capture pipeline, both transports + pairing, context menus, popup with live progress, +options page, media detection. +**DoD:** intercepts a real download in real Firefox and hands it to `mockd`; **fail-open +verified by test** (daemon killed → Firefox downloads normally); `web-ext lint` clean. + +> Lanes touch **only their own directories**. The sole shared surface is `contracts/`, and +> only PROTO writes there. See `CLAUDE.md`. + +--- + +## M2 — First vertical slice ⏱ 1–2 *(all lanes, one week, together)* + +Swap `mockd` for the real `veloxd`. Click a link in Firefox → extension captures → +File-Info dialog appears → download runs multi-segment → progress in GUI *and* popup → +file lands in the right category folder → checksum verified. + +**Exit gate:** that flow works end-to-end on a clean Ubuntu 26.04 VM, from a `.deb`, with +**snap Firefox**. This is the moment the snap native-messaging risk is settled for real. + +--- + +## M3 — IDM parity ⏱ 3–4 *(parallel again)* + +Categories & automatic file distribution · rules engine (extension/MIME/host/size → folder) +· queues + scheduler UI · speed limiter · batch download (wildcards + clipboard blob) · +Site Grabber wizard · Site Logins via Secret Service · proxy/SOCKS5 · duplicate handling · +"Refresh download address" · post-download commands · virus-scan hook. + +**Exit gate:** the IDM feature checklist in this doc's appendix is ticked or has a written +"won't do, because…". + +--- + +## M4 — Media grabber ⏱ 2–3 +HLS/DASH manifest parsing in the daemon, variant enumeration, segment-parallel fetch, mux +via ffmpeg, in-page video panel in the extension. DRM streams detected and clearly refused. + +## M5 — Polish ⏱ 2 +Theming light/dark, i18n + RTL, accessibility pass, notifications, sounds, global shortcut +via the portal, Wayland clipboard spike resolved, first-run wizard, tray/drop-target +behaviour, `--help` and man pages. + +## M6 — Packaging & release ⏱ 1–2 +`.deb` + PPA, Flatpak manifest, native-messaging manifests installed to all four locations, +AMO submission (signed XPI), autostart, upgrade/migration test from a previous DB version, +uninstall leaves no orphan sockets or manifests. + +## M7 — Hardening ⏱ 2 +libFuzzer on every parser (`Content-Disposition`, HLS, DASH, meta file, JSON-RPC), 72-hour +soak with 500 queued tasks, perf targets from `docs/04` §8 enforced in CI, threat-model +review of the WS transport, crash reporting (local only, opt-in, no telemetry). + +--- + +## Dependency graph + +``` +M0 ──┬──► 1A CORE ──┐ + ├──► 1B DAEMON ─┼──► M2 ──► M3 ──► M4 ──► M5 ──► M6 ──► M7 + ├──► 1C GUI ────┤ + └──► 1D EXT ────┘ + (all four against mockd, no cross-lane blocking) +``` + +Critical path: **M0 → 1A/1B → M2**. GUI and EXT can absorb schedule slack because `mockd` +never blocks them. If you must cut scope, cut M4 (media) before anything else — it is the +largest chunk of work with the least effect on core parity. + +--- + +## Appendix — IDM feature checklist + +| IDM feature | Milestone | Notes | +|---|---|---| +| Multi-segment accelerated download | M1 | Dynamic stealing, not static split | +| Resume broken/interrupted downloads | M1 | With `If-Range` revalidation | +| Browser integration / auto-capture | M1D+M2 | Firefox first; Chrome later if wanted | +| Download categories + auto file distribution | M3 | Rules engine | +| Queues + scheduler | M3 | Per-queue time windows | +| Speed limiter | M1/M3 | Engine in M1, UI in M3 | +| Batch downloads / wildcards | M3 | | +| Site Grabber | M3 | Depth-limited crawler with filters | +| Video/media grabber | M4 | HLS/DASH; no DRM | +| Drag-and-drop drop target | M1C | | +| Clipboard monitoring | M5 | Wayland-limited; see risks | +| Proxy / SOCKS5 / site logins | M3 | Credentials in Secret Service | +| Checksum verification | M1 | MD5/SHA-256 | +| On-completion actions (open/shutdown) | M1C/M3 | Shutdown via logind, confirmed | +| ZIP preview | — | **Won't do.** Low value on Linux | +| Dial-up / VPN auto-redial | — | **Won't do.** Obsolete | diff --git a/docs/03-gui-spec.md b/docs/03-gui-spec.md new file mode 100644 index 0000000..8156474 --- /dev/null +++ b/docs/03-gui-spec.md @@ -0,0 +1,152 @@ +# 03 — GUI specification (IDM parity) + +Owner: lane **GUI**. Qt 6 Widgets, C++23. + +> **Legal note, stated once:** replicating IDM's *layout, workflow and feature set* is +> fine — UI ideas aren't protectable. Copying its *icons, artwork, sounds or exact +> logo/branding* is not. Ship an original icon set (a Papirus/Breeze-derived set under a +> compatible licence is the cheap route) laid out in the same positions. That gives users +> the muscle memory without shipping someone else's assets. + +## 1. Main window + +``` +┌─ Velox Download Manager ─────────────────────────────────────────────── ─ □ ✕ ┐ +│ Tasks File Downloads View Help │ +├───────────────────────────────────────────────────────────────────────────────┤ +│ [+] [▶] [⏸] [⏹] [🗑] [🗑✓] [🗓] [⚙] [🌐] │ +│ Add URL Resume Pause Stop All Delete Del.Compl Scheduler Options Grabber│ +├──────────────────┬────────────────────────────────────────────────────────────┤ +│ ▼ All Downloads │ File Name │Q│ Size │ Status │Time Left│ Speed │Date│ +│ Unfinished │ ubuntu.iso │1│ 5.8GB │ 47.2 % │ 00:03:11│ 28MB/s│... │ +│ Finished │ report.pdf │ │ 2.1MB │Complete │ │ │... │ +│ ▼ Categories │ track.flac │2│ 38MB │ Queued │ │ │... │ +│ Compressed │ film.mkv │ │ 1.4GB │ Paused │ │ │... │ +│ Documents │ │ +│ Music │ │ +│ Programs │ │ +│ Video │ │ +│ ▼ Queues │ │ +│ Main Queue │ │ +│ Sync Queue │ │ +├──────────────────┴────────────────────────────────────────────────────────────┤ +│ 4 downloads, 1 active ↓ 28.4 MB/s Limit: off Queue: running ● Connected│ +└───────────────────────────────────────────────────────────────────────────────┘ +``` + +**Columns** (reorderable, resizable, hideable, persisted): File Name · Q · Size · Status · +Time Left · Transfer Rate · Last Try Date · Description. Sort on any column, ascending and +descending, sort state persisted. + +**Implementation:** `QTreeView` + `DownloadTableModel : QAbstractItemModel` fed by +`event.task.progress` deltas. Never rebuild the model on an event — apply a row patch and +emit `dataChanged` for the touched columns only. Progress bar drawn by a `QStyledItemDelegate` +in the Status column. Coalesce progress events on a 250 ms timer; at 20 active downloads +you get 4 repaints/sec, not 400. + +**Row context menu:** Open · Open With · Open Folder · Move/Rename · Redownload · +Refresh Download Address · Resume · Pause · Delete · Add to Queue ▸ · Properties · +Copy Download URL. + +**Category tree:** counts per node, drag a row onto a category to re-file it (moves the +file on disk and updates the DB in one RPC). + +## 2. Download File Info dialog (appears when a URL is added) + +IDM's signature dialog. Populated from `download.probe`. + +``` +File Name: [ ubuntu-26.04-desktop-amd64.iso ] +Save As: [ /home/sami/Downloads/Programs/ ] [ Browse… ] +Category: [ Programs ▾ ] Size: 5.8 GB Resume capability: Yes +Description:[ ] +Connections:[ 8 ▾ ] Buffer: [ 4 MiB ▾ ] ☐ Remember for this file type + + [ Download Now ] [ Download Later ] [ Add to Queue ▾ ] [ Cancel ] +``` + +"Download Later" = `startMode: "later"` (sits in the list as `PAUSED_MANUAL`). +Probe runs async: show the dialog immediately with a spinner in Size/Resume, fill in when +the reply lands, and never block the UI thread. + +## 3. Download progress dialog + +``` +┌ ubuntu-26.04-desktop-amd64.iso ──────────────────────────────── ─ □ ✕ ┐ +│ URL https://releases.ubuntu.com/26.04/ubuntu-26.04-…iso │ +│ Status Receiving data… File size 5.80 GB │ +│ Downloaded 2.74 GB (47.24 %) Transfer rate 28.41 MB/s │ +│ Time left 00:03:11 Resume capability Yes │ +├────────────────────────────────────────────────────────────────────────┤ +│ 1 ████████████░░░░░ Receiving 3.9 MB/s │ 5 ██████████░░░░ 3.2 MB/s │ +│ 2 ██████████░░░░░░░ Receiving 3.4 MB/s │ 6 ███████████░░░ 3.6 MB/s │ +│ 3 █████████████░░░░ Receiving 4.1 MB/s │ 7 ████████░░░░░░ 2.9 MB/s │ +│ 4 ███████████░░░░░░ Receiving 3.7 MB/s │ 8 ████████████░░ 3.5 MB/s │ +├────────────────────────────────────────────────────────────────────────┤ +│ [speed graph, 60 s rolling window, filled area, 1 Hz] │ +├────────────────────────────────────────────────────────────────────────┤ +│ ☐ Close dialog when done On completion: [ Do nothing ▾ ] │ +│ [ Pause ] [ Cancel ] [ Hide ] │ +└────────────────────────────────────────────────────────────────────────┘ +``` + +Per-segment bars come from `TaskDetail.segments[]`. "On completion" offers: Do nothing / +Open file / Open folder / Exit Velox / Shut down (the last one via +`org.freedesktop.login1`, and it must confirm). + +## 4. Options dialog — tabs + +| Tab | Contents | +|---|---| +| **General** | Launch on login · minimize to tray · show floating drop target · confirm on exit · language · check for updates | +| **File Types** | Per-category extension lists (the auto-capture table the extension mirrors) · "Automatically start downloading these types" · MIME overrides | +| **Save To** | Default download directory · per-category folders · temp folder · **file-exists policy** (ask / rename / overwrite / resume) · "create subfolder per site" | +| **Connection** | Connection type preset · **max connections per download (1–32)** · **write buffer per connection** · global max concurrent downloads · timeout · retries · per-host connection overrides | +| **Downloads** | Speed limiter default · virus-scan command · post-download command hook · duplicate-URL policy · integrity check (MD5/SHA-256) | +| **Proxy** | System / manual HTTP / HTTPS / SOCKS5 / PAC · per-host bypass list | +| **Site Logins** | Host → username/password, stored in the **Secret Service** (gnome-keyring), never in SQLite | +| **Sounds** | Per-event sound toggles (download complete, queue complete, error) | + +Everything on this dialog maps 1:1 onto `settings.get`/`settings.set` keys. The settings +key list lives in `contracts/schema/types/Settings.schema.json` — the GUI must not invent +a key that isn't in the schema. + +## 5. Other windows + +- **Scheduler** — per-queue: start time, stop time, days of week, one-time vs periodic, + "hang up/exit when done", max concurrent per queue. +- **Site Grabber wizard** — 4 steps (project template → start URL + depth + filters → + file-type filter → review found files, check what to download). +- **Batch download from clipboard** — parse a pasted blob of URLs, dedupe, assign category. +- **Batch download with wildcards** — `http://host/img{1..50}.jpg` expansion with preview. +- **Speed limiter** — off / limit to N KB/s, with a "apply to running downloads now" button. +- **Floating drop target** — frameless always-on-top `QWidget`, accepts dropped links, + right-click menu, position remembered. IDM's drop box, minus the branding. +- **Tray icon** — active count in tooltip, menu: Show · Add URL · Pause All · Resume All · + Speed limiter ▸ · Quit (Quit asks whether to also stop the daemon). + +## 6. Clipboard capture + +`QClipboard::dataChanged` → if the text is a URL whose extension is in the monitored list, +show a toast: "Download this link? [Download] [Ignore]". + +⚠ **Under Wayland this does not work the way it does on X11** — a Wayland client is not +notified of clipboard changes made by other applications. This is the #2 risk in +`docs/06-risks-and-spikes.md` and has a dedicated spike. The design must therefore treat +clipboard monitoring as *best-effort* and ship these as the real paths: +1. The extension's context menu ("Download with Velox") — covers the browser case, which + is the overwhelming majority. +2. A global shortcut (via `org.freedesktop.portal.GlobalShortcuts`) that reads the + clipboard *on demand* — an explicit user action, which the portal does allow. +3. "Add URL" dialog, pre-filled from the clipboard when it opens (also an explicit action). + +## 7. Theming + +`gui/resources/qss/idm-like.qss` plus a `dark.qss`. Follow the system light/dark preference +via `QStyleHints::colorScheme()`. Keep every colour in one variables block at the top of +the QSS; no hard-coded hex scattered through widget code. + +## 8. Accessibility & i18n (M5, not optional) + +Keyboard-reachable everything, `Qt::AccessibleName` on custom widgets, `tr()` from the +first commit, `.ts` files under `gui/i18n/`, RTL layout verified with Arabic. diff --git a/docs/04-engine-design.md b/docs/04-engine-design.md new file mode 100644 index 0000000..629531f --- /dev/null +++ b/docs/04-engine-design.md @@ -0,0 +1,127 @@ +# 04 — Download engine design (`libveloxcore`) + +Owner: lane **CORE**. This is the part that has to be genuinely fast. + +## 1. Task lifecycle + +``` +NEW → PROBING → QUEUED → CONNECTING → DOWNLOADING ⇄ PAUSED + │ │ + │ └→ RETRY_WAIT → CONNECTING + ▼ + ASSEMBLING → VERIFYING → COMPLETE + │ + └→ FAILED | CANCELLED +``` + +`ASSEMBLING` is normally a no-op rename (see §4). It exists as a real state only for the +HLS/DASH path, which must mux. + +## 2. Probe (`download.probe`) + +Feeds IDM's "Download File Info" dialog, so it must be fast and never partially download. + +1. `HEAD` with the browser's headers/cookies/referrer/UA verbatim. +2. If `HEAD` is refused (405/403 — common), fall back to `GET` with `Range: bytes=0-0` and + abort after headers. +3. Extract: final URL after redirects, `Content-Length`, `Content-Type`, + `Content-Disposition` filename (RFC 5987/6266, UTF-8 and legacy), `Accept-Ranges`, + `ETag`, `Last-Modified`. +4. **Resumability is proven, not assumed:** treat as resumable only if the range request + returned `206` *and* `Content-Range` matches. Servers lie about `Accept-Ranges`. +5. Filename resolution order: explicit user name → `Content-Disposition` → last path + segment (percent-decoded) → `Content-Type` extension → `download.bin`. + Then sanitize: strip `/` `\0`, control chars, trailing dots/spaces, cap at 200 bytes + *of UTF-8* (never split a codepoint), reject `.` and `..`. + +## 3. Segmentation — the part IDM is actually famous for + +Static N-way splitting is not what makes IDM feel fast; **dynamic segment stealing** is. + +``` +Initial: [====seg0====][====seg1====][====seg2====][====seg3====] +seg1 is on a fast mirror and finishes early: + [==seg0==....][ done ][==seg2==....][==seg3==....] + ▲ largest remaining tail +Steal: seg1 restarts on the second half of seg2's remaining range: + [==seg0==....][ done ][=seg2=][==seg1'==][==seg3==....] +``` + +Rules: +- Default 8 segments, user-configurable 1–32, clamped per-host by settings (some hosts + ban >4 connections; keep a per-host override table). +- Never split below `min_segment_bytes` (default 1 MiB) — more segments than that is pure + overhead and gets you rate-limited. +- On finish, a worker steals the **second half of the largest remaining range**, atomically + under the task lock, and only if that half is ≥ `min_segment_bytes`. +- A segment that fails 3× with a connection error is not retried on the same host if a + mirror exists; the range is returned to the pool and re-split. +- Non-resumable servers → exactly 1 segment, and the UI must say so + (IDM's "Resume capability: No"). + +## 4. Disk I/O — the buffer setting you asked for + +One file, opened once, `O_WRONLY`. Each segment `pwrite()`s at its own absolute offset, so +**there is no reassembly pass and no second write of the whole file.** + +- `posix_fallocate()` the full size up front → contiguous extents, no ENOSPC surprise at + 99 %, no fragmentation. +- Per-segment ring buffer, size = **`buffer_bytes`** (the user-visible "Buffer size" + setting). Default 4 MiB, range 64 KiB – 64 MiB. Curl's write callback appends; the + buffer is flushed with a single `pwrite` when full or when the segment ends. + *This is the single biggest throughput knob and it is exposed in the UI: Options → + Downloads → "Write buffer per connection".* +- Global cap `max_total_buffer_bytes` (default 256 MiB) so 32 segments × 64 MiB can't OOM + the box. The per-segment value is silently reduced to fit and the effective value is + reported back to the UI. +- `posix_fadvise(POSIX_FADV_DONTNEED)` on written ranges — do not let a 40 GB ISO evict + the user's entire page cache. +- `fdatasync()` on a timer (default 5 s) and on pause, **not** per write. +- `io_uring` is a **post-1.0 optimization**, gated behind a benchmark in `tools/bench/` + that must show ≥10 % improvement on NVMe. Do not start there. + +## 5. Resume metadata — `.veloxpart.meta` + +Written next to the part file so a download survives a daemon crash, a reboot, *and* a +database loss. Little-endian, versioned, `fdatasync`'d on every segment-boundary update: + +``` +magic "VDMP" | u16 version | u16 flags +u64 total_size | u64 downloaded +url_set (original + effective + mirrors, length-prefixed UTF-8) +etag | last_modified | content_type +u32 segment_count + per segment: u64 start, u64 end, u64 completed +sha256_partial_state (optional, for streaming hash) +crc32 of the whole record +``` + +On resume, revalidate with `If-Range: `. If the server answers +`200` instead of `206`, the file changed underneath us: surface it as +"File on server has changed — restart download?" rather than silently corrupting the file. +This is the single most common way download managers produce broken files. Do not get it wrong. + +## 6. Rate limiting + +Hierarchical token buckets: global → per-queue → per-task. Refill on a 100 ms tick; +throttle by delaying curl reads (`CURLOPT_MAX_RECV_SPEED_LARGE` as a coarse floor, plus +our own pause/unpause via `curl_easy_pause` for precision). "Speed Limiter" in the UI +toggles between Full speed / a saved limit, exactly as IDM does. + +## 7. Failure policy + +| Case | Behaviour | +|---|---| +| 5xx / timeout / reset | Exponential backoff 1 s→2→4→8→…→cap 60 s, jitter ±20 %, `max_retries` default 10 | +| 401/407 | Emit `event.auth.required`, pause task, GUI shows credential dialog, store in secret service (never in SQLite) | +| 403 after redirect | Retry once with the original referrer; many CDNs require it | +| 416 | Metadata is stale → re-probe, re-split | +| Disk full | Pause all, one notification, do not spin | +| Server drops range support mid-download | Demote to 1 segment, keep what's on disk, continue | + +## 8. Performance targets (M7 gate, `tools/bench/`) + +- Saturate a 1 Gbit link with ≤ 8 % of one core. +- ≤ 60 MB RSS with 20 active downloads at default buffers. +- 10 000-row task list: RPC `download.list` under 50 ms, GUI scroll at 60 fps. +- No allocation in the curl write callback hot path (ring buffer is preallocated). diff --git a/docs/05-extension-spec.md b/docs/05-extension-spec.md new file mode 100644 index 0000000..4c58fae --- /dev/null +++ b/docs/05-extension-spec.md @@ -0,0 +1,160 @@ +# 05 — Firefox extension specification + +Owner: lane **EXT**. TypeScript, MV3, `browser.*` promise APIs, built with esbuild, +packaged with `web-ext`. + +## 1. Why Firefox MV3 is actually good news here + +Chrome's MV3 removed blocking `webRequest`, which is why IDM-style "grab the download +before the browser starts it" is hard there. **Firefox kept blocking `webRequest` in +MV3.** That means we can inspect response headers and cancel the browser's own download, +which is exactly the interception model IDM uses. Build for Firefox first and do not +compromise the design to stay Chrome-portable. + +## 2. Capture pipeline + +``` +onBeforeSendHeaders ──► stash request headers by requestId (ring buffer, 5 min TTL) + │ +onHeadersReceived ───► shouldCapture(details, headers, settings)? + │ │ + │ yes│ + │ ▼ + │ cookies.getAll(url) ──► transport.send("capture.offer", {...}) + │ │ + │ daemon replies {action:"take", taskId} + │ ▼ + └──────────────► return {cancel: true} ← browser never starts the download +``` + +`shouldCapture` returns true when **any** of: +- `Content-Disposition: attachment` present, or +- the file extension is in the user's monitored list (Options → File Types, mirrored from + the daemon so the two never disagree), or +- `Content-Type` is in the monitored MIME list and not `text/html`, or +- `Content-Length` > `minSizeBytes` (default 1 MiB) **and** the type is not renderable. + +And **none** of: +- the tab is a `blob:`/`data:` URL we generated, +- the host is on the user's exclusion list, +- the response is a navigation to an HTML page, +- the user held the bypass modifier (default: Alt) on the click. + +**Fail-open, always.** If the daemon is unreachable or the RPC times out (750 ms budget), +`return {}` and let Firefox download it normally. A download manager that eats downloads +when its daemon is down is worse than no download manager. This rule is non-negotiable and +has a dedicated conformance test. + +Belt-and-braces second path: `browser.downloads.onCreated` → if it slipped past the +header hook, `downloads.cancel(id)` + `downloads.erase(id)` and offer to the daemon. Some +downloads (form POSTs, service-worker-generated blobs) only surface here. + +## 3. Other surfaces + +| Surface | Behaviour | +|---|---| +| Context menu (link) | "Download with Velox" | +| Context menu (image/video/audio) | "Download with Velox" | +| Context menu (page/selection) | "Download all links with Velox…" → opens the batch dialog with the harvested list | +| Toolbar popup | Active downloads with live progress (via `event.task.progress` relayed over the transport), pause/resume, "Add URL", speed indicator, daemon status dot | +| Media panel | Floating in-page button on a tab where a video/HLS/DASH stream was detected — "Download this video ▾" listing quality variants | +| Options page | Transport & pairing · monitored types · min size · exclusion list · default category/folder · bypass modifier · enable/disable capture | +| Keyboard | Configurable command to grab the current tab's URL | + +**Media detection:** `webRequest` sniffing for `.m3u8`, `.mpd`, `Content-Type: +application/vnd.apple.mpegurl` / `dash+xml`, plus a content script observing +`MediaSource.addSourceBuffer` and `