Compare commits
89
Commits
_check-main
...
_m
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
72344134cc | ||
|
|
08d7ee9263 | ||
|
|
d93c8e10a0 | ||
|
|
14128c1935 | ||
|
|
91636f8a4d | ||
|
|
efbf366c18 | ||
|
|
afaded85f8 | ||
|
|
b02bdb0466 | ||
|
|
3db0d01f9d | ||
|
|
479f882324 | ||
|
|
7514f5a4c4 | ||
|
|
477a98a31b | ||
|
|
6787784936 | ||
|
|
ab479e7885 | ||
|
|
1914eed7db | ||
|
|
b010139421 | ||
|
|
ddb5db01f5 | ||
|
|
dbdbd6049a | ||
|
|
1d7c7db076 | ||
|
|
b97ed5a5c6 | ||
|
|
7289943375 | ||
|
|
ea8fd7ca8f | ||
|
|
3da4cd6e91 | ||
|
|
207acb0c00 | ||
|
|
dab071c41a | ||
|
|
ec288db5ea | ||
|
|
51fa1201bd | ||
|
|
175185bbfc | ||
|
|
d6cf1fe7dc | ||
|
|
23407974d2 | ||
|
|
1aed222ae2 | ||
|
|
dd1676bc0a | ||
|
|
e585113daf | ||
|
|
4b279e8271 | ||
|
|
0c7ce1437c | ||
|
|
e60d6669d8 | ||
|
|
170bcfdb3e | ||
|
|
768f4f8e53 | ||
|
|
bb138169d5 | ||
|
|
6c94df0441 | ||
|
|
f1fb669209 | ||
|
|
5d81b4cdae | ||
|
|
5e3e21543a | ||
|
|
39867f4f92 | ||
|
|
dad88fce3f | ||
|
|
57b06ea5b1 | ||
|
|
82a2f11d7a | ||
|
|
99b429abfa | ||
|
|
9dce588456 | ||
|
|
1fabaf805d | ||
|
|
e6f070f0dd | ||
|
|
efd5532862 | ||
|
|
05b650b8ec | ||
|
|
8844dd616b | ||
|
|
092e99f7a0 | ||
|
|
801bcaae12 | ||
|
|
c5d596d93b | ||
|
|
90e2580b05 | ||
|
|
2e3251f0b5 | ||
|
|
2959b0f707 | ||
|
|
9025085d82 | ||
|
|
2a87abe96d | ||
|
|
850e85de2a | ||
|
|
bd0a24c87a | ||
|
|
e8a6b64c2f | ||
|
|
bd98fe42f9 | ||
|
|
3588e3b0db | ||
|
|
09b6945bd1 | ||
|
|
42d5fbb8fe | ||
|
|
6d44767373 | ||
|
|
27e9ce9fb5 | ||
|
|
d3ab00d67c | ||
|
|
62cda074c1 | ||
|
|
6db304a0ae | ||
|
|
203d4a662f | ||
|
|
c65e664c29 | ||
|
|
2d36e9fef0 | ||
|
|
f60070c420 | ||
|
|
19d3cd2b1d | ||
|
|
201ebc55d4 | ||
|
|
db7650b2fd | ||
|
|
fdacf732fa | ||
|
|
fb77c9008a | ||
|
|
4f30fa6970 | ||
|
|
bd1bc029f3 | ||
|
|
e442c99130 | ||
|
|
67b7b75336 | ||
|
|
60363a7142 | ||
|
|
ecedfac903 |
@@ -14,11 +14,12 @@ policy so it can be re-applied or audited.
|
||||
|---|---|
|
||||
| `clang-format` | now |
|
||||
| `testserver` | now |
|
||||
| `bootstrap-script` | now |
|
||||
| `build (gcc)` / `build (clang)` | when the first C++ lane merges |
|
||||
| `sanitizers (dev)` / `sanitizers (tsan)` | when the first C++ lane merges |
|
||||
| `conformance` | **when `tests/conformance/` lands — this is the M0 exit gate** |
|
||||
| `extension-lint` | when `extension/` lands |
|
||||
| `bootstrap-script` | now (validates package names against the 24.04 runner archive) |
|
||||
| `bootstrap-script-2604` | now — real `--with-clang` install in a 26.04 container; the release the project ships on |
|
||||
| `build (gcc)` / `build (clang)` | now — core, daemon and gui have merged |
|
||||
| `sanitizers (dev)` / `sanitizers (tsan)` | now — core, daemon and gui have merged |
|
||||
| `conformance` | **now — `tests/conformance/` has landed; this is the M0 exit gate** |
|
||||
| `extension-lint` | now — `extension/` has merged (MV3 manifest + esbuild build) |
|
||||
|
||||
`clang-tidy` is intentionally **not** required through M1 (`continue-on-error: true`,
|
||||
`.clang-tidy` has `WarningsAsErrors: ''`). Make it required at M2.
|
||||
@@ -30,7 +31,16 @@ policy so it can be re-applied or audited.
|
||||
|
||||
## Note on the "skipped" job steps
|
||||
|
||||
Several jobs (`conformance`, `extension-lint`, `clang-tidy`) short-circuit to a "skipped"
|
||||
echo when their lane hasn't landed. They still report **success**, so they can be marked
|
||||
required now without blocking — they start doing real work automatically on the commit
|
||||
that adds the lane.
|
||||
`extension-lint` and `clang-tidy` short-circuit to a "skipped" echo when their lane
|
||||
hasn't landed. They still report **success**, so they can be marked required now without
|
||||
blocking — they start doing real work automatically on the commit that adds the lane.
|
||||
|
||||
Their guards fail **loudly** (non-zero) once the lane is half-present — e.g. an
|
||||
`extension/manifest.json` with no lintable `package.json`. A guard keyed to a single
|
||||
filename is how a required check ends up green over nothing; the skip branch is only for
|
||||
a lane that is genuinely absent.
|
||||
|
||||
`conformance` has no skip branch. It runs `ctest -L conformance` (see
|
||||
`docs/adr/0014-conformance-runs-through-ctest.md`); the `dev` test preset's
|
||||
`noTestsAction: error` fails the job if that label ever matches nothing, so a deleted or
|
||||
renamed registration goes red instead of passing vacuously.
|
||||
|
||||
+82
-26
@@ -41,34 +41,76 @@ jobs:
|
||||
run: python3 tools/testserver/selftest.py
|
||||
|
||||
bootstrap-script:
|
||||
# Keeps tools/bootstrap.sh honest: it must run clean and its --check must pass.
|
||||
# Keeps tools/bootstrap.sh honest on the runner image: it must run clean and its
|
||||
# --check must pass. ubuntu-latest is 24.04; the project ships on 26.04, so this
|
||||
# exercises the 24.04 archive only. bootstrap-script-2604 below is what validates
|
||||
# the package names against the release the project actually targets.
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- run: sudo ./tools/bootstrap.sh --with-clang
|
||||
- run: ./tools/bootstrap.sh --check
|
||||
- run: ./tools/bootstrap.sh --check --with-clang
|
||||
# Cheap: apt-cache only. Validates the M6 packaging names now so they can't rot
|
||||
# unnoticed until M6.
|
||||
- run: ./tools/bootstrap.sh --check --with-clang --packaging
|
||||
|
||||
bootstrap-script-2604:
|
||||
# The project targets 26.04 and GitHub has no 26.04 runner image yet, so the one
|
||||
# automated place bootstrap.sh runs is on the wrong release to catch a name that is
|
||||
# valid on 24.04 and gone on 26.04 — which is exactly how libqt6svg6-dev reached a
|
||||
# contributor's VM (gui/docs/pkg-qa-requests-m1.md R1/R2). Run the real install in a
|
||||
# 26.04 container, with --with-clang: the fuzz toolchain had never been exercised
|
||||
# anywhere (CORE ran clang++-21 directly because it can't sudo).
|
||||
runs-on: ubuntu-latest
|
||||
container: ubuntu:26.04
|
||||
steps:
|
||||
- name: Base tools for checkout
|
||||
run: |
|
||||
apt-get update -qq
|
||||
apt-get install -y --no-install-recommends ca-certificates git sudo
|
||||
- uses: actions/checkout@v4
|
||||
- name: Full bootstrap on 26.04 (--with-clang)
|
||||
run: ./tools/bootstrap.sh --with-clang
|
||||
- name: Re-verify
|
||||
run: ./tools/bootstrap.sh --check --with-clang
|
||||
|
||||
extension-lint:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- id: check
|
||||
# "Has EXT landed?" is answered by a manifest, not by extension/package.json:
|
||||
# a guard keyed to one filename passes vacuously the day EXT ships the lane
|
||||
# under any other name. Skip only when the lane genuinely is not here; once a
|
||||
# manifest exists, a missing lint entrypoint is a hard failure, not a skip.
|
||||
run: |
|
||||
if [ -f extension/package.json ]; then echo "present=true" >> "$GITHUB_OUTPUT"
|
||||
else echo "present=false" >> "$GITHUB_OUTPUT"; fi
|
||||
manifest=""
|
||||
for m in extension/manifest.json extension/src/manifest.json extension/public/manifest.json; do
|
||||
if [ -f "$m" ]; then manifest="$m"; break; fi
|
||||
done
|
||||
if [ -z "$manifest" ]; then
|
||||
echo "extension/ has not landed yet (no manifest.json) — skipping web-ext lint."
|
||||
echo "present=false" >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
echo "EXT has landed: $manifest"
|
||||
echo "present=true" >> "$GITHUB_OUTPUT"
|
||||
if [ ! -f extension/package.json ]; then
|
||||
echo "::error::$manifest exists but extension/package.json does not — this job" \
|
||||
"cannot lint the extension. Wire web-ext lint in here; do not let the check" \
|
||||
"pass green over an unlinted lane."
|
||||
exit 1
|
||||
fi
|
||||
- uses: actions/setup-node@v4
|
||||
if: steps.check.outputs.present == 'true'
|
||||
with:
|
||||
node-version: '20'
|
||||
node-version: '22'
|
||||
- name: web-ext lint
|
||||
if: steps.check.outputs.present == 'true'
|
||||
working-directory: extension
|
||||
run: |
|
||||
npm ci
|
||||
npx web-ext lint --source-dir .
|
||||
- name: skipped
|
||||
if: steps.check.outputs.present == 'false'
|
||||
run: echo "extension/ has not landed yet — skipping web-ext lint"
|
||||
|
||||
# --- build + test matrix ----------------------------------------------------------
|
||||
build:
|
||||
@@ -89,7 +131,10 @@ jobs:
|
||||
- name: Build
|
||||
run: cmake --build --preset ci
|
||||
- name: Test
|
||||
run: ctest --preset ci --output-on-failure
|
||||
# -E '^conformance$' drops the end-to-end run.sh test (npm installs, its own
|
||||
# mockd, ~24 s); the dedicated `conformance` job owns that one run. The native
|
||||
# `conformance_cpp` test is not excluded and still runs on every matrix leg.
|
||||
run: ctest --preset ci --output-on-failure -E '^conformance$'
|
||||
|
||||
sanitizers:
|
||||
runs-on: ubuntu-latest
|
||||
@@ -106,7 +151,10 @@ jobs:
|
||||
- name: Build
|
||||
run: cmake --build --preset ${{ matrix.preset }}
|
||||
- name: Test
|
||||
run: ctest --preset ${{ matrix.preset }} --output-on-failure
|
||||
# See the build job: the end-to-end run.sh test is the dedicated `conformance`
|
||||
# job's; sanitizing a suite that shells out to its own unsanitized g++ build and
|
||||
# a node process buys nothing. `conformance_cpp` still runs here under the sanitizer.
|
||||
run: ctest --preset ${{ matrix.preset }} --output-on-failure -E '^conformance$'
|
||||
env:
|
||||
ASAN_OPTIONS: detect_leaks=1:halt_on_error=1
|
||||
UBSAN_OPTIONS: print_stacktrace=1:halt_on_error=1
|
||||
@@ -143,24 +191,32 @@ jobs:
|
||||
run: echo "no C++ lane has landed a CMakeLists yet — skipping clang-tidy"
|
||||
|
||||
conformance:
|
||||
# Required check on every PR once tests/conformance/ lands (branch protection is
|
||||
# configured in the repo settings, not here — see .github/BRANCH_PROTECTION.md).
|
||||
# The M0 exit gate. Proves the generated C++ daemon surface and the generated TS
|
||||
# extension surface agree with contracts/fixtures without either side having run
|
||||
# against the other. Required on every PR — branch protection is a repo setting,
|
||||
# recorded in .github/BRANCH_PROTECTION.md.
|
||||
#
|
||||
# Canonical entry point is `ctest -L conformance`. tests/conformance/CMakeLists.txt
|
||||
# (owned by PROTO) registers two tests under that label: `conformance`, which shells
|
||||
# out to run.sh end to end, and `conformance_cpp`, the finer-grained native runner.
|
||||
# CI drives it exactly as a developer does — one definition of "the suite passed",
|
||||
# and PROTO's registration is on the exercised path so it cannot rot. See
|
||||
# docs/adr/0014-conformance-runs-through-ctest.md.
|
||||
#
|
||||
# `noTestsAction: error` in the dev test preset is the rot guard: if the label ever
|
||||
# matches nothing (registration deleted, typo), ctest exits non-zero instead of
|
||||
# passing vacuously.
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- id: check
|
||||
run: |
|
||||
if [ -f tests/conformance/CMakeLists.txt ] || [ -f tests/conformance/package.json ]; then
|
||||
echo "present=true" >> "$GITHUB_OUTPUT"
|
||||
else echo "present=false" >> "$GITHUB_OUTPUT"; fi
|
||||
- name: Bootstrap toolchain
|
||||
if: steps.check.outputs.present == 'true'
|
||||
run: sudo ./tools/bootstrap.sh
|
||||
- name: Run conformance suite
|
||||
if: steps.check.outputs.present == 'true'
|
||||
run: |
|
||||
cmake --preset dev
|
||||
ctest --preset dev --output-on-failure -L conformance
|
||||
- name: skipped
|
||||
if: steps.check.outputs.present == 'false'
|
||||
run: echo "tests/conformance/ has not landed yet — skipping"
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22' # apt ships < 20; run.sh's TS replay runner needs >= 20
|
||||
- name: Configure
|
||||
run: cmake --preset dev
|
||||
- name: Build the native conformance runner
|
||||
run: cmake --build --preset dev --target velox_conformance_cpp
|
||||
- name: Run conformance (ctest -L conformance)
|
||||
run: ctest --preset dev -L conformance --output-on-failure
|
||||
|
||||
@@ -37,3 +37,11 @@ massif.out.*
|
||||
*.log
|
||||
*.veloxpart
|
||||
*.veloxpart.meta
|
||||
|
||||
# Fuzzing crash artifacts — libFuzzer writes these to CWD on a find and each holds the
|
||||
# crashing input verbatim. Ignore so they are never committed by accident (CORE caught
|
||||
# two by hand before this rule existed).
|
||||
crash-*
|
||||
oom-*
|
||||
leak-*
|
||||
timeout-*
|
||||
|
||||
@@ -68,3 +68,8 @@ you change observable behaviour, update the doc in `docs/` that describes it in
|
||||
Ask in the PR rather than guessing at the interface. A day of clarification is cheaper than
|
||||
an M2 integration rewrite. And record real decisions as an ADR in `docs/adr/` — the next
|
||||
agent to touch this will have none of your context.
|
||||
|
||||
**ADR numbers:** there is no allocator. Take the next free number in `main`'s
|
||||
`docs/adr/` (gaps from reserved-but-unwritten entries are fine to fill). Lanes draft in
|
||||
parallel, so collisions happen: whoever merges **second** renumbers, updates any
|
||||
cross-references, and keeps going — it is not worth a round trip.
|
||||
|
||||
@@ -97,12 +97,13 @@ Surveyed on this machine 2026-09-09 — **most of it is already installed**:
|
||||
| libcurl4-openssl-dev · libsqlite3-dev · nlohmann-json3-dev · libssl-dev | ✓ |
|
||||
| libavformat-dev · libavcodec-dev · ffmpeg | ✓ |
|
||||
| clang-format · clang-tidy · python3 · pkg-config | ✓ |
|
||||
| python3-jsonschema · python3-referencing (conformance static runner) | ✓ |
|
||||
|
||||
**Only these four are missing:**
|
||||
|
||||
```bash
|
||||
sudo apt update && sudo apt install -y \
|
||||
libqt6svg6-dev \ # GUI: SVG icon rendering
|
||||
qt6-svg-dev \ # GUI: SVG icon rendering
|
||||
libsecret-1-dev \ # DAEMON: Secret Service for site logins
|
||||
nodejs npm \ # EXT + PROTO: extension build, mockd, conformance runner
|
||||
clang # optional: libFuzzer targets in M7
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
# cli/ produces the `velox` binary — the scriptable RPC client. Owned by lane DAEMON.
|
||||
# Built early (AGENT-DAEMON.md build step 8, pulled forward): it is how the daemon is
|
||||
# tested before the GUI exists.
|
||||
#
|
||||
# Links velox::proto for the wire types and error codes. It speaks the same NDJSON Unix
|
||||
# socket veloxd listens on; no daemon code is linked.
|
||||
|
||||
if(NOT TARGET nlohmann_json::nlohmann_json)
|
||||
find_package(nlohmann_json 3.11 REQUIRED)
|
||||
endif()
|
||||
|
||||
add_executable(velox
|
||||
src/main.cpp
|
||||
src/client.cpp
|
||||
)
|
||||
target_include_directories(velox PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/src)
|
||||
target_compile_features(velox PRIVATE cxx_std_23)
|
||||
target_compile_options(velox PRIVATE -Wall -Wextra -Wpedantic -Werror)
|
||||
target_link_libraries(velox PRIVATE velox::proto nlohmann_json::nlohmann_json)
|
||||
|
||||
if(VELOX_BUILD_TESTS AND EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/tests/CMakeLists.txt)
|
||||
add_subdirectory(tests)
|
||||
endif()
|
||||
@@ -0,0 +1,130 @@
|
||||
#include "client.hpp"
|
||||
|
||||
#include <sys/socket.h>
|
||||
#include <sys/un.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#include <cerrno>
|
||||
#include <cstdlib>
|
||||
#include <cstring>
|
||||
|
||||
#include "velox_proto.hpp"
|
||||
|
||||
namespace velox::cli {
|
||||
|
||||
namespace {
|
||||
|
||||
std::string read_error_message(int e) { return std::strerror(e); }
|
||||
|
||||
} // namespace
|
||||
|
||||
std::string default_socket_path() {
|
||||
std::string base;
|
||||
if (const char* xdg = ::getenv("XDG_RUNTIME_DIR"); xdg != nullptr && xdg[0] != '\0') {
|
||||
base = xdg;
|
||||
} else {
|
||||
base = "/run/user/" + std::to_string(::geteuid());
|
||||
}
|
||||
if (!base.empty() && base.back() == '/') base.pop_back();
|
||||
return base + "/velox/velox.sock";
|
||||
}
|
||||
|
||||
Client::~Client() {
|
||||
if (fd_ >= 0) ::close(fd_);
|
||||
}
|
||||
|
||||
std::optional<CallError> Client::connect() {
|
||||
socket_path_ = default_socket_path();
|
||||
if (socket_path_.size() + 1 > sizeof(sockaddr_un::sun_path)) {
|
||||
return CallError{CallError::kConnect, "socket path too long: " + socket_path_, {}};
|
||||
}
|
||||
|
||||
fd_ = ::socket(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0);
|
||||
if (fd_ < 0) return CallError{CallError::kConnect, read_error_message(errno), {}};
|
||||
|
||||
sockaddr_un addr{};
|
||||
addr.sun_family = AF_UNIX;
|
||||
std::memcpy(addr.sun_path, socket_path_.c_str(), socket_path_.size());
|
||||
if (::connect(fd_, reinterpret_cast<sockaddr*>(&addr), sizeof(addr)) != 0) {
|
||||
const int e = errno;
|
||||
::close(fd_);
|
||||
fd_ = -1;
|
||||
return CallError{CallError::kConnect,
|
||||
"cannot reach veloxd at " + socket_path_ + ": " + read_error_message(e),
|
||||
{}};
|
||||
}
|
||||
|
||||
nlohmann::json hello = {
|
||||
{"clientType", "cli"},
|
||||
{"clientName", std::string("velox ") + std::string(velox::proto::kProtocolVersion)},
|
||||
{"protocolVersion", std::string(velox::proto::kProtocolVersion)},
|
||||
};
|
||||
auto r = call("session.hello", hello);
|
||||
if (!r) return r.error();
|
||||
hello_result_ = *r;
|
||||
return std::nullopt;
|
||||
}
|
||||
|
||||
std::expected<nlohmann::json, CallError> Client::call(const std::string& method,
|
||||
const nlohmann::json& params) {
|
||||
const nlohmann::json request = {
|
||||
{"jsonrpc", "2.0"},
|
||||
{"id", next_id_++},
|
||||
{"method", method},
|
||||
{"params", params},
|
||||
};
|
||||
return round_trip(request);
|
||||
}
|
||||
|
||||
std::expected<nlohmann::json, CallError> Client::round_trip(const nlohmann::json& request) {
|
||||
if (fd_ < 0) return std::unexpected(CallError{CallError::kConnect, "not connected", {}});
|
||||
|
||||
std::string out = request.dump();
|
||||
out.push_back('\n');
|
||||
std::size_t off = 0;
|
||||
while (off < out.size()) {
|
||||
const ssize_t n = ::write(fd_, out.data() + off, out.size() - off);
|
||||
if (n > 0) {
|
||||
off += static_cast<std::size_t>(n);
|
||||
continue;
|
||||
}
|
||||
if (n < 0 && errno == EINTR) continue;
|
||||
return std::unexpected(CallError{CallError::kConnect,
|
||||
"write to daemon failed: " + read_error_message(errno), {}});
|
||||
}
|
||||
|
||||
// Read until a newline completes a frame.
|
||||
for (;;) {
|
||||
if (const auto nl = inbuf_.find('\n'); nl != std::string::npos) {
|
||||
const std::string line = inbuf_.substr(0, nl);
|
||||
inbuf_.erase(0, nl + 1);
|
||||
nlohmann::json reply = nlohmann::json::parse(line, nullptr, false);
|
||||
if (reply.is_discarded()) {
|
||||
return std::unexpected(
|
||||
CallError{CallError::kProtocol, "daemon sent a malformed reply", {}});
|
||||
}
|
||||
if (reply.contains("error")) {
|
||||
const auto& e = reply.at("error");
|
||||
return std::unexpected(CallError{e.value("code", 0), e.value("message", ""),
|
||||
e.contains("data") ? e.at("data") : nlohmann::json()});
|
||||
}
|
||||
return reply.contains("result") ? reply.at("result") : nlohmann::json(nullptr);
|
||||
}
|
||||
|
||||
char chunk[8192];
|
||||
const ssize_t n = ::read(fd_, chunk, sizeof(chunk));
|
||||
if (n > 0) {
|
||||
inbuf_.append(chunk, static_cast<std::size_t>(n));
|
||||
continue;
|
||||
}
|
||||
if (n == 0) {
|
||||
return std::unexpected(CallError{CallError::kConnect,
|
||||
"daemon closed the connection", {}});
|
||||
}
|
||||
if (errno == EINTR) continue;
|
||||
return std::unexpected(CallError{CallError::kConnect,
|
||||
"read from daemon failed: " + read_error_message(errno), {}});
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace velox::cli
|
||||
@@ -0,0 +1,55 @@
|
||||
#pragma once
|
||||
|
||||
// A synchronous, blocking RPC client for the Unix socket. The CLI does one request at a
|
||||
// time and waits for the reply, so none of the daemon's async machinery is needed here —
|
||||
// just connect, session.hello, call, read one NDJSON frame back.
|
||||
|
||||
#include <expected>
|
||||
#include <optional>
|
||||
#include <string>
|
||||
|
||||
#include <nlohmann/json.hpp>
|
||||
|
||||
namespace velox::cli {
|
||||
|
||||
struct CallError {
|
||||
int code; // JSON-RPC error code, or a negative transport code below
|
||||
std::string message;
|
||||
nlohmann::json data; // may be null
|
||||
|
||||
static constexpr int kConnect = -1000; // could not reach the daemon
|
||||
static constexpr int kProtocol = -1001; // malformed reply / framing error
|
||||
};
|
||||
|
||||
class Client {
|
||||
public:
|
||||
~Client();
|
||||
|
||||
// Resolve the socket path ($XDG_RUNTIME_DIR/velox/velox.sock), connect, and complete
|
||||
// session.hello with clientType "cli". On failure returns the error and leaves the
|
||||
// client unusable.
|
||||
std::optional<CallError> connect();
|
||||
|
||||
// Send one request and return its result, or the error. `params` is passed through
|
||||
// verbatim as the JSON-RPC params.
|
||||
std::expected<nlohmann::json, CallError> call(const std::string& method,
|
||||
const nlohmann::json& params);
|
||||
|
||||
const std::string& socket_path() const noexcept { return socket_path_; }
|
||||
const nlohmann::json& hello_result() const noexcept { return hello_result_; }
|
||||
|
||||
private:
|
||||
std::expected<nlohmann::json, CallError> round_trip(const nlohmann::json& request);
|
||||
|
||||
int fd_ = -1;
|
||||
int next_id_ = 1;
|
||||
std::string socket_path_;
|
||||
std::string inbuf_;
|
||||
nlohmann::json hello_result_;
|
||||
};
|
||||
|
||||
// $XDG_RUNTIME_DIR/velox/velox.sock, or /run/user/<uid>/velox/velox.sock when the env var
|
||||
// is unset. Mirrors daemon/src/rpc/runtime_dir.cpp; kept in sync by being trivial.
|
||||
std::string default_socket_path();
|
||||
|
||||
} // namespace velox::cli
|
||||
@@ -0,0 +1,179 @@
|
||||
// velox — the command-line client for veloxd.
|
||||
//
|
||||
// velox add <url> [--dir D] [--out NAME] [--segments N] [--json]
|
||||
// velox ls [--json]
|
||||
// velox pause <id>... velox resume <id>...
|
||||
// velox rm <id>... [--delete-file]
|
||||
//
|
||||
// Exit codes: 0 ok, 1 daemon returned an error, 2 usage error, 3 cannot reach the daemon.
|
||||
|
||||
#include <cstdio>
|
||||
#include <cstdlib>
|
||||
#include <string>
|
||||
#include <string_view>
|
||||
#include <vector>
|
||||
|
||||
#include <nlohmann/json.hpp>
|
||||
|
||||
#include "client.hpp"
|
||||
|
||||
namespace {
|
||||
|
||||
using nlohmann::json;
|
||||
using velox::cli::CallError;
|
||||
using velox::cli::Client;
|
||||
|
||||
constexpr int kOk = 0;
|
||||
constexpr int kRpcError = 1;
|
||||
constexpr int kUsage = 2;
|
||||
constexpr int kNoDaemon = 3;
|
||||
|
||||
struct Args {
|
||||
std::vector<std::string> positional;
|
||||
bool json = false;
|
||||
bool delete_file = false;
|
||||
std::string dir;
|
||||
std::string out;
|
||||
long segments = 0;
|
||||
};
|
||||
|
||||
[[noreturn]] void usage(int code) {
|
||||
std::fprintf(code == kOk ? stdout : stderr,
|
||||
"usage: velox <command> [options]\n\n"
|
||||
" add <url> [--dir DIR] [--out NAME] [--segments N]\n"
|
||||
" ls\n"
|
||||
" pause <id>...\n"
|
||||
" resume <id>...\n"
|
||||
" rm <id>... [--delete-file]\n\n"
|
||||
" --json print the raw JSON-RPC result\n");
|
||||
std::exit(code);
|
||||
}
|
||||
|
||||
Args parse_args(int argc, char** argv) {
|
||||
Args a;
|
||||
for (int i = 2; i < argc; ++i) {
|
||||
const std::string_view arg = argv[i];
|
||||
if (arg == "--json") {
|
||||
a.json = true;
|
||||
} else if (arg == "--delete-file") {
|
||||
a.delete_file = true;
|
||||
} else if (arg == "--dir" && i + 1 < argc) {
|
||||
a.dir = argv[++i];
|
||||
} else if (arg == "--out" && i + 1 < argc) {
|
||||
a.out = argv[++i];
|
||||
} else if (arg == "--segments" && i + 1 < argc) {
|
||||
a.segments = std::strtol(argv[++i], nullptr, 10);
|
||||
} else if (arg == "-h" || arg == "--help") {
|
||||
usage(kOk);
|
||||
} else if (!arg.empty() && arg.front() == '-') {
|
||||
std::fprintf(stderr, "velox: unknown option %s\n", argv[i]);
|
||||
usage(kUsage);
|
||||
} else {
|
||||
a.positional.emplace_back(arg);
|
||||
}
|
||||
}
|
||||
return a;
|
||||
}
|
||||
|
||||
int report_error(const CallError& e, bool as_json) {
|
||||
if (as_json) {
|
||||
json j = {{"error", {{"code", e.code}, {"message", e.message}}}};
|
||||
if (!e.data.is_null()) j["error"]["data"] = e.data;
|
||||
std::printf("%s\n", j.dump(2).c_str());
|
||||
} else {
|
||||
std::fprintf(stderr, "velox: %s\n", e.message.c_str());
|
||||
}
|
||||
return e.code == CallError::kConnect ? kNoDaemon : kRpcError;
|
||||
}
|
||||
|
||||
void print_task_table(const json& items) {
|
||||
std::printf("%-38s %-10s %-9s %s\n", "ID", "STATE", "PROGRESS", "NAME");
|
||||
for (const auto& t : items) {
|
||||
const std::string id = t.value("taskId", "");
|
||||
const std::string state = t.value("state", "");
|
||||
const std::string name = t.value("filename", "");
|
||||
const long long total = t.value("sizeBytes", 0LL);
|
||||
const long long done = t.value("downloadedBytes", 0LL);
|
||||
char pct[12] = "-";
|
||||
if (total > 0) std::snprintf(pct, sizeof(pct), "%lld%%", done * 100 / total);
|
||||
std::printf("%-38s %-10s %-9s %s\n", id.c_str(), state.c_str(), pct, name.c_str());
|
||||
}
|
||||
}
|
||||
|
||||
int cmd_ls(Client& c, const Args& a) {
|
||||
auto r = c.call("download.list", json::object());
|
||||
if (!r) return report_error(r.error(), a.json);
|
||||
if (a.json) {
|
||||
std::printf("%s\n", r->dump(2).c_str());
|
||||
return kOk;
|
||||
}
|
||||
const auto& items = r->contains("items") ? r->at("items") : json::array();
|
||||
if (items.empty()) {
|
||||
std::printf("no downloads\n");
|
||||
return kOk;
|
||||
}
|
||||
print_task_table(items);
|
||||
return kOk;
|
||||
}
|
||||
|
||||
int cmd_add(Client& c, const Args& a) {
|
||||
if (a.positional.empty()) {
|
||||
std::fprintf(stderr, "velox add: a URL is required\n");
|
||||
return kUsage;
|
||||
}
|
||||
json params = {{"url", a.positional.front()}};
|
||||
if (!a.dir.empty()) params["saveDir"] = a.dir;
|
||||
if (!a.out.empty()) params["filename"] = a.out;
|
||||
if (a.segments > 0) params["segments"] = a.segments;
|
||||
|
||||
auto r = c.call("download.add", params);
|
||||
if (!r) return report_error(r.error(), a.json);
|
||||
if (a.json) {
|
||||
std::printf("%s\n", r->dump(2).c_str());
|
||||
} else {
|
||||
std::printf("added %s\n", r->value("taskId", "?").c_str());
|
||||
}
|
||||
return kOk;
|
||||
}
|
||||
|
||||
int cmd_bulk(Client& c, const Args& a, const char* method) {
|
||||
if (a.positional.empty()) {
|
||||
std::fprintf(stderr, "velox: at least one task id is required\n");
|
||||
return kUsage;
|
||||
}
|
||||
json params = {{"taskIds", a.positional}};
|
||||
if (std::string_view(method) == "download.remove" && a.delete_file) params["deleteFile"] = true;
|
||||
|
||||
auto r = c.call(method, params);
|
||||
if (!r) return report_error(r.error(), a.json);
|
||||
if (a.json) {
|
||||
std::printf("%s\n", r->dump(2).c_str());
|
||||
} else {
|
||||
std::printf("ok\n");
|
||||
}
|
||||
return kOk;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
int main(int argc, char** argv) {
|
||||
if (argc < 2) usage(kUsage);
|
||||
const std::string command = argv[1];
|
||||
if (command == "-h" || command == "--help") usage(kOk);
|
||||
|
||||
const Args args = parse_args(argc, argv);
|
||||
|
||||
Client client;
|
||||
if (const auto err = client.connect()) {
|
||||
return report_error(*err, args.json);
|
||||
}
|
||||
|
||||
if (command == "ls") return cmd_ls(client, args);
|
||||
if (command == "add") return cmd_add(client, args);
|
||||
if (command == "pause") return cmd_bulk(client, args, "download.pause");
|
||||
if (command == "resume") return cmd_bulk(client, args, "download.resume");
|
||||
if (command == "rm") return cmd_bulk(client, args, "download.remove");
|
||||
|
||||
std::fprintf(stderr, "velox: unknown command '%s'\n", command.c_str());
|
||||
usage(kUsage);
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
# CLI integration test: a real in-process UdsServer on a temp socket, the real Client
|
||||
# against it. Links veloxd_rpc for the server half.
|
||||
|
||||
add_executable(velox_client_test client_test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/../src/client.cpp)
|
||||
target_include_directories(velox_client_test PRIVATE
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../src
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../../daemon/tests
|
||||
)
|
||||
target_compile_features(velox_client_test PRIVATE cxx_std_23)
|
||||
target_compile_options(velox_client_test PRIVATE -Wall -Wextra -Wpedantic -Werror)
|
||||
target_link_libraries(velox_client_test PRIVATE veloxd_rpc velox::proto nlohmann_json::nlohmann_json)
|
||||
add_test(NAME velox.client COMMAND velox_client_test)
|
||||
set_tests_properties(velox.client PROPERTIES TIMEOUT 30)
|
||||
@@ -0,0 +1,130 @@
|
||||
// The CLI's Client against a real in-process daemon RPC server.
|
||||
|
||||
#include <sys/stat.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#include <cstdlib>
|
||||
#include <string>
|
||||
#include <thread>
|
||||
|
||||
#include "check.hpp"
|
||||
#include "client.hpp"
|
||||
#include "rpc/dispatcher.hpp"
|
||||
#include "rpc/event_loop.hpp"
|
||||
#include "rpc/uds_server.hpp"
|
||||
#include "store/migrations.hpp"
|
||||
#include "store/settings.hpp"
|
||||
#include "store/sqlite.hpp"
|
||||
|
||||
namespace rpc = velox::daemon::rpc;
|
||||
|
||||
static std::string g_allowed_root;
|
||||
using velox::cli::CallError;
|
||||
using velox::cli::Client;
|
||||
|
||||
void run() {
|
||||
// connect() with no daemon -> a kConnect error, not a crash.
|
||||
{
|
||||
::setenv("XDG_RUNTIME_DIR", "/tmp/velox-cli-test-nonexistent-xyz", 1);
|
||||
Client c;
|
||||
auto err = c.connect();
|
||||
CHECK(err.has_value());
|
||||
if (err) CHECK_EQ(err->code, CallError::kConnect);
|
||||
}
|
||||
|
||||
// Point XDG_RUNTIME_DIR at a fresh temp dir; the client derives
|
||||
// <XDG_RUNTIME_DIR>/velox/velox.sock and the server binds exactly that.
|
||||
char tmpl[] = "/tmp/velox-cli-test-XXXXXX";
|
||||
const char* xdg = ::mkdtemp(tmpl);
|
||||
CHECK(xdg != nullptr);
|
||||
if (xdg == nullptr) return;
|
||||
::setenv("XDG_RUNTIME_DIR", xdg, 1);
|
||||
const std::string velox_dir = std::string(xdg) + "/velox";
|
||||
::mkdir(velox_dir.c_str(), 0700);
|
||||
const std::string server_sock = velox_dir + "/velox.sock";
|
||||
|
||||
rpc::EventLoop loop;
|
||||
auto db = velox::daemon::store::Db::open(":memory:");
|
||||
CHECK(db.has_value());
|
||||
if (!db) return;
|
||||
CHECK(velox::daemon::store::migrate_to_head(*db).has_value());
|
||||
|
||||
char root_tmpl[] = "/tmp/velox-cli-root-XXXXXX";
|
||||
g_allowed_root = ::mkdtemp(root_tmpl);
|
||||
CHECK(!g_allowed_root.empty());
|
||||
{
|
||||
velox::daemon::store::Settings settings(*db);
|
||||
CHECK(settings.set_raw("saveTo.allowedRoots",
|
||||
"[\"" + g_allowed_root + "\"]").has_value());
|
||||
CHECK(settings.set_raw("saveTo.defaultDir",
|
||||
"\"" + g_allowed_root + "\"").has_value());
|
||||
}
|
||||
rpc::VeloxDispatcher dispatcher(*db);
|
||||
rpc::UdsServer server(loop, dispatcher, server_sock);
|
||||
const auto ec = server.start();
|
||||
CHECK(!ec);
|
||||
if (ec) return;
|
||||
std::thread th([&loop] { loop.run(); });
|
||||
|
||||
{
|
||||
Client c;
|
||||
auto err = c.connect();
|
||||
CHECK(!err.has_value());
|
||||
if (err) {
|
||||
loop.stop();
|
||||
th.join();
|
||||
return;
|
||||
}
|
||||
CHECK_EQ(c.hello_result().value("transport", ""), std::string("uds"));
|
||||
|
||||
auto ls = c.call("download.list", nlohmann::json::object());
|
||||
CHECK(ls.has_value());
|
||||
if (ls) {
|
||||
CHECK_EQ(ls->value("total", -1), 0);
|
||||
CHECK(ls->at("items").is_array());
|
||||
}
|
||||
|
||||
// download.add outside every allowed root -> -32011, original saveDir echoed.
|
||||
auto bad = c.call("download.add",
|
||||
{{"url", "https://example.com/x"}, {"saveDir", "/etc"}});
|
||||
CHECK(!bad.has_value());
|
||||
if (!bad) {
|
||||
CHECK_EQ(bad.error().code, -32011);
|
||||
CHECK_EQ(bad.error().data.value("path", ""), std::string("/etc"));
|
||||
}
|
||||
|
||||
// download.add into an allowed root -> a task id that then shows up in the list.
|
||||
auto ok = c.call(
|
||||
"download.add",
|
||||
{{"url", "https://example.com/movie.mp4"}, {"saveDir", g_allowed_root}});
|
||||
CHECK(ok.has_value());
|
||||
std::string task_id;
|
||||
if (ok) {
|
||||
task_id = ok->value("taskId", "");
|
||||
CHECK(!task_id.empty());
|
||||
CHECK_EQ(ok->value("state", ""), std::string("queued"));
|
||||
}
|
||||
|
||||
auto ls2 = c.call("download.list", nlohmann::json::object());
|
||||
CHECK(ls2.has_value());
|
||||
if (ls2) {
|
||||
CHECK_EQ(ls2->value("total", -1), 1);
|
||||
CHECK_EQ(ls2->at("items").at(0).value("filename", ""), std::string("movie.mp4"));
|
||||
}
|
||||
|
||||
auto detail = c.call("download.get", {{"taskId", task_id}});
|
||||
CHECK(detail.has_value());
|
||||
if (detail) CHECK_EQ(detail->at("summary").value("taskId", ""), task_id);
|
||||
|
||||
auto missing =
|
||||
c.call("download.get", {{"taskId", "00000000-0000-4000-8000-000000000000"}});
|
||||
CHECK(!missing.has_value());
|
||||
if (!missing) CHECK_EQ(missing.error().code, -32010);
|
||||
}
|
||||
|
||||
loop.stop();
|
||||
th.join();
|
||||
::unlink(server_sock.c_str());
|
||||
}
|
||||
|
||||
TEST_MAIN()
|
||||
+20
-7
@@ -3,11 +3,19 @@
|
||||
**This directory is the interface between every lane.** Owner: agent **PROTO**.
|
||||
Nobody else commits here. Everybody else *generates from* here.
|
||||
|
||||
> ## Status: **v1.0.0 — FROZEN** (2026-09-09)
|
||||
> ## Status: **v1.4.0** (frozen at v1.0.0 on 2026-09-09; minor bumps since)
|
||||
>
|
||||
> The surface below is complete and generated from: 38 methods, 9 events, 26 named types,
|
||||
> 59 fixtures. See `docs/adr/0005-protocol-1.0.0-freeze.md` for the versioning rule and
|
||||
> `docs/adr/0010-...` for the failure taxonomy and the segment range convention.
|
||||
> **v1.0.0** froze 38 methods, 9 events, 26 named types. **v1.1.0** widened `bufferBytes`
|
||||
> bounds and added the segment-budget settings. **v1.2.0** added `download.provideAuth`
|
||||
> (F2). **v1.3.0** widened when `error` is populated on a state change to cover a
|
||||
> daemon-initiated `paused` (for `docs/adr/0013-...`). **v1.4.0** (current) is a
|
||||
> **C++-binding-only** change: the generated `Dispatcher` gains a `HandlerError` /
|
||||
> `HandlerResult<T>` error channel so a handler can return `-32010` / `-32011` /
|
||||
> `-32013` with their `data` payloads instead of collapsing to `-32603`. The wire is
|
||||
> byte-identical — no schema or fixture change — but any `Dispatcher` implementer
|
||||
> must swap `Result` → `HandlerResult` on regen. Answered in
|
||||
> `contracts/proto-answers-daemon-m1.md`. See also `docs/adr/0005-...` for the
|
||||
> versioning rule and `docs/adr/0010-...` for the failure taxonomy and segment ranges.
|
||||
>
|
||||
> Lane requests are answered in writing: `contracts/proto-answers-m1.md` responds to
|
||||
> `core/docs/proto-requests-m1.md` point by point.
|
||||
@@ -27,7 +35,7 @@ Nobody else commits here. Everybody else *generates from* here.
|
||||
|
||||
```
|
||||
contracts/
|
||||
├── VERSION # protocol semver — frozen at 1.0.0
|
||||
├── VERSION # protocol semver — v1.4.0, minor-bumped from the v1.0.0 freeze
|
||||
├── openrpc.json # human-readable API doc (generated from schema/)
|
||||
├── schema/
|
||||
│ ├── envelope.schema.json # JSON-RPC 2.0 envelope + our error codes
|
||||
@@ -56,6 +64,10 @@ subset, `fixtures/` documents the fixture shape and the placeholder rules.
|
||||
renaming, retyping, or changing a default → **major** bump and a written migration note
|
||||
in `docs/adr/`. `session.hello` rejects a major mismatch with error `-32001` and a
|
||||
message the GUI renders as "Velox needs updating".
|
||||
"Retype → major" is about the **wire** — a field a client parses off the socket. A
|
||||
change that leaves the wire byte-identical but breaks a *generated binding's* source
|
||||
API (a C++ virtual's return type, a struct name) is a **minor** bump plus a migration
|
||||
note — see `docs/adr/0015-generated-binding-changes-and-versioning.md`.
|
||||
5. **Changes arrive as a PR to `contracts/` alone**, containing: schema edit + fixtures +
|
||||
regenerated code + `VERSION` bump. Lanes rebase onto it. This is the only synchronization
|
||||
point in the whole project — keep it cheap and frequent rather than big and rare.
|
||||
@@ -73,7 +85,7 @@ on rather than as prose a reader has to honour:
|
||||
| `x-errors` | the error codes this method is documented to return |
|
||||
| `x-wsRestrictions` | extra limits when the call arrives from the extension |
|
||||
|
||||
19 of the 38 methods are privileged: everything that reconfigures the daemon, destroys user
|
||||
20 of the 39 methods are privileged: everything that reconfigures the daemon, destroys user
|
||||
data, or names an arbitrary destination path. The extension may *request* a download; it
|
||||
may not choose where the bytes land.
|
||||
|
||||
@@ -90,7 +102,7 @@ All four carry **the same JSON-RPC 2.0 payloads**. The framing differences stop
|
||||
transport layer; no method behaves differently depending on how it arrived — except that
|
||||
methods marked `"privileged": true` in the schema are refused over the WebSocket transport.
|
||||
|
||||
## Method surface (v1.0.0 target — expand only via PR)
|
||||
## Method surface (v1.4.0 — expand only via PR)
|
||||
|
||||
### Session
|
||||
| Method | Params → Result |
|
||||
@@ -111,6 +123,7 @@ methods marked `"privileged": true` in the schema are refused over the WebSocket
|
||||
| `download.remove` | `{taskIds[], deleteFile:bool}` → `{removed[]}` |
|
||||
| `download.update` | `{taskId, patch:{filename?, saveDir?, categoryId?, queueId?, description?, segments?, bufferBytes?}}` → `TaskSummary` |
|
||||
| `download.refreshUrl` | `{taskId, url, headers?}` → `{ok}` *(IDM's "Refresh Download Address")* |
|
||||
| `download.provideAuth` | `{taskId, username, password, save?}` → `{ok}` — answers `event.auth.required`. UDS only; privileged. Credentials go to the Secret Service, never SQLite, never logs |
|
||||
|
||||
### Organisation
|
||||
`category.list` · `category.upsert` · `category.remove` · `queue.list` · `queue.upsert` ·
|
||||
|
||||
+1
-1
@@ -1 +1 @@
|
||||
1.0.0
|
||||
1.4.0
|
||||
|
||||
@@ -7,6 +7,13 @@ Design notes that matter to the CORE and DAEMON lanes:
|
||||
not used and not emitted. Parsing goes through `velox::proto::parse<T>(json)` which
|
||||
returns `std::expected<T, ParseError>`, so a malformed frame from the wire is an ordinary
|
||||
value the RPC loop handles, not a throw unwinding through the transfer path.
|
||||
* **Two error channels, kept separate.** `parse<T>() -> Result<T>` (i.e. `expected<T,
|
||||
ParseError>`) is the wire failing to become typed params — always `-32602`, always
|
||||
structural. A `Dispatcher::on_*` handler returns `HandlerResult<T>` (i.e. `expected<T,
|
||||
HandlerError>`), which carries any contract error code plus a free-form `data` object,
|
||||
so a handler can answer `-32010` `{taskId}`, `-32011` `{path}`, `-32013` `{httpStatus}`
|
||||
and so on. `dispatch()` forwards the handler's code/message/data straight into the
|
||||
JSON-RPC error object.
|
||||
* **Serialisation is ADL `to_json`,** so `nlohmann::json j = task;` works as expected.
|
||||
Only the outbound direction is allowed to be implicit; the wire is never trusted.
|
||||
* **`libveloxproto`, not `libveloxcore`.** This code includes nlohmann/json, which
|
||||
@@ -285,16 +292,37 @@ def emit_header(c: Contract) -> str:
|
||||
"nlohmann::json make_result(const nlohmann::json& id, nlohmann::json result);",
|
||||
"nlohmann::json make_notification(Event e, nlohmann::json params);",
|
||||
"",
|
||||
"/// A handler's own failure — as opposed to ParseError, which is the wire failing",
|
||||
"/// to become typed params. Carries any contract error code, a message, and a",
|
||||
"/// free-form `data` object that goes straight into the JSON-RPC error's `data`",
|
||||
"/// field: `{\"taskId\": ...}` for TaskNotFound, `{\"path\": ...}` for InvalidPath,",
|
||||
"/// `{\"httpStatus\": ...}` for ProbeFailed. `code` defaults to InternalError so a",
|
||||
"/// handler that sets only a message still produces a valid error response.",
|
||||
"///",
|
||||
"/// -32001/-32002/-32003 are the server layer's to raise around dispatch(), not a",
|
||||
"/// handler's: they are decided before or without reference to method params.",
|
||||
"struct HandlerError {",
|
||||
" ErrorCode code{ErrorCode::InternalError};",
|
||||
" std::string message;",
|
||||
" // `= nullptr`, not `{nullptr}`: brace-init of nlohmann::json from nullptr",
|
||||
" // yields the array [null], not JSON null. make_error() drops a null data.",
|
||||
" nlohmann::json data = nullptr;",
|
||||
"};",
|
||||
"",
|
||||
"template <class T>",
|
||||
"using HandlerResult = std::expected<T, HandlerError>;",
|
||||
"",
|
||||
"/// One virtual per method. The daemon implements this; `dispatch` below does the",
|
||||
"/// envelope handling, the transport check and the parameter parsing, so a handler",
|
||||
"/// only ever sees a validated, typed params struct.",
|
||||
"/// only ever sees a validated, typed params struct. Return `std::unexpected(",
|
||||
"/// HandlerError{...})` to answer with a specific error code and data.",
|
||||
"class Dispatcher {",
|
||||
"public:",
|
||||
" virtual ~Dispatcher() = default;",
|
||||
""]
|
||||
for m in c.methods:
|
||||
o += doc_comment(m.doc, " ")
|
||||
o.append(f" virtual Result<{cpp_type(m.result)}> {handler_name(m.name)}(const {cpp_type(m.params)}& params) = 0;")
|
||||
o.append(f" virtual HandlerResult<{cpp_type(m.result)}> {handler_name(m.name)}(const {cpp_type(m.params)}& params) = 0;")
|
||||
o.append("")
|
||||
o += ["};", "",
|
||||
"/// Parse one JSON-RPC request, route it, and return the response to write back.",
|
||||
@@ -579,8 +607,7 @@ def emit_dispatch(c: Contract) -> list[str]:
|
||||
' nlohmann::json{{"path", p.error().path}});',
|
||||
f" auto r = handler.{handler_name(m.name)}(*p);",
|
||||
" if (!r)",
|
||||
" return make_error(id, ErrorCode::InternalError, r.error().message,",
|
||||
' nlohmann::json{{"path", r.error().path}});',
|
||||
" return make_error(id, r.error().code, r.error().message, r.error().data);",
|
||||
" nlohmann::json out = *r;",
|
||||
" return make_result(id, std::move(out));",
|
||||
" }",
|
||||
|
||||
@@ -44,7 +44,7 @@ def main() -> int:
|
||||
# velox::conformance, so the names need qualifying.
|
||||
pt, rt = "proto::" + cpp_type(m.params), "proto::" + cpp_type(m.result)
|
||||
o += [
|
||||
f" proto::Result<{rt}> {handler_name(m.name)}(const {pt}& params) override {{",
|
||||
f" proto::HandlerResult<{rt}> {handler_name(m.name)}(const {pt}& params) override {{",
|
||||
" (void)params;",
|
||||
f' return golden<{rt}>("{m.name}");',
|
||||
" }",
|
||||
@@ -53,12 +53,21 @@ def main() -> int:
|
||||
|
||||
o += [
|
||||
"private:",
|
||||
" // Every fixture-backed handler only ever succeeds. A missing or unparseable",
|
||||
" // fixture is a bug in the suite, not a contract outcome, so it surfaces as",
|
||||
" // InternalError rather than being dressed up as a real error code.",
|
||||
" template <class T>",
|
||||
" proto::Result<T> golden(const std::string& method) {",
|
||||
" proto::HandlerResult<T> golden(const std::string& method) {",
|
||||
" const nlohmann::json* value = results_(method);",
|
||||
" if (value == nullptr)",
|
||||
' return std::unexpected(proto::ParseError{method, "no fixture for this method"});',
|
||||
" return proto::parse<T>(*value, method);",
|
||||
" return std::unexpected(proto::HandlerError{",
|
||||
' proto::ErrorCode::InternalError, "no fixture for method " + method});',
|
||||
" auto parsed = proto::parse<T>(*value, method);",
|
||||
" if (!parsed)",
|
||||
" return std::unexpected(proto::HandlerError{",
|
||||
" proto::ErrorCode::InternalError,",
|
||||
' "fixture for " + method + " failed to parse: " + parsed.error().message});',
|
||||
" return std::move(*parsed);",
|
||||
" }",
|
||||
"",
|
||||
" std::function<const nlohmann::json*(const std::string&)> results_;",
|
||||
|
||||
@@ -116,7 +116,8 @@
|
||||
"referrer": "https://releases.ubuntu.com/26.04/",
|
||||
"userAgent": "Velox/0.1",
|
||||
"mime": "application/octet-stream",
|
||||
"bufferBytes": 4194304,
|
||||
"bufferBytes": 16777216,
|
||||
"effectiveBufferBytes": 4194304,
|
||||
"partPath": "/home/sami/Downloads/Programs/ubuntu-26.04-desktop-amd64.iso.veloxpart",
|
||||
"checksum": null,
|
||||
"checksumVerified": null,
|
||||
@@ -128,6 +129,7 @@
|
||||
"segment ranges are contiguous and cover exactly [0, sizeBytes) with no gaps or overlaps",
|
||||
"startByte and endByte are both INCLUSIVE: segment 0 here covers 778567680 bytes, 0 through 778567679, and is copied verbatim into 'Range: bytes=0-778567679'",
|
||||
"segmentDetail has exactly summary.segments entries",
|
||||
"the GUI draws one bar per entry and is never told what a segment steal is"
|
||||
"the GUI draws one bar per entry and is never told what a segment steal is",
|
||||
"bufferBytes is what was requested (16 MiB); effectiveBufferBytes (4 MiB) is what this segment is actually using right now, after connection.maxTotalBufferBytes (128 MiB default) is divided across every live segment in the daemon -- not just this task's -- up to connection.maxActiveSegments (32 default). The clamp is global: a task can be reduced even when its own segment count alone would not force it."
|
||||
]
|
||||
}
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"name": "download.provideAuth \u2014 answer a 401 challenge and remember it",
|
||||
"description": "The task was sitting in retry_wait after event.auth.required. This does not restart the transfer itself: the daemon retries with the credential attached and the task's state.transition to connecting/downloading happens on its own, reported the normal way through event.task.state.",
|
||||
"transport": "uds",
|
||||
"request": {
|
||||
"jsonrpc": "2.0",
|
||||
"id": 80,
|
||||
"method": "download.provideAuth",
|
||||
"params": {
|
||||
"taskId": "$taskId",
|
||||
"username": "svc-releases",
|
||||
"password": "hunter2-not-a-real-password",
|
||||
"save": true
|
||||
}
|
||||
},
|
||||
"response": {
|
||||
"jsonrpc": "2.0",
|
||||
"id": 80,
|
||||
"result": {
|
||||
"ok": true
|
||||
}
|
||||
},
|
||||
"assertions": [
|
||||
"the password never appears in a log line, ever, on either side of this call",
|
||||
"save true stores the credential in the Secret Service keyed by host and realm, not in SQLite",
|
||||
"the task itself is not touched synchronously by this call \u2014 it moves out of retry_wait when the daemon's own retry succeeds, reported via event.task.state",
|
||||
"this method is refused with -32003 over the WebSocket transport"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"name": "download.provideAuth \u2014 the task no longer exists",
|
||||
"description": "The ordinary stale-client case: the user typed credentials into a dialog for a task that was removed in the meantime.",
|
||||
"transport": "uds",
|
||||
"request": {
|
||||
"jsonrpc": "2.0",
|
||||
"id": 81,
|
||||
"method": "download.provideAuth",
|
||||
"params": {
|
||||
"taskId": "00000000-0000-4000-8000-000000000000",
|
||||
"username": "x",
|
||||
"password": "y"
|
||||
}
|
||||
},
|
||||
"response": {
|
||||
"jsonrpc": "2.0",
|
||||
"id": 81,
|
||||
"error": {
|
||||
"code": -32010,
|
||||
"message": "no such task",
|
||||
"data": {
|
||||
"taskId": "00000000-0000-4000-8000-000000000000"
|
||||
}
|
||||
}
|
||||
},
|
||||
"assertions": [
|
||||
"credentials submitted for a task that no longer exists are discarded, never persisted anywhere"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"name": "download.provideAuth \u2014 refused over the WebSocket transport",
|
||||
"description": "The other half of event.auth.required's own promise: a credential-bearing method must never be reachable from the browser.",
|
||||
"transport": "ws",
|
||||
"request": {
|
||||
"jsonrpc": "2.0",
|
||||
"id": 82,
|
||||
"method": "download.provideAuth",
|
||||
"params": {
|
||||
"taskId": "$taskId",
|
||||
"username": "x",
|
||||
"password": "y"
|
||||
}
|
||||
},
|
||||
"response": {
|
||||
"jsonrpc": "2.0",
|
||||
"id": 82,
|
||||
"error": {
|
||||
"code": -32003,
|
||||
"message": "method is not permitted on this transport"
|
||||
}
|
||||
},
|
||||
"assertions": [
|
||||
"the extension has no path to this method under any circumstance"
|
||||
]
|
||||
}
|
||||
@@ -18,7 +18,7 @@
|
||||
"code": -32001,
|
||||
"message": "protocol major version mismatch: daemon speaks 1.x, client speaks 2.x",
|
||||
"data": {
|
||||
"expected": "1.0.0",
|
||||
"expected": "$any",
|
||||
"actual": "2.0.0"
|
||||
}
|
||||
}
|
||||
@@ -27,7 +27,8 @@
|
||||
"the connection is closed after this reply; no method is served on a mismatched major",
|
||||
"a differing minor or patch is accepted, never refused",
|
||||
"the message is safe to show a user verbatim",
|
||||
"the version check is transport-independent; this is replayed on the Unix socket so it is not masked by -32002"
|
||||
"the version check is transport-independent; this is replayed on the Unix socket so it is not masked by -32002",
|
||||
"data.expected is the daemon's own current protocol version string (kProtocolVersion), not a bare major and not pinnable in a golden file -- the conformance compare on error payloads is on `code` only, structural elsewhere, so echoing the live version is fine"
|
||||
],
|
||||
"transport": "uds"
|
||||
}
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
{
|
||||
"name": "event.task.state \u2014 the daemon auto-pauses on a 401",
|
||||
"description": "The CORE-auto-pause case ADR 0013 (docs/adr/) needs a wire signal for: the daemon paused this task on its own initiative -- not because the user clicked pause, a schedule window closed, or admission control reconciled a lowered cap -- and error explains why. Compare download.pause.json, where the same target state (paused) carries error: null because that pause was requested.",
|
||||
"notification": {
|
||||
"jsonrpc": "2.0",
|
||||
"method": "event.task.state",
|
||||
"params": {
|
||||
"taskId": "8c1d4e5f-6a7b-4c8d-9e0f-1a2b3c4d5e6f",
|
||||
"state": "paused",
|
||||
"previousState": "connecting",
|
||||
"summary": {
|
||||
"taskId": "8c1d4e5f-6a7b-4c8d-9e0f-1a2b3c4d5e6f",
|
||||
"filename": "film.mkv",
|
||||
"saveDir": "/home/sami/Downloads/Video",
|
||||
"url": "https://example.org/film.mkv",
|
||||
"effectiveUrl": "https://example.org/film.mkv",
|
||||
"sizeBytes": 1503238553,
|
||||
"downloadedBytes": 0,
|
||||
"state": "paused",
|
||||
"speedBps": 0,
|
||||
"etaSeconds": null,
|
||||
"resumable": true,
|
||||
"segments": 4,
|
||||
"categoryId": "video",
|
||||
"queueId": null,
|
||||
"queuePosition": null,
|
||||
"description": null,
|
||||
"createdAt": "$isoDate",
|
||||
"lastTryAt": "$isoDate",
|
||||
"completedAt": null,
|
||||
"error": {
|
||||
"code": "auth_required",
|
||||
"message": "the server asked for credentials (401)",
|
||||
"httpStatus": 401,
|
||||
"retryable": false,
|
||||
"cause": null,
|
||||
"attempt": 1,
|
||||
"nextRetryAt": null
|
||||
}
|
||||
},
|
||||
"error": {
|
||||
"code": "auth_required",
|
||||
"message": "the server asked for credentials (401)",
|
||||
"httpStatus": 401,
|
||||
"retryable": false,
|
||||
"cause": null,
|
||||
"attempt": 1,
|
||||
"nextRetryAt": null
|
||||
}
|
||||
}
|
||||
},
|
||||
"assertions": [
|
||||
"error is set here specifically because the daemon paused this task itself, not the user -- the trigger is event.auth.required on the same task shortly before",
|
||||
"the scheduler must not resume this task on a schedule window or queue restart: only download.provideAuth (or the user explicitly resuming) may clear it -- resuming blindly re-fails immediately and looks like a flapping bug",
|
||||
"a client distinguishes an auto-pause from a deliberate one by this field being non-null, not by inspecting previousState or any other heuristic"
|
||||
]
|
||||
}
|
||||
@@ -50,7 +50,7 @@
|
||||
}
|
||||
},
|
||||
"assertions": [
|
||||
"error is present exactly when state is failed or retry_wait",
|
||||
"error is present on every failed or retry_wait transition, and also on a paused transition the daemon entered unilaterally -- never on a paused transition the user or scheduler requested",
|
||||
"error.code is a TaskErrorCode, never a JSON-RPC ErrorCode \u2014 the two are different spaces",
|
||||
"retryable false means the scheduler will not pick this up again on its own"
|
||||
]
|
||||
|
||||
@@ -9,6 +9,8 @@
|
||||
"keys": [
|
||||
"connection.maxSegmentsPerDownload",
|
||||
"connection.bufferBytes",
|
||||
"connection.maxTotalBufferBytes",
|
||||
"connection.maxActiveSegments",
|
||||
"connection.maxConcurrentDownloads",
|
||||
"connection.timeoutSec"
|
||||
]
|
||||
@@ -20,7 +22,9 @@
|
||||
"result": {
|
||||
"values": {
|
||||
"connection.maxSegmentsPerDownload": 8,
|
||||
"connection.bufferBytes": 4194304,
|
||||
"connection.bufferBytes": 1048576,
|
||||
"connection.maxTotalBufferBytes": 134217728,
|
||||
"connection.maxActiveSegments": 32,
|
||||
"connection.maxConcurrentDownloads": 5,
|
||||
"connection.timeoutSec": 30
|
||||
}
|
||||
@@ -29,6 +33,8 @@
|
||||
"assertions": [
|
||||
"only the requested keys come back",
|
||||
"keys null returns everything",
|
||||
"no password is ever present: credentials live in the Secret Service"
|
||||
"no password is ever present: credentials live in the Secret Service",
|
||||
"connection.bufferBytes defaults to 1 MiB (1048576), not the old 4 MiB",
|
||||
"connection.maxTotalBufferBytes and connection.maxActiveSegments are the two knobs behind TaskDetail.effectiveBufferBytes; Options cannot show or set the clamp without them"
|
||||
]
|
||||
}
|
||||
|
||||
@@ -8,7 +8,8 @@
|
||||
"params": {
|
||||
"values": {
|
||||
"connection.maxSegmentsPerDownload": 16,
|
||||
"downloads.verifyChecksums": true
|
||||
"downloads.verifyChecksums": true,
|
||||
"connection.bufferBytes": 2097152
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -18,17 +19,20 @@
|
||||
"result": {
|
||||
"values": {
|
||||
"connection.maxSegmentsPerDownload": 16,
|
||||
"downloads.verifyChecksums": true
|
||||
"downloads.verifyChecksums": true,
|
||||
"connection.bufferBytes": 2097152
|
||||
},
|
||||
"changed": [
|
||||
"connection.maxSegmentsPerDownload",
|
||||
"downloads.verifyChecksums"
|
||||
"downloads.verifyChecksums",
|
||||
"connection.bufferBytes"
|
||||
]
|
||||
}
|
||||
},
|
||||
"assertions": [
|
||||
"event.settings.changed is emitted carrying exactly the keys in changed[]",
|
||||
"an unknown key is -32602 and nothing at all is written",
|
||||
"a directory key naming an unwritable path is -32011"
|
||||
"a directory key naming an unwritable path is -32011",
|
||||
"connection.bufferBytes accepts 64 KiB - 16 MiB; a value outside that range is -32602"
|
||||
]
|
||||
}
|
||||
|
||||
+123
-19
@@ -2,7 +2,7 @@
|
||||
"openrpc": "1.2.6",
|
||||
"info": {
|
||||
"title": "Velox Download Manager",
|
||||
"version": "1.0.0",
|
||||
"version": "1.4.0",
|
||||
"description": "The wire contract between veloxd and every client: the Qt GUI, the CLI, the native-messaging host and the Firefox extension. One JSON-RPC 2.0 payload set over four framings; only the framing differs.\n\nGENERATED from contracts/schema/ by contracts/codegen/gen_openrpc.py. Do not edit by hand.",
|
||||
"license": {
|
||||
"name": "See repository LICENSE"
|
||||
@@ -487,9 +487,9 @@
|
||||
],
|
||||
"minimum": 1,
|
||||
"maximum": 32,
|
||||
"description": "The REQUESTED connection count. An upper bound, not a promise: the daemon lowers it to the per-host cap, and to 1 when the source turns out not to be resumable. What is actually in use comes back as TaskSummary.segments. null means use connection.maxSegmentsPerDownload."
|
||||
"description": "The REQUESTED connection count. An upper bound, not a promise: the engine lowers it to the per-host cap, and to 1 when the source turns out not to be resumable. What is actually in use comes back as TaskSummary.segments. null means use connection.maxSegmentsPerDownload."
|
||||
},
|
||||
"description": "The REQUESTED connection count. An upper bound, not a promise: the daemon lowers it to the per-host cap, and to 1 when the source turns out not to be resumable. What is actually in use comes back as TaskSummary.segments. null means use connection.maxSegmentsPerDownload."
|
||||
"description": "The REQUESTED connection count. An upper bound, not a promise: the engine lowers it to the per-host cap, and to 1 when the source turns out not to be resumable. What is actually in use comes back as TaskSummary.segments. null means use connection.maxSegmentsPerDownload."
|
||||
},
|
||||
{
|
||||
"name": "bufferBytes",
|
||||
@@ -498,9 +498,11 @@
|
||||
"integer",
|
||||
"null"
|
||||
],
|
||||
"minimum": 4096,
|
||||
"maximum": 8388608
|
||||
}
|
||||
"minimum": 65536,
|
||||
"maximum": 16777216,
|
||||
"description": "Requested write buffer per segment, in bytes. null means use connection.bufferBytes. Default 1 MiB; range 64 KiB - 16 MiB. Silently reduced to fit connection.maxTotalBufferBytes across all live segments; the effective value is reported back as TaskDetail.effectiveBufferBytes."
|
||||
},
|
||||
"description": "Requested write buffer per segment, in bytes. null means use connection.bufferBytes. Default 1 MiB; range 64 KiB - 16 MiB. Silently reduced to fit connection.maxTotalBufferBytes across all live segments; the effective value is reported back as TaskDetail.effectiveBufferBytes."
|
||||
},
|
||||
{
|
||||
"name": "startMode",
|
||||
@@ -1029,6 +1031,79 @@
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "download.provideAuth",
|
||||
"summary": "Answer an event.",
|
||||
"description": "Answer an event.auth.required challenge. The task sits in retry_wait until this arrives; on success the daemon retries with the credentials attached and the task resumes on its own \u2014 this method does not itself start the transfer. Privileged and Unix-socket-only: a credential-bearing method must never be reachable from the browser, which is exactly the boundary event.auth.required's own description draws ('never back through this event, never into a log') \u2014 this is the other half of that promise. Credentials are handed to the Secret Service, never to SQLite and never logged; save only tells the daemon whether to persist them there for next time, or use them for this attempt alone.",
|
||||
"paramStructure": "by-name",
|
||||
"params": [
|
||||
{
|
||||
"name": "taskId",
|
||||
"schema": {
|
||||
"type": "string",
|
||||
"format": "uuid"
|
||||
},
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"name": "username",
|
||||
"schema": {
|
||||
"type": "string",
|
||||
"maxLength": 256
|
||||
},
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"name": "password",
|
||||
"schema": {
|
||||
"type": "string",
|
||||
"maxLength": 1024
|
||||
},
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"name": "save",
|
||||
"schema": {
|
||||
"type": [
|
||||
"boolean",
|
||||
"null"
|
||||
],
|
||||
"description": "true persists the credential in the Secret Service, keyed by host and realm, for future downloads from the same site. false or null uses it for this task's retry only. Never affects SQLite or the daemon's logs either way."
|
||||
},
|
||||
"description": "true persists the credential in the Secret Service, keyed by host and realm, for future downloads from the same site. false or null uses it for this task's retry only. Never affects SQLite or the daemon's logs either way."
|
||||
}
|
||||
],
|
||||
"result": {
|
||||
"name": "download.provideAuthResult",
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"ok"
|
||||
],
|
||||
"properties": {
|
||||
"ok": {
|
||||
"type": "boolean"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"x-privileged": true,
|
||||
"x-transports": [
|
||||
"uds"
|
||||
],
|
||||
"x-deadlineMs": 5000,
|
||||
"errors": [
|
||||
{
|
||||
"code": -32003,
|
||||
"message": "Method is privileged and was called over a transport that may not use it."
|
||||
},
|
||||
{
|
||||
"code": -32010,
|
||||
"message": "No task with that id."
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "download.refreshUrl",
|
||||
"summary": "IDM's 'Refresh Download Address'.",
|
||||
@@ -1366,8 +1441,9 @@
|
||||
"integer",
|
||||
"null"
|
||||
],
|
||||
"minimum": 4096,
|
||||
"maximum": 8388608
|
||||
"minimum": 65536,
|
||||
"maximum": 16777216,
|
||||
"description": "The REQUESTED write buffer per segment. Subject to the same maxTotalBufferBytes reduction as DownloadSpec.bufferBytes; the effective value comes back on the next download.get."
|
||||
},
|
||||
"checksum": {
|
||||
"oneOf": [
|
||||
@@ -3131,15 +3207,16 @@
|
||||
],
|
||||
"minimum": 1,
|
||||
"maximum": 32,
|
||||
"description": "The REQUESTED connection count. An upper bound, not a promise: the daemon lowers it to the per-host cap, and to 1 when the source turns out not to be resumable. What is actually in use comes back as TaskSummary.segments. null means use connection.maxSegmentsPerDownload."
|
||||
"description": "The REQUESTED connection count. An upper bound, not a promise: the engine lowers it to the per-host cap, and to 1 when the source turns out not to be resumable. What is actually in use comes back as TaskSummary.segments. null means use connection.maxSegmentsPerDownload."
|
||||
},
|
||||
"bufferBytes": {
|
||||
"type": [
|
||||
"integer",
|
||||
"null"
|
||||
],
|
||||
"minimum": 4096,
|
||||
"maximum": 8388608
|
||||
"minimum": 65536,
|
||||
"maximum": 16777216,
|
||||
"description": "Requested write buffer per segment, in bytes. null means use connection.bufferBytes. Default 1 MiB; range 64 KiB - 16 MiB. Silently reduced to fit connection.maxTotalBufferBytes across all live segments; the effective value is reported back as TaskDetail.effectiveBufferBytes."
|
||||
},
|
||||
"startMode": {
|
||||
"$ref": "#/components/schemas/StartMode"
|
||||
@@ -3760,6 +3837,8 @@
|
||||
"connection.preset",
|
||||
"connection.maxSegmentsPerDownload",
|
||||
"connection.bufferBytes",
|
||||
"connection.maxTotalBufferBytes",
|
||||
"connection.maxActiveSegments",
|
||||
"connection.maxConcurrentDownloads",
|
||||
"connection.timeoutSec",
|
||||
"connection.maxRetries",
|
||||
@@ -3883,8 +3962,9 @@
|
||||
},
|
||||
"connection.bufferBytes": {
|
||||
"type": "integer",
|
||||
"minimum": 4096,
|
||||
"maximum": 8388608
|
||||
"minimum": 65536,
|
||||
"maximum": 16777216,
|
||||
"description": "Default per-segment write buffer, in bytes, when a task does not request its own. Default 1 MiB (1048576); range 64 KiB - 16 MiB. This is the single biggest throughput knob and is exposed in Options -> Downloads -> 'Write buffer per connection'."
|
||||
},
|
||||
"connection.maxConcurrentDownloads": {
|
||||
"type": "integer",
|
||||
@@ -3973,6 +4053,18 @@
|
||||
},
|
||||
"sounds.onError": {
|
||||
"type": "string"
|
||||
},
|
||||
"connection.maxTotalBufferBytes": {
|
||||
"type": "integer",
|
||||
"minimum": 16777216,
|
||||
"maximum": 2147483648,
|
||||
"description": "Global cap on write-buffer memory across every live segment, in bytes. Default 128 MiB (134217728). Every live segment's buffer is reduced to fit maxTotalBufferBytes / (live segment count, capped at maxActiveSegments); the reduced value is reported per task as TaskDetail.effectiveBufferBytes. Exists so a burst of large downloads with a large per-segment buffer cannot exhaust memory."
|
||||
},
|
||||
"connection.maxActiveSegments": {
|
||||
"type": "integer",
|
||||
"minimum": 1,
|
||||
"maximum": 256,
|
||||
"description": "Global ceiling on segments actually transferring at once, across every task. Default 32. This is the real bound behind '20 active downloads': the rest of each download's segments queue rather than all dialling out simultaneously. DAEMON's scheduler needs this value to decide what to admit; CORE enforces it."
|
||||
}
|
||||
},
|
||||
"title": "Settings"
|
||||
@@ -4040,8 +4132,18 @@
|
||||
"integer",
|
||||
"null"
|
||||
],
|
||||
"minimum": 4096,
|
||||
"maximum": 8388608
|
||||
"minimum": 65536,
|
||||
"maximum": 16777216,
|
||||
"description": "The REQUESTED write buffer per segment. See effectiveBufferBytes for what is actually in use."
|
||||
},
|
||||
"effectiveBufferBytes": {
|
||||
"type": [
|
||||
"integer",
|
||||
"null"
|
||||
],
|
||||
"minimum": 65536,
|
||||
"maximum": 16777216,
|
||||
"description": "The write buffer actually in use per live segment, right now. May be well below bufferBytes: the engine reduces every live segment's buffer to fit connection.maxTotalBufferBytes across connection.maxActiveSegments concurrently-transferring segments, and reports the reduced value here so the GUI can show '16 MiB (using 4 MiB)'. null before the task has started its first segment."
|
||||
},
|
||||
"partPath": {
|
||||
"type": [
|
||||
@@ -4082,7 +4184,7 @@
|
||||
"title": "TaskDetail"
|
||||
},
|
||||
"TaskError": {
|
||||
"description": "Why a task is in the failed or retry_wait state. Distinct from the JSON-RPC Error, which describes a failed call rather than a failed download \u2014 the two live in different code spaces on purpose, and `code` here is a TaskErrorCode string, never a JSON-RPC integer.",
|
||||
"description": "Why a task is in the failed, retry_wait, or (when the daemon paused it on its own initiative rather than the user) paused state. Distinct from the JSON-RPC Error, which describes a failed call rather than a failed download \u2014 the two live in different code spaces on purpose, and `code` here is a TaskErrorCode string, never a JSON-RPC integer. A pause the user or the scheduler requested carries no error: this field only explains a paused state the daemon entered unilaterally (auth_required, server_file_changed, disk_full and the like), never a deliberate one.",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
@@ -4467,7 +4569,8 @@
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
]
|
||||
],
|
||||
"description": "Set when state is failed or retry_wait, and also when state is paused and the daemon entered that state on its own initiative rather than at a user's or scheduler's request. null on every other state, including a deliberate pause."
|
||||
}
|
||||
},
|
||||
"title": "TaskSummary"
|
||||
@@ -4782,7 +4885,7 @@
|
||||
},
|
||||
{
|
||||
"name": "event.task.state",
|
||||
"description": "A task changed lifecycle state. Carries the summary so the row can be repainted in full without a round trip, and error whenever the new state is failed or retry_wait.",
|
||||
"description": "A task changed lifecycle state. Carries the summary so the row can be repainted in full without a round trip, and error whenever the daemon has something to say about why: on every failed or retry_wait transition, and on a paused transition the daemon entered unilaterally rather than at a user's or scheduler's request.",
|
||||
"params": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
@@ -4826,7 +4929,8 @@
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
]
|
||||
],
|
||||
"description": "Set when the new state is failed or retry_wait, and also when it is paused and the daemon entered that state on its own initiative \u2014 auth_required, server_file_changed, disk_full and the like \u2014 rather than because of a user action, a schedule window closing, or an admission-control decision. null on every other transition, including every deliberately-requested pause. A client must not assume a paused task has no error just because it usually doesn't; check this field rather than the state name alone."
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
# PROTO → DAEMON — answers to `daemon/docs/proto-requests-m1.md`
|
||||
|
||||
Status: **answered**. Against `contracts/` at **1.4.0** (`lane/proto`).
|
||||
Raised by DAEMON while building `rpc/` against 1.3.0.
|
||||
|
||||
---
|
||||
|
||||
## P1 — the generated `Dispatcher` has no error channel below `-32603` · **landed in 1.4.0**
|
||||
|
||||
Done, essentially as sketched. The generated C++ binding now has two error channels,
|
||||
kept deliberately separate:
|
||||
|
||||
| Channel | Type | Raised by | Always |
|
||||
|---|---|---|---|
|
||||
| parse | `Result<T>` = `expected<T, ParseError>` | `dispatch()` turning the wire into typed params | `-32602`, structural, `data.path` a JSON pointer |
|
||||
| handler | `HandlerResult<T>` = `expected<T, HandlerError>` | a `Dispatcher::on_*` method | any contract code + free-form `data` |
|
||||
|
||||
```cpp
|
||||
struct HandlerError {
|
||||
ErrorCode code{ErrorCode::InternalError}; // default: a bare HandlerError{} is a valid -32603
|
||||
std::string message;
|
||||
nlohmann::json data = nullptr; // forwarded straight into the JSON-RPC error's data
|
||||
};
|
||||
template <class T> using HandlerResult = std::expected<T, HandlerError>;
|
||||
```
|
||||
|
||||
Every `Dispatcher::on_*` now returns `HandlerResult<T>`. `dispatch()`'s handler-error
|
||||
branch went from a hard-coded `InternalError` to:
|
||||
|
||||
```cpp
|
||||
if (!r) return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
```
|
||||
|
||||
So the three in-handler fixtures are now satisfiable by a conformant server:
|
||||
|
||||
| Fixture | `return std::unexpected(HandlerError{ ... })` |
|
||||
|---|---|
|
||||
| `download.get.not-found` | `ErrorCode::TaskNotFound, "no such task", {{"taskId", id}}` |
|
||||
| `download.add.invalid-path` | `ErrorCode::InvalidPath, "outside allowed roots", {{"path", p}}` |
|
||||
| `download.probe.probe-failed` | `ErrorCode::ProbeFailed, "HTTP 403", {{"httpStatus", 403}}` |
|
||||
|
||||
`session.pair.rate-limited` (`-32014`) is a handler result too if you want it there —
|
||||
nothing stops a handler returning `HandlerError{ErrorCode::RateLimited, ...,
|
||||
{{"retryAfterSec", 60}}}`. `-32001/-32002/-32003` stay yours to raise in the server layer
|
||||
around `dispatch()`, as you're already doing; they're decided before or without reference
|
||||
to method params, and `HandlerError`'s own doc comment says so.
|
||||
|
||||
Verified end to end: a handler returning each of the above through the real `dispatch()`
|
||||
path produces the right code with the `data` payload intact, and a bare `HandlerError{}`
|
||||
still yields a clean `-32603` with no `data` field. (Watch the nlohmann brace-init trap:
|
||||
`HandlerError{code, msg, {{"k", v}}}` gives an object, but a lone `{nullptr}` would give
|
||||
the array `[null]` — the struct's member initializer is `= nullptr` for exactly that
|
||||
reason.)
|
||||
|
||||
`FixtureDispatcher` and `conformance_main.cpp`: the generated dispatcher swapped
|
||||
`Result` → `HandlerResult` automatically; `conformance_main.cpp` only ever inspects
|
||||
`dispatch()`'s JSON output and needed no change.
|
||||
|
||||
**Version:** minor, 1.3.0 → 1.4.0. The wire is byte-identical — no schema, fixture,
|
||||
or OpenRPC change — but every implementer of `Dispatcher` must swap `Result` →
|
||||
`HandlerResult` on their `on_*` overrides or they won't compile, and a version bump is
|
||||
how lanes are told to regenerate and adapt. Minor, not major: a major would make
|
||||
`session.hello` refuse a client whose wire behaviour is unchanged. The rule — a
|
||||
generated-binding API break with an unchanged wire is minor + migration note, because
|
||||
`VERSION` is the protocol version, not the C++ ABI — is written up as
|
||||
`docs/adr/0015-generated-binding-changes-and-versioning.md` (this instance is
|
||||
mechanical; the ADR records the rule for the next one, which GUI will also consume
|
||||
since it already links `velox::proto`). `kProtocolVersion` moves to `"1.4.0"` with it.
|
||||
|
||||
## P2 — clarifications
|
||||
|
||||
**`session.hello.version-mismatch` `data.expected`.** You're right that `"1.0.0"` in the
|
||||
fixture is stale. Fixed: it's now `$any`. Echo `kProtocolVersion` (`"1.4.0"`) there — the
|
||||
conformance compare on an error fixture is on `code` only, structural elsewhere, so the
|
||||
live version string is fine and can't be pinned in a golden file that outlives version
|
||||
bumps anyway. `actual` stays the concrete bad version the fake client sent (`"2.0.0"`).
|
||||
|
||||
**`SessionHelloResult.transport` always populated.** No change requested, noted. The
|
||||
field's own description already invites it (`"Lets a client know up front which privileged
|
||||
methods will be refused"`), so always setting `"uds"` / `"ws"` is using it as intended.
|
||||
`std::optional` stays because a hand-rolled or older server may legitimately omit it and a
|
||||
client must tolerate that.
|
||||
@@ -99,8 +99,8 @@ Agreed with your ranking: these are minor under rule 4 and land as small PRs to
|
||||
|
||||
| # | Item | Verdict | Shape |
|
||||
|---|---|---|---|
|
||||
| **B2a** | readable effective buffer size | **accepted** | `effectiveBufferBytes` on `TaskSummary`, next to the effective segment count, so the requested/effective split reads the same way for both. You are right about the `additionalProperties: false` trap — no daemon can tack it on, so it needs a schema PR either way. |
|
||||
| **F2** | credential return path for 401/407 | **accepted as proposed** | `download.provideAuth {taskId, username, password, save?}` → `{ok}`. Unix socket only, privileged: a credential-bearing method must never be reachable from the browser. Secrets go to the Secret Service; `save` only tells DAEMON whether to persist. |
|
||||
| **B2a** | readable effective buffer size | **landed in 1.1.0** | `TaskDetail.effectiveBufferBytes` (placed on `TaskDetail`, not `TaskSummary` — `bufferBytes` itself was already `TaskDetail`-only, so the pair stays together). See `docs/adr/0012-buffer-and-segment-budget.md`, which also lands B4's bounds and the two new settings keys in the same PR. |
|
||||
| **F2** | credential return path for 401/407 | **landed in 1.2.0** | `download.provideAuth {taskId, username, password, save?}` → `{ok}`, exactly as proposed: Unix socket only, privileged. It answers the challenge; it does not itself resume the task — the daemon retries with the credential attached and the usual `event.task.state` reports the task leaving `retry_wait`. |
|
||||
| **F1** | "needs user decision" carrier | **the simple option** | `state: paused` + `event.notify` is the intended carrier for M1: CORE reports `server_file_changed`, DAEMON pauses and notifies, GUI offers restart. A dedicated `event.task.decision` + `download.decide` is a real design with a state machine attached, and it should not be invented in a hurry — raise it again in M3 if the notify path proves too thin. A string comparison on `error.code` covers the engine side either way, which is now a `TaskErrorCode` comparison rather than a magic number. |
|
||||
| **F3** | `checksum` string format | **already frozen, differently** | `download.add {checksum}` is **not** a string. It is a `Checksum` object: `{algorithm: "md5"\|"sha1"\|"sha256"\|"sha512", value: "<hex>"}`, with `value` patterned `^[0-9a-fA-F]{32,128}$`. Parse your `"<algo>:<hex>"` form at the CLI or GUI edge, not on the wire. Note `sha512` is accepted by the contract even though the appendix lists MD5/SHA-256 — reject it in the engine if you do not implement it, rather than the contract forbidding it. |
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://velox.dev/schema/events/event.task.state.schema.json",
|
||||
"title": "event.task.state",
|
||||
"description": "A task changed lifecycle state. Carries the summary so the row can be repainted in full without a round trip, and error whenever the new state is failed or retry_wait.",
|
||||
"description": "A task changed lifecycle state. Carries the summary so the row can be repainted in full without a round trip, and error whenever the daemon has something to say about why: on every failed or retry_wait transition, and on a paused transition the daemon entered unilaterally rather than at a user's or scheduler's request.",
|
||||
"x-direction": "server-to-client",
|
||||
"type": "object",
|
||||
"properties": {
|
||||
@@ -49,7 +49,8 @@
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
]
|
||||
],
|
||||
"description": "Set when the new state is failed or retry_wait, and also when it is paused and the daemon entered that state on its own initiative \u2014 auth_required, server_file_changed, disk_full and the like \u2014 rather than because of a user action, a schedule window closing, or an admission-control decision. null on every other transition, including every deliberately-requested pause. A client must not assume a paused task has no error just because it usually doesn't; check this field rather than the state name alone."
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://velox.dev/schema/methods/download.provideAuth.schema.json",
|
||||
"title": "download.provideAuth",
|
||||
"description": "Answer an event.auth.required challenge. The task sits in retry_wait until this arrives; on success the daemon retries with the credentials attached and the task resumes on its own — this method does not itself start the transfer. Privileged and Unix-socket-only: a credential-bearing method must never be reachable from the browser, which is exactly the boundary event.auth.required's own description draws ('never back through this event, never into a log') — this is the other half of that promise. Credentials are handed to the Secret Service, never to SQLite and never logged; save only tells the daemon whether to persist them there for next time, or use them for this attempt alone.",
|
||||
"x-privileged": true,
|
||||
"x-transports": ["uds"],
|
||||
"x-deadlineMs": 5000,
|
||||
"x-errors": [-32003, -32010],
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"params": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["taskId", "username", "password"],
|
||||
"properties": {
|
||||
"taskId": { "type": "string", "format": "uuid" },
|
||||
"username": { "type": "string", "maxLength": 256 },
|
||||
"password": { "type": "string", "maxLength": 1024 },
|
||||
"save": {
|
||||
"type": ["boolean", "null"],
|
||||
"description": "true persists the credential in the Secret Service, keyed by host and realm, for future downloads from the same site. false or null uses it for this task's retry only. Never affects SQLite or the daemon's logs either way."
|
||||
}
|
||||
}
|
||||
},
|
||||
"result": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["ok"],
|
||||
"properties": {
|
||||
"ok": { "type": "boolean" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -78,8 +78,9 @@
|
||||
"integer",
|
||||
"null"
|
||||
],
|
||||
"minimum": 4096,
|
||||
"maximum": 8388608
|
||||
"minimum": 65536,
|
||||
"maximum": 16777216,
|
||||
"description": "The REQUESTED write buffer per segment. Subject to the same maxTotalBufferBytes reduction as DownloadSpec.bufferBytes; the effective value comes back on the next download.get."
|
||||
},
|
||||
"checksum": {
|
||||
"oneOf": [
|
||||
|
||||
@@ -80,15 +80,16 @@
|
||||
],
|
||||
"minimum": 1,
|
||||
"maximum": 32,
|
||||
"description": "The REQUESTED connection count. An upper bound, not a promise: the daemon lowers it to the per-host cap, and to 1 when the source turns out not to be resumable. What is actually in use comes back as TaskSummary.segments. null means use connection.maxSegmentsPerDownload."
|
||||
"description": "The REQUESTED connection count. An upper bound, not a promise: the engine lowers it to the per-host cap, and to 1 when the source turns out not to be resumable. What is actually in use comes back as TaskSummary.segments. null means use connection.maxSegmentsPerDownload."
|
||||
},
|
||||
"bufferBytes": {
|
||||
"type": [
|
||||
"integer",
|
||||
"null"
|
||||
],
|
||||
"minimum": 4096,
|
||||
"maximum": 8388608
|
||||
"minimum": 65536,
|
||||
"maximum": 16777216,
|
||||
"description": "Requested write buffer per segment, in bytes. null means use connection.bufferBytes. Default 1 MiB; range 64 KiB - 16 MiB. Silently reduced to fit connection.maxTotalBufferBytes across all live segments; the effective value is reported back as TaskDetail.effectiveBufferBytes."
|
||||
},
|
||||
"startMode": {
|
||||
"$ref": "https://velox.dev/schema/types/StartMode.schema.json"
|
||||
|
||||
@@ -5,25 +5,48 @@
|
||||
"description": "Every settings key that exists. The Options dialog maps 1:1 onto this list and the GUI must not invent a key that is not here. Kept in lockstep with Settings.schema.json by a conformance check.",
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"general.launchOnLogin", "general.minimizeToTray", "general.showDropTarget",
|
||||
"general.confirmOnExit", "general.language", "general.checkForUpdates",
|
||||
|
||||
"capture.enabled", "capture.monitoredExtensions", "capture.monitoredMimeTypes",
|
||||
"capture.minSizeBytes", "capture.excludedHosts", "capture.bypassModifier",
|
||||
"general.launchOnLogin",
|
||||
"general.minimizeToTray",
|
||||
"general.showDropTarget",
|
||||
"general.confirmOnExit",
|
||||
"general.language",
|
||||
"general.checkForUpdates",
|
||||
"capture.enabled",
|
||||
"capture.monitoredExtensions",
|
||||
"capture.monitoredMimeTypes",
|
||||
"capture.minSizeBytes",
|
||||
"capture.excludedHosts",
|
||||
"capture.bypassModifier",
|
||||
"capture.autoStartTypes",
|
||||
|
||||
"saveTo.defaultDir", "saveTo.tempDir", "saveTo.allowedRoots",
|
||||
"saveTo.fileExistsPolicy", "saveTo.createSubfolderPerSite",
|
||||
|
||||
"connection.preset", "connection.maxSegmentsPerDownload", "connection.bufferBytes",
|
||||
"connection.maxConcurrentDownloads", "connection.timeoutSec", "connection.maxRetries",
|
||||
"saveTo.defaultDir",
|
||||
"saveTo.tempDir",
|
||||
"saveTo.allowedRoots",
|
||||
"saveTo.fileExistsPolicy",
|
||||
"saveTo.createSubfolderPerSite",
|
||||
"connection.preset",
|
||||
"connection.maxSegmentsPerDownload",
|
||||
"connection.bufferBytes",
|
||||
"connection.maxTotalBufferBytes",
|
||||
"connection.maxActiveSegments",
|
||||
"connection.maxConcurrentDownloads",
|
||||
"connection.timeoutSec",
|
||||
"connection.maxRetries",
|
||||
"connection.retryBackoffSec",
|
||||
|
||||
"downloads.speedLimitBps", "downloads.speedLimitEnabled", "downloads.virusScanCommand",
|
||||
"downloads.postDownloadCommand", "downloads.duplicatePolicy", "downloads.verifyChecksums",
|
||||
|
||||
"proxy.mode", "proxy.host", "proxy.port", "proxy.username", "proxy.bypassHosts", "proxy.pacUrl",
|
||||
|
||||
"sounds.enabled", "sounds.onComplete", "sounds.onQueueComplete", "sounds.onError"
|
||||
"downloads.speedLimitBps",
|
||||
"downloads.speedLimitEnabled",
|
||||
"downloads.virusScanCommand",
|
||||
"downloads.postDownloadCommand",
|
||||
"downloads.duplicatePolicy",
|
||||
"downloads.verifyChecksums",
|
||||
"proxy.mode",
|
||||
"proxy.host",
|
||||
"proxy.port",
|
||||
"proxy.username",
|
||||
"proxy.bypassHosts",
|
||||
"proxy.pacUrl",
|
||||
"sounds.enabled",
|
||||
"sounds.onComplete",
|
||||
"sounds.onQueueComplete",
|
||||
"sounds.onError"
|
||||
]
|
||||
}
|
||||
|
||||
@@ -101,8 +101,9 @@
|
||||
},
|
||||
"connection.bufferBytes": {
|
||||
"type": "integer",
|
||||
"minimum": 4096,
|
||||
"maximum": 8388608
|
||||
"minimum": 65536,
|
||||
"maximum": 16777216,
|
||||
"description": "Default per-segment write buffer, in bytes, when a task does not request its own. Default 1 MiB (1048576); range 64 KiB - 16 MiB. This is the single biggest throughput knob and is exposed in Options -> Downloads -> 'Write buffer per connection'."
|
||||
},
|
||||
"connection.maxConcurrentDownloads": {
|
||||
"type": "integer",
|
||||
@@ -191,6 +192,18 @@
|
||||
},
|
||||
"sounds.onError": {
|
||||
"type": "string"
|
||||
},
|
||||
"connection.maxTotalBufferBytes": {
|
||||
"type": "integer",
|
||||
"minimum": 16777216,
|
||||
"maximum": 2147483648,
|
||||
"description": "Global cap on write-buffer memory across every live segment, in bytes. Default 128 MiB (134217728). Every live segment's buffer is reduced to fit maxTotalBufferBytes / (live segment count, capped at maxActiveSegments); the reduced value is reported per task as TaskDetail.effectiveBufferBytes. Exists so a burst of large downloads with a large per-segment buffer cannot exhaust memory."
|
||||
},
|
||||
"connection.maxActiveSegments": {
|
||||
"type": "integer",
|
||||
"minimum": 1,
|
||||
"maximum": 256,
|
||||
"description": "Global ceiling on segments actually transferring at once, across every task. Default 32. This is the real bound behind '20 active downloads': the rest of each download's segments queue rather than all dialling out simultaneously. DAEMON's scheduler needs this value to decide what to admit; CORE enforces it."
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -54,8 +54,18 @@
|
||||
"integer",
|
||||
"null"
|
||||
],
|
||||
"minimum": 4096,
|
||||
"maximum": 8388608
|
||||
"minimum": 65536,
|
||||
"maximum": 16777216,
|
||||
"description": "The REQUESTED write buffer per segment. See effectiveBufferBytes for what is actually in use."
|
||||
},
|
||||
"effectiveBufferBytes": {
|
||||
"type": [
|
||||
"integer",
|
||||
"null"
|
||||
],
|
||||
"minimum": 65536,
|
||||
"maximum": 16777216,
|
||||
"description": "The write buffer actually in use per live segment, right now. May be well below bufferBytes: the engine reduces every live segment's buffer to fit connection.maxTotalBufferBytes across connection.maxActiveSegments concurrently-transferring segments, and reports the reduced value here so the GUI can show '16 MiB (using 4 MiB)'. null before the task has started its first segment."
|
||||
},
|
||||
"partPath": {
|
||||
"type": [
|
||||
|
||||
@@ -2,17 +2,60 @@
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://velox.dev/schema/types/TaskError.schema.json",
|
||||
"title": "TaskError",
|
||||
"description": "Why a task is in the failed or retry_wait state. Distinct from the JSON-RPC Error, which describes a failed call rather than a failed download — the two live in different code spaces on purpose, and `code` here is a TaskErrorCode string, never a JSON-RPC integer.",
|
||||
"description": "Why a task is in the failed, retry_wait, or (when the daemon paused it on its own initiative rather than the user) paused state. Distinct from the JSON-RPC Error, which describes a failed call rather than a failed download \u2014 the two live in different code spaces on purpose, and `code` here is a TaskErrorCode string, never a JSON-RPC integer. A pause the user or the scheduler requested carries no error: this field only explains a paused state the daemon entered unilaterally (auth_required, server_file_changed, disk_full and the like), never a deliberate one.",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["code", "message", "retryable"],
|
||||
"required": [
|
||||
"code",
|
||||
"message",
|
||||
"retryable"
|
||||
],
|
||||
"properties": {
|
||||
"code": { "$ref": "https://velox.dev/schema/types/TaskErrorCode.schema.json" },
|
||||
"message": { "type": "string", "description": "Human-readable, safe to show a user. Never carries a credential, a token or a full local path outside the download roots." },
|
||||
"httpStatus": { "type": ["integer", "null"], "minimum": 100, "maximum": 599, "description": "Set for the codes listed in TaskErrorCode's x-carriesHttpStatus, and null otherwise." },
|
||||
"retryable": { "type": "boolean", "description": "Whether the scheduler will pick this task up again on its own. Carried per-occurrence rather than derived from the code, because 'probe_failed' is retryable or not depending on what the probe hit." },
|
||||
"cause": { "oneOf": [{ "$ref": "https://velox.dev/schema/types/TaskErrorCode.schema.json" }, { "type": "null" }], "description": "The underlying failure, for codes that wrap one. max_retries_exhausted sets it to whatever the last attempt actually failed with, so a user learns the reason rather than just that Velox gave up." },
|
||||
"attempt": { "type": ["integer", "null"], "minimum": 0, "description": "How many attempts have been made so far." },
|
||||
"nextRetryAt":{ "type": ["string", "null"], "format": "date-time" }
|
||||
"code": {
|
||||
"$ref": "https://velox.dev/schema/types/TaskErrorCode.schema.json"
|
||||
},
|
||||
"message": {
|
||||
"type": "string",
|
||||
"description": "Human-readable, safe to show a user. Never carries a credential, a token or a full local path outside the download roots."
|
||||
},
|
||||
"httpStatus": {
|
||||
"type": [
|
||||
"integer",
|
||||
"null"
|
||||
],
|
||||
"minimum": 100,
|
||||
"maximum": 599,
|
||||
"description": "Set for the codes listed in TaskErrorCode's x-carriesHttpStatus, and null otherwise."
|
||||
},
|
||||
"retryable": {
|
||||
"type": "boolean",
|
||||
"description": "Whether the scheduler will pick this task up again on its own. Carried per-occurrence rather than derived from the code, because 'probe_failed' is retryable or not depending on what the probe hit."
|
||||
},
|
||||
"cause": {
|
||||
"oneOf": [
|
||||
{
|
||||
"$ref": "https://velox.dev/schema/types/TaskErrorCode.schema.json"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"description": "The underlying failure, for codes that wrap one. max_retries_exhausted sets it to whatever the last attempt actually failed with, so a user learns the reason rather than just that Velox gave up."
|
||||
},
|
||||
"attempt": {
|
||||
"type": [
|
||||
"integer",
|
||||
"null"
|
||||
],
|
||||
"minimum": 0,
|
||||
"description": "How many attempts have been made so far."
|
||||
},
|
||||
"nextRetryAt": {
|
||||
"type": [
|
||||
"string",
|
||||
"null"
|
||||
],
|
||||
"format": "date-time"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -132,7 +132,8 @@
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
]
|
||||
],
|
||||
"description": "Set when state is failed or retry_wait, and also when state is paused and the daemon entered that state on its own initiative rather than at a user's or scheduler's request. null on every other state, including a deliberate pause."
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+56
-19
@@ -1,20 +1,37 @@
|
||||
# libveloxcore — the download engine. Lane CORE.
|
||||
#
|
||||
# No JSON, no SQL, no Qt, no RPC in this tree (CLAUDE.md §3, AGENT-CORE brief).
|
||||
# This file is self-contained; it is wired into the build by PKG uncommenting
|
||||
# `add_subdirectory(core)` in the root CMakeLists.txt (see core/docs/pkg-requests-m1.md).
|
||||
# core/ produces TWO targets (ADR 0009):
|
||||
# veloxcore — the download engine. No JSON, no SQL, no Qt, no RPC. Ever (CLAUDE.md §3).
|
||||
# veloxproto — the generated wire types, which ARE JSON. NOT linked by veloxcore.
|
||||
# The `no JSON in core/` rule constrains core/src/ and core/include/; core/generated/ is
|
||||
# the sanctioned exception. Wired in by PKG via add_subdirectory(core) in the root file.
|
||||
|
||||
find_package(Threads REQUIRED)
|
||||
find_package(CURL 8.0 REQUIRED)
|
||||
find_package(OpenSSL REQUIRED)
|
||||
|
||||
add_library(veloxcore STATIC
|
||||
src/util/error.cpp
|
||||
src/util/log.cpp
|
||||
src/util/thread_pool.cpp
|
||||
src/net/curl_error.cpp
|
||||
src/net/http_client.cpp
|
||||
src/net/text_codec.cpp
|
||||
src/net/content_disposition.cpp
|
||||
src/net/url.cpp
|
||||
src/net/probe.cpp
|
||||
src/io/sparse_file.cpp
|
||||
src/io/write_buffer.cpp
|
||||
src/meta/veloxpart.cpp
|
||||
src/segment/segmenter.cpp
|
||||
src/segment/budget.cpp
|
||||
src/task/digest.cpp
|
||||
src/task/download_task.cpp
|
||||
src/engine.cpp
|
||||
)
|
||||
add_library(velox::core ALIAS veloxcore)
|
||||
|
||||
target_include_directories(veloxcore PUBLIC
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/include
|
||||
target_include_directories(veloxcore
|
||||
PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/include
|
||||
PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/src # net/*.cpp -> "net/curl_error.hpp"
|
||||
)
|
||||
|
||||
target_compile_features(veloxcore PUBLIC cxx_std_23)
|
||||
@@ -25,21 +42,41 @@ target_compile_options(veloxcore PRIVATE
|
||||
-Wall -Wextra -Wpedantic -Werror
|
||||
)
|
||||
|
||||
target_link_libraries(veloxcore PUBLIC Threads::Threads)
|
||||
target_link_libraries(veloxcore PUBLIC Threads::Threads CURL::libcurl PRIVATE OpenSSL::Crypto)
|
||||
|
||||
# Later stages add: find_package(CURL 8.0) for net/, find_package(OpenSSL) for meta/.
|
||||
# Later stages add: find_package(OpenSSL) for meta/ (streaming SHA-256 + resume CRC).
|
||||
|
||||
# --- libveloxproto — generated wire code (ADR 0009) --------------------------------------
|
||||
# Its own target so libveloxcore stays JSON-free. Consumed by veloxd, the CLI, the GUI and
|
||||
# the conformance runner. The root CMakeLists only find_package(nlohmann_json)'s when
|
||||
# daemon/ has landed, so find it here too — this must build even if core is the only lane.
|
||||
if(NOT TARGET nlohmann_json::nlohmann_json)
|
||||
find_package(nlohmann_json 3.11 REQUIRED)
|
||||
endif()
|
||||
|
||||
add_library(veloxproto STATIC generated/velox_proto.cpp)
|
||||
add_library(velox::proto ALIAS veloxproto)
|
||||
|
||||
target_include_directories(veloxproto PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/generated)
|
||||
target_compile_features(veloxproto PUBLIC cxx_std_23)
|
||||
target_link_libraries(veloxproto PUBLIC nlohmann_json::nlohmann_json)
|
||||
|
||||
# Generated code is committed and never hand-edited (CLAUDE.md §2); do not fail the build
|
||||
# on a codegen quirk that trips -Werror. Warnings stay on for visibility.
|
||||
target_compile_options(veloxproto PRIVATE -Wall -Wextra -Wno-error)
|
||||
|
||||
# A build-time tripwire for the split ADR 0009 exists to protect: veloxcore must never end
|
||||
# up linking veloxproto.
|
||||
get_target_property(_core_links veloxcore LINK_LIBRARIES)
|
||||
if(_core_links AND "veloxproto" IN_LIST _core_links)
|
||||
message(FATAL_ERROR "veloxcore links veloxproto — ADR 0009 violation (engine sees JSON).")
|
||||
endif()
|
||||
|
||||
if(VELOX_BUILD_TESTS)
|
||||
add_subdirectory(tests)
|
||||
endif()
|
||||
|
||||
# libFuzzer is clang-only; a GCC configure with -DVELOX_BUILD_FUZZ=ON (the `ci` preset)
|
||||
# must not hard-fail. No fuzz targets exist yet — they arrive with net/probe (stage 3)
|
||||
# and meta/veloxpart (stage 5).
|
||||
if(VELOX_BUILD_FUZZ AND NOT CMAKE_CXX_COMPILER_ID MATCHES "Clang")
|
||||
message(STATUS "veloxcore: VELOX_BUILD_FUZZ set but compiler is "
|
||||
"${CMAKE_CXX_COMPILER_ID}; fuzz targets need Clang and will be skipped.")
|
||||
endif()
|
||||
# When fuzz targets land (stage 3: Content-Disposition, URL; stage 5: .veloxpart.meta),
|
||||
# they are added here under `if(VELOX_BUILD_FUZZ AND CMAKE_CXX_COMPILER_ID MATCHES "Clang")`
|
||||
# and live in ${CMAKE_SOURCE_DIR}/tools/fuzz (owned by lane CORE).
|
||||
# Fuzz targets live in ${CMAKE_SOURCE_DIR}/tools/fuzz (lane CORE) and are wired in by the
|
||||
# top-level CMakeLists.txt, which add_subdirectory()s every tools/* with a CMakeLists.
|
||||
# tools/fuzz/CMakeLists.txt self-guards on VELOX_BUILD_FUZZ + a Clang compiler.
|
||||
# Present: Content-Disposition, URL (stage 3). Coming: .veloxpart.meta (stage 5).
|
||||
|
||||
@@ -0,0 +1,177 @@
|
||||
# CORE response to ADR 0011 (admission control & the segment budget)
|
||||
|
||||
**Verdict: accept the decision, with three amendments and one caveat on §3.5.** None of
|
||||
the amendments change DAEMON's task-unit model — `daemon/src/sched/` is unblocked. The
|
||||
caveat tightens what "admission implies progress" can promise.
|
||||
|
||||
Signing off on §1 (each ceiling enforced once, by unit owner), §2 (the single clamp),
|
||||
§4 (per-host caps split by unit from one table), §6 (contract gap / daemon-local stopgap),
|
||||
and the three rejected alternatives — the shared-semaphore rejection especially.
|
||||
|
||||
---
|
||||
|
||||
## Answers to the five open questions
|
||||
|
||||
### Q1 — Min-1 before seconds: implementable in the stealer without inversion? **Yes.**
|
||||
|
||||
Slot allocation is a pure function of `(running tasks, their held-slot counts, their
|
||||
effective per-task caps, DAEMON's priority order, total budget)`. It runs to completion on
|
||||
every budget-changing edge (segment complete / start / pause / fail, task admit / pause /
|
||||
resume, a steal, a cap change), in two ordered passes:
|
||||
|
||||
1. **Guarantee pass.** Walk tasks holding **zero** slots *in DAEMON's priority order*; give
|
||||
each one slot while the pool is non-empty.
|
||||
2. **Growth pass.** While the pool is non-empty, round-robin over running tasks in priority
|
||||
order, granting one slot to any task below its effective cap
|
||||
(`min(spec.segments ?? maxSegmentsPerDownload, host_cap, resumable ? ∞ : 1)`), until the
|
||||
pool empties or nobody wants more.
|
||||
|
||||
The inversion DAEMON is worried about at slot release does not occur, because **a released
|
||||
slot always goes to the pool and the allocator re-runs from pass 1** — it is never handed
|
||||
back locally to the releasing task. If task A's segment finishes and both A and a
|
||||
lower-priority C now hold zero, the guarantee pass processes the zero-slot set *in priority
|
||||
order*, so A gets it back, not C. A task that drops to zero re-enters the guarantee queue
|
||||
at its priority position, not the back.
|
||||
|
||||
Liveness of min-1 depends on DAEMON honouring §2's clamp — never running more tasks than
|
||||
`maxActiveSegments`. If it admits 3 running tasks against a budget of 2, one starves by
|
||||
construction and no fairness rule fixes it. §2 already says this; calling it out because
|
||||
it is load-bearing for Q1.
|
||||
|
||||
### Q2 — Probe pool outside the segment budget, size 4: **confirmed.**
|
||||
|
||||
CORE's probe path is a dedicated pool, independent of `maxActiveSegments`. A probe is a
|
||||
`Request` with `method = HEAD` (or `range = {0,0}`) whose `on_head` returns `abort`; it
|
||||
never allocates a transfer slot. Default pool size 4, exposed as
|
||||
`set_probe_pool_size(uint32_t)`. Probe cancellation is immediate (a timed-out probe frees
|
||||
its pool slot at once), so `capture.offer` answering `ignore`-first-probe-after works.
|
||||
Probes still open real sockets — DAEMON should still bound probe *submission* on its side;
|
||||
CORE bounds *concurrency*, not queue depth.
|
||||
|
||||
### Q3 — `set_max_active_segments()` live-apply: **drain, never kill.**
|
||||
|
||||
- **Raise:** new slots available immediately; allocator runs; starved then growing tasks
|
||||
take them.
|
||||
- **Lower:** in-flight segments run to their next boundary. No new segment starts while
|
||||
`active > new_ceiling`. Completed segments' slots are withheld until
|
||||
`active ≤ new_ceiling`. Nothing is aborted, no partial range is lost.
|
||||
- **Edge:** if `new_ceiling < running_task_count`, CORE honours min-1 for the top
|
||||
`new_ceiling` tasks in priority order; the remainder are held at zero slots and reported
|
||||
in `tasks_starved`. CORE does **not** auto-pause them — that's policy. After a live
|
||||
lower, DAEMON must reconcile its running set against the new clamp (pause the
|
||||
lowest-priority excess).
|
||||
|
||||
### Q4 — Priority shape: **an ordered list, pushed on change — not an integer, not per tick.**
|
||||
|
||||
`set_task_order(std::span<const TaskId>)`, called by DAEMON whenever the order changes
|
||||
(admit, remove, reorder, priority edit, queue switch). CORE caches it and the allocator
|
||||
walks it. An integer priority would force CORE to implement tie-breaking (FIFO by
|
||||
admission time, queue precedence) — that's DAEMON policy and DAEMON already has the total
|
||||
order. Not per tick: an unchanged order isn't re-sent. A running task absent from the list
|
||||
sorts last (shouldn't happen; defensive).
|
||||
|
||||
### Q5 — Budget-change callback coalescing: **4 Hz for counts, immediate on the edge.**
|
||||
|
||||
`on_budget_changed` coalesced at ≤4 Hz to match `event.task.progress` — DAEMON isn't
|
||||
making sub-250 ms admission decisions. **Exception:** fire immediately when
|
||||
`tasks_starved` crosses 0→non-zero or non-zero→0, so DAEMON's invariant check and any UI
|
||||
reaction see that transition without up to 250 ms of lag.
|
||||
|
||||
---
|
||||
|
||||
## Amendments requested to the ADR
|
||||
|
||||
### A1 — §3.3 wording: distinguish *steal* from *yield*
|
||||
|
||||
"A steal is slot-neutral" is true for the steal the ADR means (a worker that finished its
|
||||
range takes the tail of the largest remaining range — the same worker, the same slot).
|
||||
Min-1 also needs a second, non-neutral operation:
|
||||
|
||||
- **Yield** — the allocator marks an over-quota task to release one slot *at its next
|
||||
segment boundary*. When that segment completes the slot goes to the pool → guarantee
|
||||
pass → starved task. It is a slot transfer, not slot-neutral, and it is bounded by the
|
||||
yielding segment's remaining bytes (never a mid-segment kill).
|
||||
|
||||
Please add "yield" to §3.3 as the mechanism that satisfies §3.1 when the budget is full.
|
||||
"Steal" stays exactly as written.
|
||||
|
||||
### A2 — §3.5 / §3.6: "admission implies progress" is bounded-delay, not immediate
|
||||
|
||||
When the budget is full of healthy incumbents, a newly admitted task's first slot
|
||||
materialises only when some incumbent segment reaches a boundary (yield) — bounded by that
|
||||
segment's remaining bytes, hard-capped by the stall timeout (`low_speed_secs`, default
|
||||
30 s, after which a stalled segment fails and frees its slot). So the true bound is
|
||||
|
||||
```
|
||||
time_to_first_slot ≤ min(incumbent's next boundary, low_speed_secs) + connect_timeout
|
||||
```
|
||||
|
||||
not "connect timeout + per-host cap" alone. Two consequences:
|
||||
|
||||
- §3.6's **2 s** assertion window is too tight — a legitimately full budget with a slow
|
||||
incumbent tail can hold a new task at zero for longer than 2 s with nothing wrong.
|
||||
Recommend the warning threshold be `low_speed_secs + connect_timeout_ms` (~45 s), or
|
||||
configurable.
|
||||
- CORE will expose `starved_since` (a monotonic timestamp) per starved task via the
|
||||
diagnostics call below, so DAEMON can tell "briefly waiting for a boundary" from
|
||||
"wedged" without guessing.
|
||||
|
||||
Optional future tightening (not M1): a **preemptive split** — truncate an incumbent's
|
||||
largest remaining range ahead of its current offset and hand the freed tail to the starved
|
||||
task as a new segment. Zero bytes lost, first slot within one round-trip. CORE will add
|
||||
this if the yield delay proves painful in the soak test; it doesn't change the API.
|
||||
|
||||
### A3 — C1 API: add a starved-set accessor and pin two definitions
|
||||
|
||||
```
|
||||
struct EngineBudget { uint32_t total; uint32_t active; uint32_t tasks_starved; };
|
||||
EngineBudget budget() const;
|
||||
uint32_t segments_active(TaskId) const; // slots held, any state
|
||||
std::vector<TaskId> starved_tasks() const; // diagnostics / velox ls --json
|
||||
std::optional<SteadyTime> starved_since(TaskId) const; // per A2
|
||||
void on_budget_changed(std::function<void(EngineBudget)>);
|
||||
```
|
||||
|
||||
Definitions, so the projection to `TaskSummary.segments` (ADR 0010: effective count) is
|
||||
unambiguous:
|
||||
|
||||
- **`segments_active(id)`** = slots the task holds, counting a segment in `connecting`
|
||||
(0 bytes yet) as well as `downloading`. This is what the user sees as "using N
|
||||
connections."
|
||||
- **`tasks_starved`** counts running tasks with `segments_active == 0`. A task with a
|
||||
`connecting` segment is **not** starved — it is progressing.
|
||||
|
||||
---
|
||||
|
||||
## C2–C6 confirmations
|
||||
|
||||
- **C2:** drain-not-kill, per Q3. Confirmed as DAEMON assumed.
|
||||
- **C3:** `set_host_segment_cap(std::string host, uint32_t)` — confirmed. CORE keeps the
|
||||
`host → cap` map, applies it in the per-task effective cap and in the stealer (no Nth
|
||||
connection to a host capped at N-1). DAEMON owns the table; CORE derives a task's host
|
||||
from its URL + mirror set.
|
||||
- **C4:** agreed it's PROTO's, same bundle as CORE's B2a / `buffer-sizing.md` asks
|
||||
(`connection.maxActiveSegments`, `connection.maxTotalBufferBytes`). DAEMON's local-value
|
||||
stopgap is fine.
|
||||
- **C5:** signed off — see Q1 plus amendments A1/A2. Min-1 is buildable; "implies
|
||||
progress" is bounded-delay; steal stays slot-neutral, yield is the transfer op.
|
||||
- **C6:** confirmed — see Q2.
|
||||
|
||||
---
|
||||
|
||||
## New API surface CORE will expose for `sched/` (summary)
|
||||
|
||||
```
|
||||
void set_max_active_segments(uint32_t); // drain-not-kill
|
||||
void set_host_segment_cap(std::string host, uint32_t);
|
||||
void set_task_order(std::span<const TaskId>); // pushed on change
|
||||
void set_probe_pool_size(uint32_t); // default 4
|
||||
EngineBudget budget() const;
|
||||
uint32_t segments_active(TaskId) const;
|
||||
std::vector<TaskId> starved_tasks() const;
|
||||
std::optional<SteadyTime> starved_since(TaskId) const;
|
||||
void on_budget_changed(std::function<void(EngineBudget)>); // 4 Hz + starved edge
|
||||
```
|
||||
|
||||
All of it lands with stage 6 (segmenter/stealer) / stage 8 (download_task). None of it is
|
||||
on the M1 critical path ahead of where DAEMON needs it; flag if the ordering is wrong.
|
||||
@@ -0,0 +1,97 @@
|
||||
# CORE response to ADR 0013 (task state-machine ownership)
|
||||
|
||||
**Verdict: accept as written.** No amendments to the decision. `daemon/src/sched/`'s
|
||||
pause/resume logic is unblocked from CORE's side once PROTO lands open item 3 (the
|
||||
`error`-on-`paused` widening — see below).
|
||||
|
||||
The split in §1, the shared-`paused` + idempotency contract in §2, the cross-reason resume
|
||||
rule in §3, the `retry_wait ≠ starvation` resolution in §4, and the restart handling in §5
|
||||
all match what CORE agreed in `contracts/proto-answers-m1.md` D1 and the ADR 0011 response.
|
||||
|
||||
Everything below is a **design commitment** — CORE's state machine is stage 8 and
|
||||
`starved_tasks()` is stage 6/8; none of it is built yet. So "does CORE already do X" is
|
||||
answered as "CORE will do X, and here is the spec it will be built to", not as a report on
|
||||
existing code.
|
||||
|
||||
---
|
||||
|
||||
## Answers to the four open items
|
||||
|
||||
### 1 — `starved_tasks()` / `tasks_starved` exclude `retry_wait` and auto-paused, by construction
|
||||
|
||||
Not a behavior change (nothing to change yet); a commitment baked into the accessor's
|
||||
definition. Pinning ADR 0011's "running task with `segments_active == 0`":
|
||||
|
||||
`tasks_starved` counts **only tasks whose task-state is `connecting` or `downloading` and
|
||||
whose `segments_active == 0`.** That is precisely "the segment allocator has not granted a
|
||||
slot to a task that is asking for one." Consequences:
|
||||
|
||||
| Task state | In `tasks_starved`? | Why |
|
||||
|---|---|---|
|
||||
| `probing` | no | uses the probe pool (ADR 0011 §5), not the segment budget |
|
||||
| `connecting`, `segments_active == 0` | **yes** | admitted + probed, waiting on the allocator's first grant — the real starvation case |
|
||||
| `connecting`/`downloading`, `segments_active ≥ 1` | no | a segment in its own `connecting` sub-state counts as a held slot (ADR 0011 A3) |
|
||||
| `downloading`, `segments_active == 0` | **yes** | held slots, lost them all (e.g. every segment failed and is being re-requested) — transient, still a real "allocator owes this task a slot" |
|
||||
| `retry_wait` | no | CORE's backoff timer holds it at zero *deliberately*; it is not asking the allocator for anything until it re-enters `connecting` |
|
||||
| `paused` (DAEMON- **or** CORE-initiated) | no | a paused task is not asking for a slot |
|
||||
| `new`, `queued`, `assembling`, `verifying`, `complete`, `failed`, `cancelled` | no | not in the `{connecting, downloading}` set |
|
||||
|
||||
So `retry_wait` and auto-paused tasks fall outside `tasks_starved` **because they are not
|
||||
in the counted state set**, not because of a special case that could rot. `starved_tasks()`
|
||||
returns exactly the TaskIds in that count; `starved_since(id)` is defined only for them.
|
||||
|
||||
A `retry_wait` task still counts against `connection.maxConcurrentDownloads` from DAEMON's
|
||||
side (it is running, not requeued) and contributes nothing to the allocator's guarantee
|
||||
pass until it re-enters `connecting` — matches §4 exactly.
|
||||
|
||||
### 2 — `pause()` is idempotent (committed)
|
||||
|
||||
- `pause(TaskId)` on a task already in `paused` (however it got there) → **no-op, returns
|
||||
success**. DAEMON never needs to know CORE auto-paused first.
|
||||
- `pause(TaskId)` on a task in a terminal state (`complete`/`failed`/`cancelled`) →
|
||||
**no-op, returns success** — a pause racing a completion is not an error.
|
||||
- `resume(TaskId)` on a task that is not paused → **no-op, returns success**.
|
||||
- The only error `pause()`/`resume()` return is `task_not_found`.
|
||||
- CORE does not report a state-change event for a no-op pause (nothing changed). DAEMON
|
||||
reads the resulting state via the normal state-change callback / `TaskDetail`, not from
|
||||
`pause()`'s return.
|
||||
- If CORE is mid-transition to `paused` (its own auto-pause) when DAEMON's `pause()`
|
||||
arrives, the task ends up `paused` once, with one state-change event.
|
||||
|
||||
### 3 — PROTO: `error`-on-`paused` widening
|
||||
|
||||
Not CORE's to land, but CORE confirms its half: when CORE auto-pauses, it reports the
|
||||
transition through the same state-change callback every CORE transition uses, with the
|
||||
`ErrorInfo` populated — `auth_required` (401/407), `server_file_changed`, `disk_full`,
|
||||
`path_rejected` for a mid-run destination failure. Those are the `vdm::Error` values from
|
||||
the B1 taxonomy (`core/docs/proto-requests-m1.md`), already implemented in
|
||||
`core/include/vdm/util/error.hpp`. **CORE is ready**; PROTO only needs to permit `error` to
|
||||
be present on `event.task.state` when `state == "paused"`, and DAEMON to project CORE's
|
||||
`ErrorInfo` onto it. `error: null` on a DAEMON-initiated pause is correct and sufficient.
|
||||
|
||||
Until PROTO lands it, DAEMON cannot implement §3's cross-reason resume rule
|
||||
correctness-preservingly — agree it should not guess from timing.
|
||||
|
||||
### 4 — "auto-pause" naming
|
||||
|
||||
CORE has no established internal term (the state machine is unbuilt). **CORE adopts
|
||||
"auto-pause"** for the informal concept. On the wire and in the API there is no new term:
|
||||
the discriminator is `state == paused` plus the `Error` code (present ⇒ CORE-initiated,
|
||||
absent ⇒ DAEMON-initiated), exactly as the ADR's §2 and the rejected-alternatives section
|
||||
describe.
|
||||
|
||||
---
|
||||
|
||||
## Two notes back to DAEMON (not objections)
|
||||
|
||||
- **§1 table, `paused` from `assembling`/`verifying`:** the *state transition* is honoured
|
||||
from any CORE state as the table says. The *work* interruption is best-effort: a pause
|
||||
during `verifying` discards the in-progress hash and re-hashes from the start of the
|
||||
file on resume (cheap, bounded); a pause during `assembling` (HLS/DASH mux) is an M4
|
||||
concern and may not be cleanly interruptible mid-mux. Neither affects M1 or the API
|
||||
shape.
|
||||
- **§5 restart:** CORE agrees fully. CORE holds no persistent state; `start(TaskId)`
|
||||
transparently checks for a valid `.veloxpart.meta` sidecar (stage 5), re-validates with
|
||||
`If-Range` (`docs/04` §5), and either resumes from the recorded offsets or restarts if
|
||||
the validator failed / the sidecar is corrupt. DAEMON does nothing special on restart
|
||||
beyond rewriting CORE-owned states to `queued` and re-admitting — which is what §5 says.
|
||||
@@ -0,0 +1,186 @@
|
||||
# CORE → DAEMON — the engine API (M1)
|
||||
|
||||
**Status: reviewed and signed off** by DAEMON on 2026-09-10
|
||||
(`daemon/docs/engine-api-review.md`). Nothing forced a `sched/` or dispatch rewrite. The
|
||||
five open questions are resolved at the bottom; the review's four confirms are folded into
|
||||
the headers (`sha512` added to `Checksum::Algo`; the parent-directory contract made
|
||||
explicit; `cancel()` ordering documented; `rate_limiter()` accessor added).
|
||||
|
||||
Integration order: DAEMON wires this in after `daemon/src/sched/` lands (the scheduler is
|
||||
what calls `start`/`pause`/`resume`/`cancel` and drives `set_task_order`); `sched/` builds
|
||||
against these headers in parallel with CORE stage 8 and does not need the bodies.
|
||||
|
||||
---
|
||||
|
||||
This is the `libveloxcore` public surface `veloxd` links and calls
|
||||
to actually run a download. It is the thing the AGENT-CORE brief asked for on day one and
|
||||
that slipped: DAEMON has an RPC surface and a store and, until this is agreed, nothing to
|
||||
call. Sketch headers: `core/include/vdm/engine.hpp`, `core/include/vdm/task/download.hpp`
|
||||
(both compile now; `Engine` / `DownloadHandle` bodies land in CORE stage 8). Nothing here
|
||||
touches `contracts/` — DAEMON projects these callbacks onto `TaskSummary` / `TaskDetail` /
|
||||
`event.*`.
|
||||
|
||||
Please review the field semantics, the threading/lifetime rules, and the
|
||||
pause/resume/cancel contract, and raise anything that would force a `daemon/src/sched/` or
|
||||
RPC-dispatch rewrite later. Open questions are at the bottom.
|
||||
|
||||
---
|
||||
|
||||
## 1. `DownloadSpec` — what DAEMON hands the engine
|
||||
|
||||
DAEMON has already run the rules engine, picked the category folder, canonicalised the
|
||||
path and checked it against the allowed roots (`-32011` is DAEMON's error, raised before
|
||||
`start()`), and resolved the filename. The engine's spec is the concrete result.
|
||||
|
||||
| field | who fills it | notes |
|
||||
|---|---|---|
|
||||
| `url`, `mirrors` | DAEMON | `mirrors` are alternative URLs for the *same bytes*; the segmenter's requeue prefers a different host after 3 connection failures (docs/04 §3). |
|
||||
| `headers`, `cookies`, `referrer`, `user_agent` | DAEMON, verbatim from the capture | replayed on every segment request and on the probe. |
|
||||
| `save_path` | DAEMON | **absolute and final.** The engine never canonicalises or root-checks. `<save_path>.veloxpart` and `.veloxpart.meta` sit beside it; on success the part file is renamed in place. |
|
||||
| `segments` | user / `connection.maxSegmentsPerDownload` | requested upper bound 1–32; the engine lowers it to the per-host cap and to 1 for a non-resumable source. Effective value comes back in `Progress.effective_segments`. |
|
||||
| `buffer_bytes` | user / `connection.bufferBytes` | requested per-segment; silently reduced to fit `maxTotalBufferBytes` across all live segments. Effective value in `Progress.effective_buffer_bytes` → your `TaskDetail.effectiveBufferBytes`. |
|
||||
| `proxy`, `auth` | DAEMON | `auth` carries credentials only if known up front (Secret Service). Leave `scheme == none` to get an `on_auth_required` on a 401/407 instead. |
|
||||
| `checksum` | user (`download.add.checksum`) | `{algo, hex}`. Verified during `verifying`; a mismatch is a terminal `failed` with `Error::checksum_mismatch`. |
|
||||
| `probe_hint` | DAEMON | the `ProbeResult` you already got for the File Info dialog. Supplying it skips the engine's own probe — the task starts in `connecting`, not `probing`. The engine still revalidates with `If-Range` on resume. |
|
||||
| `allow_resume` | DAEMON | `true`: if a CRC-valid `.veloxpart.meta` sits beside `save_path`, resume from it. `false`: start fresh, overwrite. Your restart flow (ADR 0013 §5) sets this per task. |
|
||||
| `max_retries` | user / default 10 | per segment, before `failed` with `Error::max_retries_exhausted`. |
|
||||
|
||||
`start()` returns immediately. It never throws and never blocks on the network; a bad URL,
|
||||
DNS failure, or unwritable `save_path` is delivered through `on_finished`.
|
||||
|
||||
## 2. The state machine the engine drives
|
||||
|
||||
`EngineState` is the CORE-owned subset of the wire `TaskState` (ADR 0013 §1):
|
||||
|
||||
```
|
||||
probing ─▶ connecting ⇄ downloading ─▶ assembling ─▶ verifying ─▶ complete
|
||||
│ │ ▲ │ │ (M4 mux; a no-op rename in M1)
|
||||
│ │ └─ retry_wait ┘
|
||||
▼ ▼
|
||||
(any) ──────▶ paused ──(resume)──▶ connecting
|
||||
(any CORE state) ──────────────────▶ failed (terminal, engine-initiated)
|
||||
(any state, on cancel()) ──────────▶ cancelled (terminal, DAEMON-initiated)
|
||||
```
|
||||
|
||||
`new` and `queued` are yours; the engine never emits them. `start()` corresponds to your
|
||||
`queued → probing`. Every transition is reported through `on_state(from, to, error?)`,
|
||||
including the auto-pauses (§4) and the terminals. `previousState` on your
|
||||
`event.task.state` maps straight from the `from` argument.
|
||||
|
||||
## 3. Threading and lifetime
|
||||
|
||||
- **Callbacks run on an engine thread** — a transfer worker, the progress timer, or a
|
||||
dispatch thread — **never** the thread that called `start()` / `pause()` / etc.
|
||||
- **Per task, callbacks are serialised.** You will never get two callbacks for the same
|
||||
handle at once. Across tasks they run concurrently.
|
||||
- **A callback must not block.** It runs on a thread doing real transfer work; a slow
|
||||
callback stalls that work. Hand off to your own queue/loop.
|
||||
- **A callback must not re-enter the same handle synchronously** — no `pause()` from
|
||||
inside `on_state`, etc. Post it. (Calling into a *different* handle, or into
|
||||
`segment_budget()`, is fine.)
|
||||
- **`on_finished` is always the last callback** for a task. After it returns the engine
|
||||
makes no further callbacks for that handle and the handle's control methods are no-ops.
|
||||
- **The handle is copyable and thread-safe.** Dropping the last copy does **not** cancel —
|
||||
the task runs on. Call `cancel()` to stop it. (DAEMON holds the handle for the task's
|
||||
life anyway.)
|
||||
- **`Engine` must outlive every handle.** `~Engine()` cancels all running tasks and joins
|
||||
their workers before returning — expect it to block briefly.
|
||||
- **Logging**: the engine writes through `vdm::set_log_sink()` (a `core/util` global).
|
||||
Install your sink once at startup; the engine never opens a file itself.
|
||||
|
||||
## 4. `paused` is shared, and idempotency is the contract (ADR 0013 §2)
|
||||
|
||||
Both sides put a task in `paused`, for disjoint reasons:
|
||||
|
||||
- **DAEMON-initiated**: `handle.pause()` — user pause, a schedule window closing, a queue
|
||||
stop, `Queue.onComplete`, the admission governor reconciling a lowered
|
||||
`maxActiveSegments`.
|
||||
- **Engine auto-pause**: `on_auth_required` (401/407), `on_decision_needed`
|
||||
(`server_file_changed` / stale range), disk full. The engine transitions to `paused`
|
||||
on its own and fires `on_state(_, paused, ErrorInfo{...})` — the same path as any other
|
||||
transition. `ErrorInfo.code` present ⇒ engine-initiated; absent ⇒ you did it. That is
|
||||
the only discriminator, and it is what your `error`-on-`paused` widening (open item 3 of
|
||||
ADR 0013) carries on the wire.
|
||||
|
||||
**Idempotency, now a signature:**
|
||||
|
||||
| call | already in that state / terminal | otherwise |
|
||||
|---|---|---|
|
||||
| `pause()` | no-op, no error | stop new segment requests, flush + `fdatasync` in-flight buffers, write `.veloxpart.meta`, release the budget slots, `on_state(_, paused, nullopt)`. Bounded by the slowest in-flight flush. |
|
||||
| `resume()` | no-op if not `paused`; no-op if terminal | revalidate with `If-Range`, re-acquire budget slots, `paused → connecting`, resume from the sidecar offsets. |
|
||||
| `cancel(discard_partial)` | no-op if already terminal | stop everything, `on_state(_, cancelled, nullopt)`, `on_finished(Err{cancelled})`. `discard_partial` also unlinks `.veloxpart[.meta]` — wire this to `download.remove {deleteFile}`. |
|
||||
|
||||
`resume()` after an auto-pause for `auth_required` **without** a preceding
|
||||
`provide_auth()` is a no-op — the task stays paused. This is ADR 0013 §3's "resume must
|
||||
not cross reasons", enforced on CORE's side: the scheduler cannot accidentally un-pause a
|
||||
task waiting on credentials.
|
||||
|
||||
- **`provide_auth(user, pass, remember)`** — acts only while the task is auto-paused for
|
||||
auth; supplies the credential for the retry and resumes. `remember` asks *you* to
|
||||
persist to the Secret Service; the engine never stores it. No-op otherwise.
|
||||
- **`decide(Decision)`** — acts only while auto-paused for a decision. `restart` discards
|
||||
the partial and re-downloads; `keep_partial` continues against what is on disk (the
|
||||
user's stated risk); `abort` → `failed`. No-op otherwise.
|
||||
- **`refresh_url(url, headers)`** — IDM's "Refresh Download Address": the engine re-probes
|
||||
the new URL to validate it, then **restarts every segment on it** (the signed-URL-expiry
|
||||
case the wire method exists for), keeping the bytes already on disk. Mirror rotation is
|
||||
*not* this — that is `spec.mirrors` + the segmenter's requeue-to-a-different-host.
|
||||
- **`on_decision_needed`** is only for the cases the engine cannot resolve itself. A
|
||||
routine 416 / stale range is the engine's own re-probe + re-split loop; it escalates
|
||||
`DecisionRequest{range_metadata_stale}` **only when that loop fails**, at which point
|
||||
"retry the same range" is already exhausted — so `{restart, keep_partial, abort}` is the
|
||||
whole choice set.
|
||||
|
||||
## 5. Progress
|
||||
|
||||
`on_progress` is coalesced to **≤ 4 Hz per task** inside the engine — the same cadence as
|
||||
`event.task.progress`, so your batcher can forward without re-throttling. It carries
|
||||
aggregate `downloaded` / `speed_bps` / `eta_seconds`, the effective segment count and
|
||||
buffer size, and a `SegmentProgress[]` (index, inclusive `[start,end]`, `completed`,
|
||||
per-segment speed, state) for the GUI's segment bars. `total` is absent for a chunked
|
||||
source until the stream ends.
|
||||
|
||||
`handle.state()` and `handle.progress()` are synchronous lock-guarded snapshots for
|
||||
`download.get` / `download.list` — call them any time, including from your RPC thread.
|
||||
|
||||
## 6. What the engine does NOT do
|
||||
|
||||
- No filename resolution, no category matching, no path canonicalisation, no allowed-root
|
||||
check — all DAEMON, before `start()`.
|
||||
- No queueing, scheduling, priority, or "when queue completes" — DAEMON, via
|
||||
`segment_budget().set_task_order()` and by choosing when to call `start()` / `pause()`.
|
||||
- No persistence beyond `.veloxpart.meta`. On a daemon restart the engine knows nothing;
|
||||
you reload from SQLite, rewrite CORE-owned states to `queued`, and re-`start()` with
|
||||
`allow_resume = true` (ADR 0013 §5).
|
||||
- No credential storage. Ever (`CLAUDE.md` §4).
|
||||
|
||||
---
|
||||
|
||||
## Resolved (DAEMON review, 2026-09-10)
|
||||
|
||||
1. **`probe_hint` stays optional.** DAEMON has a `ProbeResult` only on the File-Info path;
|
||||
capture-take, `velox add`, `addBatch` and restart have none. The engine probes when
|
||||
it's absent.
|
||||
2. **One `cancel(discard_partial)`.** `download.cancel` = `cancel(false)`;
|
||||
`download.remove` = `cancel(true)` for a live task (plus DAEMON's row/file cleanup), or
|
||||
pure DAEMON-side for an already-terminal one. No `handle.remove()`.
|
||||
3. **`{restart, keep_partial, abort}` is the whole set.** The engine owns the routine
|
||||
416 re-probe/re-split and only escalates `on_decision_needed` when that loop fails —
|
||||
"retry the same range" is exhausted by then.
|
||||
4. **Per-task 4 Hz is fine.** DAEMON coalesces across tasks for `event.task.progress`
|
||||
regardless; `on_progress_batch` is a nice-to-have and must not block stage 8.
|
||||
5. **`refresh_url` restarts all segments on the new URL** after a validating re-probe (the
|
||||
signed-URL case). Mirror rotation is `spec.mirrors` + the segmenter, not this.
|
||||
|
||||
## Review confirms, folded in
|
||||
|
||||
- **(a)** `vdm::TaskId` is a cheap-copy hashable value; DAEMON never constructs one — it
|
||||
only receives it from `start()` / callbacks and passes it back to `set_task_order()`
|
||||
etc. ✔ (`vdm/ids.hpp`)
|
||||
- **(b)** DAEMON `mkdir -p`s `save_path`'s parent before `start()`. The engine opens the
|
||||
file and fails with `Error::path_rejected` if the directory is missing. ✔ (documented on
|
||||
`DownloadSpec`)
|
||||
- **(c)** `Checksum::Algo` now has `sha512`, matching the wire `Checksum` set. ✔
|
||||
- **(d)** `cancel()` always fires `on_state(_, cancelled, nullopt)` then
|
||||
`on_finished(Err{Error::canceled})` (note: the taxonomy value is `canceled`), in that
|
||||
order. ✔ (documented on `DownloadHandle::cancel`)
|
||||
@@ -0,0 +1,69 @@
|
||||
# CORE → PROTO — `tests/conformance/cpp/CMakeLists.txt` should link `veloxproto` now
|
||||
|
||||
Status: **open**. Small, mechanical. Filed rather than fixed because `tests/conformance/`
|
||||
is PROTO's lane.
|
||||
|
||||
## What's stale
|
||||
|
||||
`tests/conformance/cpp/CMakeLists.txt` says in its header comment:
|
||||
|
||||
> Links libveloxproto (the generated protocol code in core/generated/), not libveloxcore
|
||||
|
||||
…but it actually **compiles `core/generated/velox_proto.cpp` straight into the
|
||||
executable** and finds `nlohmann_json` itself:
|
||||
|
||||
```cmake
|
||||
add_executable(velox_conformance_cpp
|
||||
conformance_main.cpp
|
||||
${CMAKE_SOURCE_DIR}/core/generated/velox_proto.cpp)
|
||||
|
||||
target_include_directories(velox_conformance_cpp PRIVATE
|
||||
${CMAKE_SOURCE_DIR}/core/generated
|
||||
${CMAKE_CURRENT_SOURCE_DIR})
|
||||
target_link_libraries(velox_conformance_cpp PRIVATE nlohmann_json::nlohmann_json)
|
||||
```
|
||||
|
||||
That was the only option while ADR 0009's `libveloxproto` target didn't exist. **It exists
|
||||
now** — `core/CMakeLists.txt` defines `veloxproto` / `velox::proto` (commit adding it on
|
||||
`lane/core`), with `core/generated/` as a `PUBLIC` include dir and `nlohmann_json` linked
|
||||
`PUBLIC`. The comment and the code now agree only if the runner links the target.
|
||||
|
||||
## Requested change
|
||||
|
||||
```cmake
|
||||
if(TARGET velox::proto)
|
||||
add_executable(velox_conformance_cpp conformance_main.cpp)
|
||||
target_link_libraries(velox_conformance_cpp PRIVATE velox::proto)
|
||||
else()
|
||||
# Standalone configure of tests/conformance/ (no core/ in the tree): fall back to
|
||||
# compiling the generated source directly, as today.
|
||||
if(NOT TARGET nlohmann_json::nlohmann_json)
|
||||
find_package(nlohmann_json 3.11 REQUIRED)
|
||||
endif()
|
||||
add_executable(velox_conformance_cpp
|
||||
conformance_main.cpp
|
||||
${CMAKE_SOURCE_DIR}/core/generated/velox_proto.cpp)
|
||||
target_include_directories(velox_conformance_cpp PRIVATE
|
||||
${CMAKE_SOURCE_DIR}/core/generated)
|
||||
target_link_libraries(velox_conformance_cpp PRIVATE nlohmann_json::nlohmann_json)
|
||||
endif()
|
||||
|
||||
target_include_directories(velox_conformance_cpp PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
|
||||
target_compile_features(velox_conformance_cpp PRIVATE cxx_std_23)
|
||||
add_test(NAME conformance_cpp COMMAND velox_conformance_cpp ${CMAKE_SOURCE_DIR})
|
||||
set_tests_properties(conformance_cpp PROPERTIES LABELS "conformance")
|
||||
```
|
||||
|
||||
The `if(TARGET ...)` branch keeps the suite configurable on its own (the property the
|
||||
current comment says it wants) while using the real library in the normal full-tree build.
|
||||
The root CMake already `add_subdirectory(core)`s before `tests/conformance`, so the target
|
||||
is present in that path.
|
||||
|
||||
## Why it matters beyond tidiness
|
||||
|
||||
GUI is blocked on `libveloxproto` being a real link target (it can't `add_subdirectory` a
|
||||
sibling lane's `core/generated/` and re-guess the nlohmann find). Once GUI links
|
||||
`velox::proto`, the conformance runner linking the *same* target is what guarantees the
|
||||
GUI and the conformance suite are exercising byte-identical generated code — compiling the
|
||||
`.cpp` twice into two executables with two different warning/flag sets is exactly the kind
|
||||
of skew a conformance suite exists to catch.
|
||||
+175
-85
@@ -3,7 +3,7 @@
|
||||
//
|
||||
// Source: contracts/schema/**
|
||||
// Generator: contracts/codegen/gen_cpp.py
|
||||
// Contract: v1.0.0
|
||||
// Contract: v1.4.0
|
||||
//
|
||||
// Hand-editing this file is a merge blocker. Fix the schema and regenerate:
|
||||
// python3 contracts/codegen/gen_cpp.py
|
||||
@@ -401,6 +401,8 @@ std::string_view to_string(SettingKey v) noexcept {
|
||||
case SettingKey::ConnectionPreset: return "connection.preset";
|
||||
case SettingKey::ConnectionMaxSegmentsPerDownload: return "connection.maxSegmentsPerDownload";
|
||||
case SettingKey::ConnectionBufferBytes: return "connection.bufferBytes";
|
||||
case SettingKey::ConnectionMaxTotalBufferBytes: return "connection.maxTotalBufferBytes";
|
||||
case SettingKey::ConnectionMaxActiveSegments: return "connection.maxActiveSegments";
|
||||
case SettingKey::ConnectionMaxConcurrentDownloads: return "connection.maxConcurrentDownloads";
|
||||
case SettingKey::ConnectionTimeoutSec: return "connection.timeoutSec";
|
||||
case SettingKey::ConnectionMaxRetries: return "connection.maxRetries";
|
||||
@@ -447,6 +449,8 @@ Result<SettingKey> parse_SettingKey(std::string_view s) {
|
||||
if (s == "connection.preset") return SettingKey::ConnectionPreset;
|
||||
if (s == "connection.maxSegmentsPerDownload") return SettingKey::ConnectionMaxSegmentsPerDownload;
|
||||
if (s == "connection.bufferBytes") return SettingKey::ConnectionBufferBytes;
|
||||
if (s == "connection.maxTotalBufferBytes") return SettingKey::ConnectionMaxTotalBufferBytes;
|
||||
if (s == "connection.maxActiveSegments") return SettingKey::ConnectionMaxActiveSegments;
|
||||
if (s == "connection.maxConcurrentDownloads") return SettingKey::ConnectionMaxConcurrentDownloads;
|
||||
if (s == "connection.timeoutSec") return SettingKey::ConnectionTimeoutSec;
|
||||
if (s == "connection.maxRetries") return SettingKey::ConnectionMaxRetries;
|
||||
@@ -1601,8 +1605,8 @@ template <> Result<DownloadSpec> parse<DownloadSpec>(const nlohmann::json& j, st
|
||||
if (it != j.end() && !it->is_null()) {
|
||||
if (!(*it).is_number_integer()) return std::unexpected(ParseError{std::string(fp), "expected an integer"});
|
||||
auto val = (*it).get<std::int64_t>();
|
||||
if (val < 4096) return std::unexpected(ParseError{std::string(fp), "value is below the minimum of 4096"});
|
||||
if (val > 8388608) return std::unexpected(ParseError{std::string(fp), "value is above the maximum of 8388608"});
|
||||
if (val < 65536) return std::unexpected(ParseError{std::string(fp), "value is below the minimum of 65536"});
|
||||
if (val > 16777216) return std::unexpected(ParseError{std::string(fp), "value is above the maximum of 16777216"});
|
||||
out.bufferBytes = std::move(val);
|
||||
}
|
||||
}
|
||||
@@ -2428,6 +2432,8 @@ void to_json(nlohmann::json& j, const Settings& v) {
|
||||
if (v.sounds_onComplete.has_value()) j["sounds.onComplete"] = *v.sounds_onComplete;
|
||||
if (v.sounds_onQueueComplete.has_value()) j["sounds.onQueueComplete"] = *v.sounds_onQueueComplete;
|
||||
if (v.sounds_onError.has_value()) j["sounds.onError"] = *v.sounds_onError;
|
||||
if (v.connection_maxTotalBufferBytes.has_value()) j["connection.maxTotalBufferBytes"] = *v.connection_maxTotalBufferBytes;
|
||||
if (v.connection_maxActiveSegments.has_value()) j["connection.maxActiveSegments"] = *v.connection_maxActiveSegments;
|
||||
}
|
||||
|
||||
template <> Result<Settings> parse<Settings>(const nlohmann::json& j, std::string_view path) {
|
||||
@@ -2660,8 +2666,8 @@ template <> Result<Settings> parse<Settings>(const nlohmann::json& j, std::strin
|
||||
if (it != j.end() && !it->is_null()) {
|
||||
if (!(*it).is_number_integer()) return std::unexpected(ParseError{std::string(fp), "expected an integer"});
|
||||
auto val = (*it).get<std::int64_t>();
|
||||
if (val < 4096) return std::unexpected(ParseError{std::string(fp), "value is below the minimum of 4096"});
|
||||
if (val > 8388608) return std::unexpected(ParseError{std::string(fp), "value is above the maximum of 8388608"});
|
||||
if (val < 65536) return std::unexpected(ParseError{std::string(fp), "value is below the minimum of 65536"});
|
||||
if (val > 16777216) return std::unexpected(ParseError{std::string(fp), "value is above the maximum of 16777216"});
|
||||
out.connection_bufferBytes = std::move(val);
|
||||
}
|
||||
}
|
||||
@@ -2865,6 +2871,28 @@ template <> Result<Settings> parse<Settings>(const nlohmann::json& j, std::strin
|
||||
out.sounds_onError = std::move(val);
|
||||
}
|
||||
}
|
||||
{
|
||||
const std::string fp = join(path, "connection.maxTotalBufferBytes");
|
||||
const auto it = j.find("connection.maxTotalBufferBytes");
|
||||
if (it != j.end() && !it->is_null()) {
|
||||
if (!(*it).is_number_integer()) return std::unexpected(ParseError{std::string(fp), "expected an integer"});
|
||||
auto val = (*it).get<std::int64_t>();
|
||||
if (val < 16777216) return std::unexpected(ParseError{std::string(fp), "value is below the minimum of 16777216"});
|
||||
if (val > 2147483648) return std::unexpected(ParseError{std::string(fp), "value is above the maximum of 2147483648"});
|
||||
out.connection_maxTotalBufferBytes = std::move(val);
|
||||
}
|
||||
}
|
||||
{
|
||||
const std::string fp = join(path, "connection.maxActiveSegments");
|
||||
const auto it = j.find("connection.maxActiveSegments");
|
||||
if (it != j.end() && !it->is_null()) {
|
||||
if (!(*it).is_number_integer()) return std::unexpected(ParseError{std::string(fp), "expected an integer"});
|
||||
auto val = (*it).get<std::int64_t>();
|
||||
if (val < 1) return std::unexpected(ParseError{std::string(fp), "value is below the minimum of 1"});
|
||||
if (val > 256) return std::unexpected(ParseError{std::string(fp), "value is above the maximum of 256"});
|
||||
out.connection_maxActiveSegments = std::move(val);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
@@ -3183,6 +3211,7 @@ void to_json(nlohmann::json& j, const TaskDetail& v) {
|
||||
if (v.userAgent.has_value()) j["userAgent"] = *v.userAgent;
|
||||
if (v.mime.has_value()) j["mime"] = *v.mime;
|
||||
if (v.bufferBytes.has_value()) j["bufferBytes"] = *v.bufferBytes;
|
||||
if (v.effectiveBufferBytes.has_value()) j["effectiveBufferBytes"] = *v.effectiveBufferBytes;
|
||||
if (v.partPath.has_value()) j["partPath"] = *v.partPath;
|
||||
if (v.checksum.has_value()) j["checksum"] = *v.checksum;
|
||||
if (v.checksumVerified.has_value()) j["checksumVerified"] = *v.checksumVerified;
|
||||
@@ -3264,11 +3293,22 @@ template <> Result<TaskDetail> parse<TaskDetail>(const nlohmann::json& j, std::s
|
||||
if (it != j.end() && !it->is_null()) {
|
||||
if (!(*it).is_number_integer()) return std::unexpected(ParseError{std::string(fp), "expected an integer"});
|
||||
auto val = (*it).get<std::int64_t>();
|
||||
if (val < 4096) return std::unexpected(ParseError{std::string(fp), "value is below the minimum of 4096"});
|
||||
if (val > 8388608) return std::unexpected(ParseError{std::string(fp), "value is above the maximum of 8388608"});
|
||||
if (val < 65536) return std::unexpected(ParseError{std::string(fp), "value is below the minimum of 65536"});
|
||||
if (val > 16777216) return std::unexpected(ParseError{std::string(fp), "value is above the maximum of 16777216"});
|
||||
out.bufferBytes = std::move(val);
|
||||
}
|
||||
}
|
||||
{
|
||||
const std::string fp = join(path, "effectiveBufferBytes");
|
||||
const auto it = j.find("effectiveBufferBytes");
|
||||
if (it != j.end() && !it->is_null()) {
|
||||
if (!(*it).is_number_integer()) return std::unexpected(ParseError{std::string(fp), "expected an integer"});
|
||||
auto val = (*it).get<std::int64_t>();
|
||||
if (val < 65536) return std::unexpected(ParseError{std::string(fp), "value is below the minimum of 65536"});
|
||||
if (val > 16777216) return std::unexpected(ParseError{std::string(fp), "value is above the maximum of 16777216"});
|
||||
out.effectiveBufferBytes = std::move(val);
|
||||
}
|
||||
}
|
||||
{
|
||||
const std::string fp = join(path, "partPath");
|
||||
const auto it = j.find("partPath");
|
||||
@@ -4329,6 +4369,78 @@ template <> Result<DownloadProbeResult> parse<DownloadProbeResult>(const nlohman
|
||||
return out;
|
||||
}
|
||||
|
||||
void to_json(nlohmann::json& j, const DownloadProvideAuthParams& v) {
|
||||
j = nlohmann::json::object();
|
||||
j["taskId"] = v.taskId;
|
||||
j["username"] = v.username;
|
||||
j["password"] = v.password;
|
||||
if (v.save.has_value()) j["save"] = *v.save;
|
||||
}
|
||||
|
||||
template <> Result<DownloadProvideAuthParams> parse<DownloadProvideAuthParams>(const nlohmann::json& j, std::string_view path) {
|
||||
if (!j.is_object()) return std::unexpected(ParseError{std::string(path), "expected an object"});
|
||||
DownloadProvideAuthParams out;
|
||||
{
|
||||
const std::string fp = join(path, "taskId");
|
||||
const auto it = j.find("taskId");
|
||||
if (it == j.end() || it->is_null())
|
||||
return std::unexpected(ParseError{fp, "required field is missing"});
|
||||
if (!(*it).is_string()) return std::unexpected(ParseError{std::string(fp), "expected a string"});
|
||||
auto val = (*it).get<std::string>();
|
||||
out.taskId = std::move(val);
|
||||
}
|
||||
{
|
||||
const std::string fp = join(path, "username");
|
||||
const auto it = j.find("username");
|
||||
if (it == j.end() || it->is_null())
|
||||
return std::unexpected(ParseError{fp, "required field is missing"});
|
||||
if (!(*it).is_string()) return std::unexpected(ParseError{std::string(fp), "expected a string"});
|
||||
auto val = (*it).get<std::string>();
|
||||
if (val.size() > 256u) return std::unexpected(ParseError{std::string(fp), "value is longer than 256 characters"});
|
||||
out.username = std::move(val);
|
||||
}
|
||||
{
|
||||
const std::string fp = join(path, "password");
|
||||
const auto it = j.find("password");
|
||||
if (it == j.end() || it->is_null())
|
||||
return std::unexpected(ParseError{fp, "required field is missing"});
|
||||
if (!(*it).is_string()) return std::unexpected(ParseError{std::string(fp), "expected a string"});
|
||||
auto val = (*it).get<std::string>();
|
||||
if (val.size() > 1024u) return std::unexpected(ParseError{std::string(fp), "value is longer than 1024 characters"});
|
||||
out.password = std::move(val);
|
||||
}
|
||||
{
|
||||
const std::string fp = join(path, "save");
|
||||
const auto it = j.find("save");
|
||||
if (it != j.end() && !it->is_null()) {
|
||||
if (!(*it).is_boolean()) return std::unexpected(ParseError{std::string(fp), "expected a boolean"});
|
||||
auto val = (*it).get<bool>();
|
||||
out.save = std::move(val);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
void to_json(nlohmann::json& j, const DownloadProvideAuthResult& v) {
|
||||
j = nlohmann::json::object();
|
||||
j["ok"] = v.ok;
|
||||
}
|
||||
|
||||
template <> Result<DownloadProvideAuthResult> parse<DownloadProvideAuthResult>(const nlohmann::json& j, std::string_view path) {
|
||||
if (!j.is_object()) return std::unexpected(ParseError{std::string(path), "expected an object"});
|
||||
DownloadProvideAuthResult out;
|
||||
{
|
||||
const std::string fp = join(path, "ok");
|
||||
const auto it = j.find("ok");
|
||||
if (it == j.end() || it->is_null())
|
||||
return std::unexpected(ParseError{fp, "required field is missing"});
|
||||
if (!(*it).is_boolean()) return std::unexpected(ParseError{std::string(fp), "expected a boolean"});
|
||||
auto val = (*it).get<bool>();
|
||||
out.ok = std::move(val);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
void to_json(nlohmann::json& j, const DownloadRefreshUrlParams& v) {
|
||||
j = nlohmann::json::object();
|
||||
j["taskId"] = v.taskId;
|
||||
@@ -4711,8 +4823,8 @@ template <> Result<DownloadUpdateParamsPatch> parse<DownloadUpdateParamsPatch>(c
|
||||
if (it != j.end() && !it->is_null()) {
|
||||
if (!(*it).is_number_integer()) return std::unexpected(ParseError{std::string(fp), "expected an integer"});
|
||||
auto val = (*it).get<std::int64_t>();
|
||||
if (val < 4096) return std::unexpected(ParseError{std::string(fp), "value is below the minimum of 4096"});
|
||||
if (val > 8388608) return std::unexpected(ParseError{std::string(fp), "value is above the maximum of 8388608"});
|
||||
if (val < 65536) return std::unexpected(ParseError{std::string(fp), "value is below the minimum of 65536"});
|
||||
if (val > 16777216) return std::unexpected(ParseError{std::string(fp), "value is above the maximum of 16777216"});
|
||||
out.bufferBytes = std::move(val);
|
||||
}
|
||||
}
|
||||
@@ -6855,6 +6967,7 @@ std::string_view to_string(Method m) noexcept {
|
||||
case Method::DownloadList: return "download.list";
|
||||
case Method::DownloadPause: return "download.pause";
|
||||
case Method::DownloadProbe: return "download.probe";
|
||||
case Method::DownloadProvideAuth: return "download.provideAuth";
|
||||
case Method::DownloadRefreshUrl: return "download.refreshUrl";
|
||||
case Method::DownloadRemove: return "download.remove";
|
||||
case Method::DownloadResume: return "download.resume";
|
||||
@@ -6898,6 +7011,7 @@ std::optional<Method> method_from_string(std::string_view s) noexcept {
|
||||
if (s == "download.list") return Method::DownloadList;
|
||||
if (s == "download.pause") return Method::DownloadPause;
|
||||
if (s == "download.probe") return Method::DownloadProbe;
|
||||
if (s == "download.provideAuth") return Method::DownloadProvideAuth;
|
||||
if (s == "download.refreshUrl") return Method::DownloadRefreshUrl;
|
||||
if (s == "download.remove") return Method::DownloadRemove;
|
||||
if (s == "download.resume") return Method::DownloadResume;
|
||||
@@ -6941,6 +7055,7 @@ bool is_privileged(Method m) noexcept {
|
||||
case Method::DownloadList: return false;
|
||||
case Method::DownloadPause: return false;
|
||||
case Method::DownloadProbe: return false;
|
||||
case Method::DownloadProvideAuth: return true;
|
||||
case Method::DownloadRefreshUrl: return false;
|
||||
case Method::DownloadRemove: return true;
|
||||
case Method::DownloadResume: return false;
|
||||
@@ -6985,6 +7100,7 @@ bool is_allowed_on(Method m, Transport t) noexcept {
|
||||
case Method::DownloadList: return t == Transport::Uds ? true : true;
|
||||
case Method::DownloadPause: return t == Transport::Uds ? true : true;
|
||||
case Method::DownloadProbe: return t == Transport::Uds ? true : true;
|
||||
case Method::DownloadProvideAuth: return t == Transport::Uds ? true : false;
|
||||
case Method::DownloadRefreshUrl: return t == Transport::Uds ? true : true;
|
||||
case Method::DownloadRemove: return t == Transport::Uds ? true : false;
|
||||
case Method::DownloadResume: return t == Transport::Uds ? true : true;
|
||||
@@ -7029,6 +7145,7 @@ std::int32_t deadline_ms(Method m) noexcept {
|
||||
case Method::DownloadList: return 5000;
|
||||
case Method::DownloadPause: return 5000;
|
||||
case Method::DownloadProbe: return 30000;
|
||||
case Method::DownloadProvideAuth: return 5000;
|
||||
case Method::DownloadRefreshUrl: return 30000;
|
||||
case Method::DownloadRemove: return 10000;
|
||||
case Method::DownloadResume: return 5000;
|
||||
@@ -7128,8 +7245,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_capture_getRules(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7140,8 +7256,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_capture_offer(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7152,8 +7267,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_category_list(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7164,8 +7278,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_category_remove(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7176,8 +7289,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_category_upsert(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7188,8 +7300,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_download_add(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7200,8 +7311,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_download_addBatch(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7212,8 +7322,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_download_cancel(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7224,8 +7333,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_download_get(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7236,8 +7344,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_download_list(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7248,8 +7355,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_download_pause(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7260,8 +7366,18 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_download_probe(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
case Method::DownloadProvideAuth: {
|
||||
auto p = parse<DownloadProvideAuthParams>(params, "params");
|
||||
if (!p)
|
||||
return make_error(id, ErrorCode::InvalidParams, p.error().message,
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_download_provideAuth(*p);
|
||||
if (!r)
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7272,8 +7388,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_download_refreshUrl(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7284,8 +7399,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_download_remove(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7296,8 +7410,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_download_resume(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7308,8 +7421,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_download_start(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7320,8 +7432,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_download_update(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7332,8 +7443,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_grabber_harvest(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7344,8 +7454,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_grabber_start(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7356,8 +7465,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_grabber_status(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7368,8 +7476,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_limiter_get(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7380,8 +7487,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_limiter_set(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7392,8 +7498,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_media_addVariant(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7404,8 +7509,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_media_listVariants(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7416,8 +7520,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_queue_list(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7428,8 +7531,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_queue_reorder(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7440,8 +7542,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_queue_start(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7452,8 +7553,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_queue_stop(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7464,8 +7564,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_queue_upsert(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7476,8 +7575,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_rules_list(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7488,8 +7586,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_rules_upsert(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7500,8 +7597,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_schedule_get(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7512,8 +7608,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_schedule_set(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7524,8 +7619,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_session_hello(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7536,8 +7630,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_session_pair(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7548,8 +7641,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_session_subscribe(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7560,8 +7652,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_settings_get(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
@@ -7572,8 +7663,7 @@ nlohmann::json dispatch(Dispatcher& handler, Transport transport, const nlohmann
|
||||
nlohmann::json{{"path", p.error().path}});
|
||||
auto r = handler.on_settings_set(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
return make_error(id, r.error().code, r.error().message, r.error().data);
|
||||
nlohmann::json out = *r;
|
||||
return make_result(id, std::move(out));
|
||||
}
|
||||
|
||||
+138
-46
@@ -3,7 +3,7 @@
|
||||
//
|
||||
// Source: contracts/schema/**
|
||||
// Generator: contracts/codegen/gen_cpp.py
|
||||
// Contract: v1.0.0
|
||||
// Contract: v1.4.0
|
||||
//
|
||||
// Hand-editing this file is a merge blocker. Fix the schema and regenerate:
|
||||
// python3 contracts/codegen/gen_cpp.py
|
||||
@@ -27,7 +27,7 @@
|
||||
// docs/adr/0009-generated-protocol-library.md.
|
||||
namespace velox::proto {
|
||||
|
||||
inline constexpr std::string_view kProtocolVersion = "1.0.0";
|
||||
inline constexpr std::string_view kProtocolVersion = "1.4.0";
|
||||
|
||||
/// Why a payload could not be turned into a typed value. `path` is a JSON Pointer
|
||||
/// into the offending document, so a conformance failure names the exact field.
|
||||
@@ -219,6 +219,8 @@ enum class SettingKey {
|
||||
ConnectionPreset, // "connection.preset"
|
||||
ConnectionMaxSegmentsPerDownload, // "connection.maxSegmentsPerDownload"
|
||||
ConnectionBufferBytes, // "connection.bufferBytes"
|
||||
ConnectionMaxTotalBufferBytes, // "connection.maxTotalBufferBytes"
|
||||
ConnectionMaxActiveSegments, // "connection.maxActiveSegments"
|
||||
ConnectionMaxConcurrentDownloads, // "connection.maxConcurrentDownloads"
|
||||
ConnectionTimeoutSec, // "connection.timeoutSec"
|
||||
ConnectionMaxRetries, // "connection.maxRetries"
|
||||
@@ -531,10 +533,13 @@ struct DownloadSpec {
|
||||
std::optional<std::string> categoryId{};
|
||||
/// Required when startMode is 'queue'.
|
||||
std::optional<std::string> queueId{};
|
||||
/// The REQUESTED connection count. An upper bound, not a promise: the daemon lowers it to the
|
||||
/// The REQUESTED connection count. An upper bound, not a promise: the engine lowers it to the
|
||||
/// per-host cap, and to 1 when the source turns out not to be resumable. What is actually in
|
||||
/// use comes back as TaskSummary.segments. null means use connection.maxSegmentsPerDownload.
|
||||
std::optional<std::int64_t> segments{};
|
||||
/// Requested write buffer per segment, in bytes. null means use connection.bufferBytes. Default
|
||||
/// 1 MiB; range 64 KiB - 16 MiB. Silently reduced to fit connection.maxTotalBufferBytes across
|
||||
/// all live segments; the effective value is reported back as TaskDetail.effectiveBufferBytes.
|
||||
std::optional<std::int64_t> bufferBytes{};
|
||||
std::optional<StartMode> startMode{};
|
||||
std::optional<std::string> description{};
|
||||
@@ -708,6 +713,9 @@ struct Settings {
|
||||
std::optional<bool> saveTo_createSubfolderPerSite{};
|
||||
std::optional<SettingsConnectionPreset> connection_preset{};
|
||||
std::optional<std::int64_t> connection_maxSegmentsPerDownload{};
|
||||
/// Default per-segment write buffer, in bytes, when a task does not request its own. Default 1
|
||||
/// MiB (1048576); range 64 KiB - 16 MiB. This is the single biggest throughput knob and is
|
||||
/// exposed in Options -> Downloads -> 'Write buffer per connection'.
|
||||
std::optional<std::int64_t> connection_bufferBytes{};
|
||||
std::optional<std::int64_t> connection_maxConcurrentDownloads{};
|
||||
std::optional<std::int64_t> connection_timeoutSec{};
|
||||
@@ -729,11 +737,26 @@ struct Settings {
|
||||
std::optional<std::string> sounds_onComplete{};
|
||||
std::optional<std::string> sounds_onQueueComplete{};
|
||||
std::optional<std::string> sounds_onError{};
|
||||
/// Global cap on write-buffer memory across every live segment, in bytes. Default 128 MiB
|
||||
/// (134217728). Every live segment's buffer is reduced to fit maxTotalBufferBytes / (live
|
||||
/// segment count, capped at maxActiveSegments); the reduced value is reported per task as
|
||||
/// TaskDetail.effectiveBufferBytes. Exists so a burst of large downloads with a large
|
||||
/// per-segment buffer cannot exhaust memory.
|
||||
std::optional<std::int64_t> connection_maxTotalBufferBytes{};
|
||||
/// Global ceiling on segments actually transferring at once, across every task. Default 32.
|
||||
/// This is the real bound behind '20 active downloads': the rest of each download's segments
|
||||
/// queue rather than all dialling out simultaneously. DAEMON's scheduler needs this value to
|
||||
/// decide what to admit; CORE enforces it.
|
||||
std::optional<std::int64_t> connection_maxActiveSegments{};
|
||||
};
|
||||
|
||||
/// Why a task is in the failed or retry_wait state. Distinct from the JSON-RPC Error, which
|
||||
/// describes a failed call rather than a failed download — the two live in different code
|
||||
/// spaces on purpose, and `code` here is a TaskErrorCode string, never a JSON-RPC integer.
|
||||
/// Why a task is in the failed, retry_wait, or (when the daemon paused it on its own initiative
|
||||
/// rather than the user) paused state. Distinct from the JSON-RPC Error, which describes a
|
||||
/// failed call rather than a failed download — the two live in different code spaces on
|
||||
/// purpose, and `code` here is a TaskErrorCode string, never a JSON-RPC integer. A pause the
|
||||
/// user or the scheduler requested carries no error: this field only explains a paused state
|
||||
/// the daemon entered unilaterally (auth_required, server_file_changed, disk_full and the
|
||||
/// like), never a deliberate one.
|
||||
struct TaskError {
|
||||
TaskErrorCode code{};
|
||||
/// Human-readable, safe to show a user. Never carries a credential, a token or a full local
|
||||
@@ -789,6 +812,9 @@ struct TaskSummary {
|
||||
std::string createdAt{};
|
||||
std::optional<std::string> lastTryAt{};
|
||||
std::optional<std::string> completedAt{};
|
||||
/// Set when state is failed or retry_wait, and also when state is paused and the daemon entered
|
||||
/// that state on its own initiative rather than at a user's or scheduler's request. null on
|
||||
/// every other state, including a deliberate pause.
|
||||
std::optional<TaskError> error{};
|
||||
};
|
||||
|
||||
@@ -805,7 +831,15 @@ struct TaskDetail {
|
||||
std::optional<std::string> referrer{};
|
||||
std::optional<std::string> userAgent{};
|
||||
std::optional<std::string> mime{};
|
||||
/// The REQUESTED write buffer per segment. See effectiveBufferBytes for what is actually in
|
||||
/// use.
|
||||
std::optional<std::int64_t> bufferBytes{};
|
||||
/// The write buffer actually in use per live segment, right now. May be well below bufferBytes:
|
||||
/// the engine reduces every live segment's buffer to fit connection.maxTotalBufferBytes across
|
||||
/// connection.maxActiveSegments concurrently-transferring segments, and reports the reduced
|
||||
/// value here so the GUI can show '16 MiB (using 4 MiB)'. null before the task has started its
|
||||
/// first segment.
|
||||
std::optional<std::int64_t> effectiveBufferBytes{};
|
||||
/// Absolute path of the .veloxpart file while the task is unfinished.
|
||||
std::optional<std::string> partPath{};
|
||||
std::optional<Checksum> checksum{};
|
||||
@@ -977,6 +1011,20 @@ struct DownloadProbeResult {
|
||||
std::optional<bool> requiresAuth{};
|
||||
};
|
||||
|
||||
struct DownloadProvideAuthParams {
|
||||
std::string taskId{};
|
||||
std::string username{};
|
||||
std::string password{};
|
||||
/// true persists the credential in the Secret Service, keyed by host and realm, for future
|
||||
/// downloads from the same site. false or null uses it for this task's retry only. Never
|
||||
/// affects SQLite or the daemon's logs either way.
|
||||
std::optional<bool> save{};
|
||||
};
|
||||
|
||||
struct DownloadProvideAuthResult {
|
||||
bool ok{};
|
||||
};
|
||||
|
||||
struct DownloadRefreshUrlParams {
|
||||
std::string taskId{};
|
||||
std::string url{};
|
||||
@@ -1030,6 +1078,8 @@ struct DownloadUpdateParamsPatch {
|
||||
/// as DownloadSpec.segments. Takes effect on the next start; a running task is not re-segmented
|
||||
/// underneath the user.
|
||||
std::optional<std::int64_t> segments{};
|
||||
/// The REQUESTED write buffer per segment. Subject to the same maxTotalBufferBytes reduction as
|
||||
/// DownloadSpec.bufferBytes; the effective value comes back on the next download.get.
|
||||
std::optional<std::int64_t> bufferBytes{};
|
||||
std::optional<Checksum> checksum{};
|
||||
};
|
||||
@@ -1351,6 +1401,12 @@ struct TaskStateEvent {
|
||||
TaskState state{};
|
||||
std::optional<TaskState> previousState{};
|
||||
std::optional<TaskSummary> summary{};
|
||||
/// Set when the new state is failed or retry_wait, and also when it is paused and the daemon
|
||||
/// entered that state on its own initiative — auth_required, server_file_changed, disk_full and
|
||||
/// the like — rather than because of a user action, a schedule window closing, or an
|
||||
/// admission-control decision. null on every other transition, including every
|
||||
/// deliberately-requested pause. A client must not assume a paused task has no error just
|
||||
/// because it usually doesn't; check this field rather than the state name alone.
|
||||
std::optional<TaskError> error{};
|
||||
};
|
||||
|
||||
@@ -1425,6 +1481,8 @@ void to_json(nlohmann::json& j, const DownloadListResult& v);
|
||||
void to_json(nlohmann::json& j, const DownloadPauseParams& v);
|
||||
void to_json(nlohmann::json& j, const DownloadProbeParams& v);
|
||||
void to_json(nlohmann::json& j, const DownloadProbeResult& v);
|
||||
void to_json(nlohmann::json& j, const DownloadProvideAuthParams& v);
|
||||
void to_json(nlohmann::json& j, const DownloadProvideAuthResult& v);
|
||||
void to_json(nlohmann::json& j, const DownloadRefreshUrlParams& v);
|
||||
void to_json(nlohmann::json& j, const DownloadRefreshUrlResult& v);
|
||||
void to_json(nlohmann::json& j, const DownloadRemoveParams& v);
|
||||
@@ -1570,6 +1628,8 @@ template <> Result<DownloadListResult> parse<DownloadListResult>(const nlohmann:
|
||||
template <> Result<DownloadPauseParams> parse<DownloadPauseParams>(const nlohmann::json& j, std::string_view path);
|
||||
template <> Result<DownloadProbeParams> parse<DownloadProbeParams>(const nlohmann::json& j, std::string_view path);
|
||||
template <> Result<DownloadProbeResult> parse<DownloadProbeResult>(const nlohmann::json& j, std::string_view path);
|
||||
template <> Result<DownloadProvideAuthParams> parse<DownloadProvideAuthParams>(const nlohmann::json& j, std::string_view path);
|
||||
template <> Result<DownloadProvideAuthResult> parse<DownloadProvideAuthResult>(const nlohmann::json& j, std::string_view path);
|
||||
template <> Result<DownloadRefreshUrlParams> parse<DownloadRefreshUrlParams>(const nlohmann::json& j, std::string_view path);
|
||||
template <> Result<DownloadRefreshUrlResult> parse<DownloadRefreshUrlResult>(const nlohmann::json& j, std::string_view path);
|
||||
template <> Result<DownloadRemoveParams> parse<DownloadRemoveParams>(const nlohmann::json& j, std::string_view path);
|
||||
@@ -1657,6 +1717,7 @@ enum class Method {
|
||||
DownloadList, // download.list
|
||||
DownloadPause, // download.pause
|
||||
DownloadProbe, // download.probe
|
||||
DownloadProvideAuth, // download.provideAuth
|
||||
DownloadRefreshUrl, // download.refreshUrl
|
||||
DownloadRemove, // download.remove
|
||||
DownloadResume, // download.resume
|
||||
@@ -1685,7 +1746,7 @@ enum class Method {
|
||||
SettingsSet, // settings.set
|
||||
};
|
||||
|
||||
inline constexpr std::size_t kMethodCount = 38;
|
||||
inline constexpr std::size_t kMethodCount = 39;
|
||||
|
||||
std::string_view to_string(Method m) noexcept;
|
||||
std::optional<Method> method_from_string(std::string_view s) noexcept;
|
||||
@@ -1724,9 +1785,30 @@ nlohmann::json make_error(const nlohmann::json& id, ErrorCode code, std::string_
|
||||
nlohmann::json make_result(const nlohmann::json& id, nlohmann::json result);
|
||||
nlohmann::json make_notification(Event e, nlohmann::json params);
|
||||
|
||||
/// A handler's own failure — as opposed to ParseError, which is the wire failing
|
||||
/// to become typed params. Carries any contract error code, a message, and a
|
||||
/// free-form `data` object that goes straight into the JSON-RPC error's `data`
|
||||
/// field: `{"taskId": ...}` for TaskNotFound, `{"path": ...}` for InvalidPath,
|
||||
/// `{"httpStatus": ...}` for ProbeFailed. `code` defaults to InternalError so a
|
||||
/// handler that sets only a message still produces a valid error response.
|
||||
///
|
||||
/// -32001/-32002/-32003 are the server layer's to raise around dispatch(), not a
|
||||
/// handler's: they are decided before or without reference to method params.
|
||||
struct HandlerError {
|
||||
ErrorCode code{ErrorCode::InternalError};
|
||||
std::string message;
|
||||
// `= nullptr`, not `{nullptr}`: brace-init of nlohmann::json from nullptr
|
||||
// yields the array [null], not JSON null. make_error() drops a null data.
|
||||
nlohmann::json data = nullptr;
|
||||
};
|
||||
|
||||
template <class T>
|
||||
using HandlerResult = std::expected<T, HandlerError>;
|
||||
|
||||
/// One virtual per method. The daemon implements this; `dispatch` below does the
|
||||
/// envelope handling, the transport check and the parameter parsing, so a handler
|
||||
/// only ever sees a validated, typed params struct.
|
||||
/// only ever sees a validated, typed params struct. Return `std::unexpected(
|
||||
/// HandlerError{...})` to answer with a specific error code and data.
|
||||
class Dispatcher {
|
||||
public:
|
||||
virtual ~Dispatcher() = default;
|
||||
@@ -1734,191 +1816,201 @@ public:
|
||||
/// The daemon's capture policy, so the extension's shouldCapture decision cannot drift from the
|
||||
/// daemon's. Fetched on connect and whenever event.settings.changed names a capture.* key. If
|
||||
/// this call fails the extension keeps its last known rules and stays fail-open.
|
||||
virtual Result<CaptureRules> on_capture_getRules(const CaptureGetRulesParams& params) = 0;
|
||||
virtual HandlerResult<CaptureRules> on_capture_getRules(const CaptureGetRulesParams& params) = 0;
|
||||
|
||||
/// Firefox offers an intercepted response to the daemon. The daemon MUST reply within 750 ms;
|
||||
/// the extension abandons the offer and lets Firefox download normally on timeout. This
|
||||
/// deadline is the whole reason capture fails open, and it is conformance-tested: a daemon that
|
||||
/// is slow, down, or erroring must never cost the user a download.
|
||||
virtual Result<CaptureOfferResult> on_capture_offer(const CaptureOfferParams& params) = 0;
|
||||
virtual HandlerResult<CaptureOfferResult> on_capture_offer(const CaptureOfferParams& params) = 0;
|
||||
|
||||
/// Every category with its folder and extension list. The extension calls this to populate its
|
||||
/// default-category picker, which is why it is not privileged; it is read-only and exposes only
|
||||
/// paths the user already configured.
|
||||
virtual Result<CategoryListResult> on_category_list(const CategoryListParams& params) = 0;
|
||||
virtual HandlerResult<CategoryListResult> on_category_list(const CategoryListParams& params) = 0;
|
||||
|
||||
/// Delete a user-created category. Built-in categories are refused with -32602. Tasks filed
|
||||
/// under it are reassigned to reassignTo, or to the default category when that is null; no task
|
||||
/// is ever orphaned.
|
||||
virtual Result<CategoryRemoveResult> on_category_remove(const CategoryRemoveParams& params) = 0;
|
||||
virtual HandlerResult<CategoryRemoveResult> on_category_remove(const CategoryRemoveParams& params) = 0;
|
||||
|
||||
/// Create or replace a category. Omit categoryId to create; supply it to replace. Changing
|
||||
/// saveDir does not move existing files — the GUI asks separately and issues download.update
|
||||
/// per task, so a re-point is never a surprise mass file move.
|
||||
virtual Result<CategoryUpsertResult> on_category_upsert(const CategoryUpsertParams& params) = 0;
|
||||
virtual HandlerResult<CategoryUpsertResult> on_category_upsert(const CategoryUpsertParams& params) = 0;
|
||||
|
||||
/// Create one task. saveDir is canonicalized and checked against saveTo.allowedRoots before
|
||||
/// anything is written; a path that escapes them is refused with -32011 and no file is created.
|
||||
virtual Result<DownloadAddResult> on_download_add(const DownloadSpec& params) = 0;
|
||||
virtual HandlerResult<DownloadAddResult> on_download_add(const DownloadSpec& params) = 0;
|
||||
|
||||
/// Create many tasks in one call: the clipboard blob, the wildcard expander, and the
|
||||
/// extension's 'Download all links'. Partial success is normal and is reported per item rather
|
||||
/// than failing the whole batch.
|
||||
virtual Result<DownloadAddBatchResult> on_download_addBatch(const DownloadAddBatchParams& params) = 0;
|
||||
virtual HandlerResult<DownloadAddBatchResult> on_download_addBatch(const DownloadAddBatchParams& params) = 0;
|
||||
|
||||
/// Stop the given tasks and mark them cancelled. The .veloxpart file is kept so the user can
|
||||
/// still resume from the list; download.remove is what deletes bytes.
|
||||
virtual Result<BulkTaskResult> on_download_cancel(const DownloadCancelParams& params) = 0;
|
||||
virtual HandlerResult<BulkTaskResult> on_download_cancel(const DownloadCancelParams& params) = 0;
|
||||
|
||||
/// Full detail for one task, including per-segment state. Backs the progress dialog. Poll it no
|
||||
/// faster than the progress dialog repaints; the table must use events instead.
|
||||
virtual Result<TaskDetail> on_download_get(const DownloadGetParams& params) = 0;
|
||||
virtual HandlerResult<TaskDetail> on_download_get(const DownloadGetParams& params) = 0;
|
||||
|
||||
/// The main table. Filtering, sorting and paging all happen in the daemon so the GUI never
|
||||
/// materializes 100k rows to show 40. Called once on connect; after that the table is
|
||||
/// maintained from events, never re-fetched on a progress tick.
|
||||
virtual Result<DownloadListResult> on_download_list(const DownloadListParams& params) = 0;
|
||||
virtual HandlerResult<DownloadListResult> on_download_list(const DownloadListParams& params) = 0;
|
||||
|
||||
/// Suspend transfers and flush every segment's progress to the .veloxpart.meta file, so a pause
|
||||
/// is indistinguishable from a crash as far as resume is concerned. Never loses bytes already
|
||||
/// written.
|
||||
virtual Result<BulkTaskResult> on_download_pause(const DownloadPauseParams& params) = 0;
|
||||
virtual HandlerResult<BulkTaskResult> on_download_pause(const DownloadPauseParams& params) = 0;
|
||||
|
||||
/// Ask what is at a URL without creating a task. Populates the File Info dialog. Runs a HEAD,
|
||||
/// falling back to a ranged GET when HEAD is refused, which is also how resumability is
|
||||
/// established. Never blocks the RPC loop; the dialog opens immediately and fills in when this
|
||||
/// lands.
|
||||
virtual Result<DownloadProbeResult> on_download_probe(const DownloadProbeParams& params) = 0;
|
||||
virtual HandlerResult<DownloadProbeResult> on_download_probe(const DownloadProbeParams& params) = 0;
|
||||
|
||||
/// Answer an event.auth.required challenge. The task sits in retry_wait until this arrives; on
|
||||
/// success the daemon retries with the credentials attached and the task resumes on its own —
|
||||
/// this method does not itself start the transfer. Privileged and Unix-socket-only: a
|
||||
/// credential-bearing method must never be reachable from the browser, which is exactly the
|
||||
/// boundary event.auth.required's own description draws ('never back through this event, never
|
||||
/// into a log') — this is the other half of that promise. Credentials are handed to the Secret
|
||||
/// Service, never to SQLite and never logged; save only tells the daemon whether to persist
|
||||
/// them there for next time, or use them for this attempt alone.
|
||||
virtual HandlerResult<DownloadProvideAuthResult> on_download_provideAuth(const DownloadProvideAuthParams& params) = 0;
|
||||
|
||||
/// IDM's 'Refresh Download Address'. Point an existing task at a freshly-issued URL when a
|
||||
/// signed link has expired, keeping every byte already on disk. The daemon re-probes and
|
||||
/// compares size and validator: if they still match, the transfer resumes from where it
|
||||
/// stopped; if they do not, it says so rather than silently restarting.
|
||||
virtual Result<DownloadRefreshUrlResult> on_download_refreshUrl(const DownloadRefreshUrlParams& params) = 0;
|
||||
virtual HandlerResult<DownloadRefreshUrlResult> on_download_refreshUrl(const DownloadRefreshUrlParams& params) = 0;
|
||||
|
||||
/// Drop tasks from the list, optionally deleting the bytes on disk. Privileged: this is the
|
||||
/// only method that destroys user data, and the extension is never allowed to reach it. The
|
||||
/// daemon deletes the .veloxpart and .veloxpart.meta pair, and the finished file only when
|
||||
/// deleteFile is true.
|
||||
virtual Result<DownloadRemoveResult> on_download_remove(const DownloadRemoveParams& params) = 0;
|
||||
virtual HandlerResult<DownloadRemoveResult> on_download_remove(const DownloadRemoveParams& params) = 0;
|
||||
|
||||
/// Continue paused tasks. Resumption is revalidated with If-Range against the stored ETag or
|
||||
/// Last-Modified; a 200 where 206 was expected means the file changed on the server, and the
|
||||
/// task moves to failed with a clear error rather than corrupting the part file.
|
||||
virtual Result<BulkTaskResult> on_download_resume(const DownloadResumeParams& params) = 0;
|
||||
virtual HandlerResult<BulkTaskResult> on_download_resume(const DownloadResumeParams& params) = 0;
|
||||
|
||||
/// Begin or restart the given tasks. A task in 'queued' jumps its queue; a task already
|
||||
/// downloading is a no-op reported as changed false.
|
||||
virtual Result<BulkTaskResult> on_download_start(const DownloadStartParams& params) = 0;
|
||||
virtual HandlerResult<BulkTaskResult> on_download_start(const DownloadStartParams& params) = 0;
|
||||
|
||||
/// Change a task's mutable fields. Moving saveDir or filename moves the file on disk in the
|
||||
/// same operation, which is what makes dragging a row onto a category work as one RPC.
|
||||
/// Privileged: it can name a destination path.
|
||||
virtual Result<TaskSummary> on_download_update(const DownloadUpdateParams& params) = 0;
|
||||
virtual HandlerResult<TaskSummary> on_download_update(const DownloadUpdateParams& params) = 0;
|
||||
|
||||
/// Turn selected crawl results into tasks. This is the only grabber call that creates
|
||||
/// downloads, and it names exactly the files the user ticked — a crawl never starts a download
|
||||
/// on its own.
|
||||
virtual Result<GrabberHarvestResult> on_grabber_harvest(const GrabberHarvestParams& params) = 0;
|
||||
virtual HandlerResult<GrabberHarvestResult> on_grabber_harvest(const GrabberHarvestParams& params) = 0;
|
||||
|
||||
/// Start a depth-limited crawl. Nothing is downloaded by this call: it only walks pages and
|
||||
/// collects candidate links, which the wizard then shows for selection. Privileged because an
|
||||
/// unbounded crawl is a resource commitment the browser must not be able to make on the user's
|
||||
/// behalf.
|
||||
virtual Result<GrabberStartResult> on_grabber_start(const GrabberStartParams& params) = 0;
|
||||
virtual HandlerResult<GrabberStartResult> on_grabber_start(const GrabberStartParams& params) = 0;
|
||||
|
||||
/// Poll one crawl. Also delivered as event.grabber.progress; the poll exists so the wizard can
|
||||
/// be reopened on a job it did not start and still catch up.
|
||||
virtual Result<GrabberStatusResult> on_grabber_status(const GrabberStatusParams& params) = 0;
|
||||
virtual HandlerResult<GrabberStatusResult> on_grabber_status(const GrabberStatusParams& params) = 0;
|
||||
|
||||
/// Current global speed limit. Privileged: changing or reading the limiter belongs to the GUI
|
||||
/// and CLI; the extension shows throughput from event.speed.global instead.
|
||||
virtual Result<Limiter> on_limiter_get(const LimiterGetParams& params) = 0;
|
||||
virtual HandlerResult<Limiter> on_limiter_get(const LimiterGetParams& params) = 0;
|
||||
|
||||
/// Set the global token-bucket limit. With applyToRunning true the change re-tunes transfers
|
||||
/// already in flight instead of taking effect only on the next task — the Speed Limiter
|
||||
/// window's 'apply now' button.
|
||||
virtual Result<Limiter> on_limiter_set(const Limiter& params) = 0;
|
||||
virtual HandlerResult<Limiter> on_limiter_set(const Limiter& params) = 0;
|
||||
|
||||
/// Turn one enumerated variant into a task. The daemon fetches the segments in parallel and
|
||||
/// muxes them with ffmpeg; the result is an ordinary task that appears in the list like any
|
||||
/// other download. Refused with -32602 when the variant is DRM-protected.
|
||||
virtual Result<MediaAddVariantResult> on_media_addVariant(const MediaAddVariantParams& params) = 0;
|
||||
virtual HandlerResult<MediaAddVariantResult> on_media_addVariant(const MediaAddVariantParams& params) = 0;
|
||||
|
||||
/// Parse an HLS or DASH manifest in the daemon and enumerate its renditions. The extension
|
||||
/// never parses a manifest — that logic lives in one language, in one place. Variants with drm
|
||||
/// true are reported so the UI can grey them out; DRM-protected streams are refused, not
|
||||
/// attempted.
|
||||
virtual Result<MediaListVariantsResult> on_media_listVariants(const MediaListVariantsParams& params) = 0;
|
||||
virtual HandlerResult<MediaListVariantsResult> on_media_listVariants(const MediaListVariantsParams& params) = 0;
|
||||
|
||||
/// Every queue with its run state and ordering. Not privileged: the extension's 'Add to Queue'
|
||||
/// picker needs it.
|
||||
virtual Result<QueueListResult> on_queue_list(const QueueListParams& params) = 0;
|
||||
virtual HandlerResult<QueueListResult> on_queue_list(const QueueListParams& params) = 0;
|
||||
|
||||
/// Rewrite a queue's run order. taskIds must be a permutation of the queue's current
|
||||
/// membership; anything else is -32602 rather than a partial reorder, so a stale drag from an
|
||||
/// out-of-date view cannot quietly reshuffle the queue.
|
||||
virtual Result<QueueReorderResult> on_queue_reorder(const QueueReorderParams& params) = 0;
|
||||
virtual HandlerResult<QueueReorderResult> on_queue_reorder(const QueueReorderParams& params) = 0;
|
||||
|
||||
/// Start a queue running. The scheduler then admits up to maxConcurrent tasks from it, in
|
||||
/// order, and keeps that many running until the queue drains or is stopped.
|
||||
virtual Result<QueueStartResult> on_queue_start(const QueueStartParams& params) = 0;
|
||||
virtual HandlerResult<QueueStartResult> on_queue_start(const QueueStartParams& params) = 0;
|
||||
|
||||
/// Stop admitting new tasks from a queue. Tasks already running are paused when pauseRunning is
|
||||
/// true, and otherwise allowed to finish — the difference between 'stop the queue' and 'stop
|
||||
/// everything', which IDM conflates and users trip over.
|
||||
virtual Result<QueueStopResult> on_queue_stop(const QueueStopParams& params) = 0;
|
||||
virtual HandlerResult<QueueStopResult> on_queue_stop(const QueueStopParams& params) = 0;
|
||||
|
||||
/// Create or replace a queue, including its schedule and concurrency cap. Omit queueId to
|
||||
/// create. taskIds in the payload is ignored — membership changes through download.update and
|
||||
/// queue.reorder so that two clients editing at once cannot silently drop a task.
|
||||
virtual Result<QueueUpsertResult> on_queue_upsert(const QueueUpsertParams& params) = 0;
|
||||
virtual HandlerResult<QueueUpsertResult> on_queue_upsert(const QueueUpsertParams& params) = 0;
|
||||
|
||||
/// The rules engine's table, in priority order. Privileged: these are the daemon's routing
|
||||
/// policy. The extension gets its own narrowed view through capture.getRules instead.
|
||||
virtual Result<RulesListResult> on_rules_list(const RulesListParams& params) = 0;
|
||||
virtual HandlerResult<RulesListResult> on_rules_list(const RulesListParams& params) = 0;
|
||||
|
||||
/// Create, replace, or delete rules in one atomic write. 'upsert' carries the rules to store
|
||||
/// and 'remove' the ruleIds to drop; applying both at once means a reprioritisation never
|
||||
/// leaves the table in a half-valid state.
|
||||
virtual Result<RulesUpsertResult> on_rules_upsert(const RulesUpsertParams& params) = 0;
|
||||
virtual HandlerResult<RulesUpsertResult> on_rules_upsert(const RulesUpsertParams& params) = 0;
|
||||
|
||||
/// The schedule for one queue, or every schedule when queueId is null. Backs the Scheduler
|
||||
/// window.
|
||||
virtual Result<ScheduleGetResult> on_schedule_get(const ScheduleGetParams& params) = 0;
|
||||
virtual HandlerResult<ScheduleGetResult> on_schedule_get(const ScheduleGetParams& params) = 0;
|
||||
|
||||
/// Set or clear a queue's schedule. A null schedule clears it and leaves the queue under manual
|
||||
/// control. Times are local wall-clock and are re-evaluated on a DST change rather than being
|
||||
/// resolved to absolute instants at set time.
|
||||
virtual Result<ScheduleSetResult> on_schedule_set(const ScheduleSetParams& params) = 0;
|
||||
virtual HandlerResult<ScheduleSetResult> on_schedule_set(const ScheduleSetParams& params) = 0;
|
||||
|
||||
/// First call on every connection, on every transport. The daemon compares protocolVersion
|
||||
/// majors and refuses a mismatch with -32001 so a stale GUI or extension fails loudly on
|
||||
/// connect instead of subtly at the tenth field. On the WebSocket transport a valid token is
|
||||
/// required unless the client is about to call session.pair.
|
||||
virtual Result<SessionHelloResult> on_session_hello(const SessionHelloParams& params) = 0;
|
||||
virtual HandlerResult<SessionHelloResult> on_session_hello(const SessionHelloParams& params) = 0;
|
||||
|
||||
/// WebSocket transport only. Triggers a GUI or desktop-notification prompt showing a four-digit
|
||||
/// code; the user must approve before a token is issued. Failed attempts are rate-limited to
|
||||
/// 5/min followed by a 60 s lockout (-32014) so a token cannot be brute-forced by another local
|
||||
/// process. The daemon stores only a hash of the token.
|
||||
virtual Result<SessionPairResult> on_session_pair(const SessionPairParams& params) = 0;
|
||||
virtual HandlerResult<SessionPairResult> on_session_pair(const SessionPairParams& params) = 0;
|
||||
|
||||
/// Choose which notifications this connection receives. Subscribing replaces the previous
|
||||
/// selection rather than adding to it, so a client can narrow its firehose without
|
||||
/// reconnecting. Nothing is delivered until this is called.
|
||||
virtual Result<SessionSubscribeResult> on_session_subscribe(const SessionSubscribeParams& params) = 0;
|
||||
virtual HandlerResult<SessionSubscribeResult> on_session_subscribe(const SessionSubscribeParams& params) = 0;
|
||||
|
||||
/// Read settings. keys null means everything. Privileged: the settings bag names local
|
||||
/// filesystem paths and the allowed write roots, which the extension has no business
|
||||
/// enumerating — it gets capture.getRules instead.
|
||||
virtual Result<SettingsGetResult> on_settings_get(const SettingsGetParams& params) = 0;
|
||||
virtual HandlerResult<SettingsGetResult> on_settings_get(const SettingsGetParams& params) = 0;
|
||||
|
||||
/// Write settings. Only the keys present in values change. Rejected with -32602 if a key is
|
||||
/// unknown or a value fails the Settings schema, and with -32011 if a directory key names a
|
||||
/// path that cannot be written. Emits event.settings.changed with exactly the keys that took
|
||||
/// effect.
|
||||
virtual Result<SettingsSetResult> on_settings_set(const SettingsSetParams& params) = 0;
|
||||
virtual HandlerResult<SettingsSetResult> on_settings_set(const SettingsSetParams& params) = 0;
|
||||
|
||||
};
|
||||
|
||||
|
||||
@@ -1,8 +1,12 @@
|
||||
# `libveloxcore` — public API
|
||||
|
||||
**Status: M1 in progress.** Only `util/` is landed. The download-facing API
|
||||
(`DownloadSpec`, `DownloadTask`, probe, typed callbacks) arrives with later stages and
|
||||
is reviewed by DAEMON before M2 (AGENT-CORE DoD).
|
||||
**Status: M1 in progress.** `util/`, `net/` (http_client, probe, url, content_disposition),
|
||||
`io/` (sparse_file, write_buffer), `meta/veloxpart`, and `segment/` (segmenter, budget)
|
||||
are landed. The **download entry point** — `vdm::Engine`, `vdm::task::DownloadSpec` /
|
||||
`DownloadHandle` / `DownloadCallbacks` — is sketched in `vdm/engine.hpp` and
|
||||
`vdm/task/download.hpp` and **out for DAEMON review**: see
|
||||
[`core/docs/engine-api-m1.md`](../../docs/engine-api-m1.md). Bodies land in CORE stage 8;
|
||||
build against the value types now.
|
||||
|
||||
Layering (CLAUDE.md §3): this library knows nothing about JSON, SQL, Qt, or RPC. Input is
|
||||
a spec value; output is bytes on disk plus typed callbacks. DAEMON projects engine state
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
// vdm/engine.hpp — the download engine's single entry point. REVIEW SKETCH (stage 7
|
||||
// pre-work); bodies land in stage 8. See core/docs/engine-api-m1.md.
|
||||
//
|
||||
// The engine owns the HTTP client, the probe pool, the segment budget, and the disk I/O.
|
||||
// Its input is a DownloadSpec; its output is bytes at save_path plus typed callbacks. No
|
||||
// JSON, no SQL, no Qt, no RPC — DAEMON projects the callbacks onto the wire contract.
|
||||
//
|
||||
// This header compiles standalone.
|
||||
|
||||
#ifndef VDM_ENGINE_HPP
|
||||
#define VDM_ENGINE_HPP
|
||||
|
||||
#include <cstdint>
|
||||
#include <memory>
|
||||
|
||||
#include "vdm/rate/token_bucket.hpp"
|
||||
#include "vdm/segment/budget.hpp"
|
||||
#include "vdm/task/download.hpp"
|
||||
|
||||
namespace vdm {
|
||||
|
||||
class Engine {
|
||||
public:
|
||||
struct Config {
|
||||
// Defaults used when a DownloadSpec leaves the field unset. Live-adjustable via
|
||||
// the setters below (they take effect on the next segment (re)assignment, not by
|
||||
// resizing an in-flight buffer).
|
||||
std::uint32_t default_segments = 8; // connection.maxSegmentsPerDownload
|
||||
std::uint64_t default_buffer_bytes = 1u << 20; // connection.bufferBytes (1 MiB)
|
||||
std::uint64_t min_segment_bytes = 1u << 20; // never split below this
|
||||
std::uint64_t max_total_buffer_bytes = 128ull << 20; // connection.maxTotalBufferBytes
|
||||
std::uint32_t max_active_segments = 32; // connection.maxActiveSegments
|
||||
std::uint32_t probe_pool_size = 4; // ADR 0011 §5, outside the budget
|
||||
long default_max_retries = 10; // per segment
|
||||
std::uint32_t http_workers = 0; // 0 => hardware-derived (<=4)
|
||||
};
|
||||
|
||||
Engine(); // default Config
|
||||
explicit Engine(Config cfg);
|
||||
~Engine(); // cancels every running task and joins before returning
|
||||
|
||||
Engine(const Engine &) = delete;
|
||||
Engine &operator=(const Engine &) = delete;
|
||||
|
||||
// Start a download. Returns immediately with a handle; the task begins in `probing`
|
||||
// (or `connecting` when spec.probe_hint is supplied). Every failure — bad URL, DNS,
|
||||
// an unwritable save_path — is delivered through callbacks.on_finished, never thrown.
|
||||
[[nodiscard]] task::DownloadHandle start(task::DownloadSpec spec,
|
||||
task::DownloadCallbacks callbacks);
|
||||
|
||||
// The global segment allocator. DAEMON's scheduler drives admission through this
|
||||
// (set_max_active_segments / set_host_segment_cap / set_task_order) and reads
|
||||
// occupancy from it (budget() / segments_active() / starved_tasks() /
|
||||
// on_budget_changed). See ADR 0011.
|
||||
[[nodiscard]] segment::SegmentBudget &segment_budget() noexcept;
|
||||
|
||||
// The hierarchical speed limiter (docs/04 §6): global -> per-queue -> per-task token
|
||||
// buckets. `limiter.set {globalBps, enabled}` -> rate_limiter().set_global_limit();
|
||||
// per-queue / per-task limits and the task<->queue attachment come from DAEMON too.
|
||||
// The engine paces every segment read through it.
|
||||
[[nodiscard]] rate::RateLimiter &rate_limiter() noexcept;
|
||||
|
||||
// Live settings (connection.* changes from settings.set). Each affects future work.
|
||||
void set_default_segments(std::uint32_t n);
|
||||
void set_default_buffer_bytes(std::uint64_t bytes);
|
||||
void set_max_total_buffer_bytes(std::uint64_t bytes);
|
||||
void set_probe_pool_size(std::uint32_t n);
|
||||
|
||||
// A standalone probe for the File Info dialog, on the same pool as spec-less probes
|
||||
// (never charged against the segment budget). capture.offer's 750 ms deadline is
|
||||
// DAEMON's to enforce — it should answer `ignore` and probe after, never block on
|
||||
// this.
|
||||
void probe(net::ProbeRequest req, std::function<void(Result<net::ProbeResult>)> done);
|
||||
|
||||
private:
|
||||
struct Impl;
|
||||
std::unique_ptr<Impl> impl_;
|
||||
};
|
||||
|
||||
} // namespace vdm
|
||||
|
||||
#endif // VDM_ENGINE_HPP
|
||||
@@ -0,0 +1,54 @@
|
||||
// vdm/ids.hpp — opaque engine-internal identifiers.
|
||||
//
|
||||
// TaskId is CORE's handle for a download. DAEMON owns the wire UUID and keeps a
|
||||
// UUID <-> TaskId map; CORE never sees the UUID (layering rule — no wire types in the
|
||||
// engine). Assigned by CORE when DAEMON registers a task.
|
||||
//
|
||||
// This header compiles standalone.
|
||||
|
||||
#ifndef VDM_IDS_HPP
|
||||
#define VDM_IDS_HPP
|
||||
|
||||
#include <chrono>
|
||||
#include <compare>
|
||||
#include <cstddef>
|
||||
#include <cstdint>
|
||||
#include <functional>
|
||||
|
||||
namespace vdm {
|
||||
|
||||
struct TaskId {
|
||||
std::uint64_t value = 0;
|
||||
|
||||
[[nodiscard]] constexpr bool valid() const noexcept { return value != 0; }
|
||||
friend constexpr auto operator<=>(const TaskId &, const TaskId &) = default;
|
||||
};
|
||||
|
||||
// A scheduler queue. A task may belong to one — for the per-queue rate limit and
|
||||
// per-queue concurrency. A task with no queue rate-limits against the global bucket only.
|
||||
struct QueueId {
|
||||
std::uint64_t value = 0;
|
||||
|
||||
[[nodiscard]] constexpr bool valid() const noexcept { return value != 0; }
|
||||
friend constexpr auto operator<=>(const QueueId &, const QueueId &) = default;
|
||||
};
|
||||
|
||||
using SteadyTime = std::chrono::steady_clock::time_point;
|
||||
|
||||
} // namespace vdm
|
||||
|
||||
template <>
|
||||
struct std::hash<vdm::TaskId> {
|
||||
std::size_t operator()(vdm::TaskId id) const noexcept {
|
||||
return std::hash<std::uint64_t>{}(id.value);
|
||||
}
|
||||
};
|
||||
|
||||
template <>
|
||||
struct std::hash<vdm::QueueId> {
|
||||
std::size_t operator()(vdm::QueueId id) const noexcept {
|
||||
return std::hash<std::uint64_t>{}(id.value);
|
||||
}
|
||||
};
|
||||
|
||||
#endif // VDM_IDS_HPP
|
||||
@@ -0,0 +1,80 @@
|
||||
// vdm/io/sparse_file.hpp — the one output file, written at absolute offsets.
|
||||
//
|
||||
// docs/04 §4: one file, opened once, O_WRONLY; posix_fallocate the full size up front;
|
||||
// each segment pwrite()s at its own offset so there is no reassembly pass; fadvise
|
||||
// DONTNEED on written ranges; fdatasync on a timer, never per write.
|
||||
//
|
||||
// This header compiles standalone.
|
||||
|
||||
#ifndef VDM_IO_SPARSE_FILE_HPP
|
||||
#define VDM_IO_SPARSE_FILE_HPP
|
||||
|
||||
#include <cstdint>
|
||||
#include <string>
|
||||
#include <string_view>
|
||||
|
||||
#include "vdm/util/bytes.hpp"
|
||||
#include "vdm/util/result.hpp"
|
||||
|
||||
namespace vdm::io {
|
||||
|
||||
class SparseFile {
|
||||
public:
|
||||
struct OpenOptions {
|
||||
std::uint64_t total_size = 0; // full final size; 0 = unknown (chunked transfer)
|
||||
bool preallocate = true; // posix_fallocate; falls back to ftruncate
|
||||
bool truncate_existing = false; // start fresh (true) vs. resume into an existing
|
||||
// part file (false)
|
||||
};
|
||||
|
||||
SparseFile() = default;
|
||||
~SparseFile();
|
||||
|
||||
SparseFile(SparseFile &&) noexcept;
|
||||
SparseFile &operator=(SparseFile &&) noexcept;
|
||||
SparseFile(const SparseFile &) = delete;
|
||||
SparseFile &operator=(const SparseFile &) = delete;
|
||||
|
||||
// Open `path` for writing at absolute offsets, creating it if needed. With
|
||||
// `preallocate` and a known `total_size`, posix_fallocate the whole file (contiguous
|
||||
// extents, no ENOSPC surprise at 99%). EOPNOTSUPP/ENOSYS (tmpfs, some network FS)
|
||||
// falls back to ftruncate — sparse, no extent reservation — and is reported by
|
||||
// preallocated().
|
||||
[[nodiscard]] Result<void> open(std::string_view path, const OpenOptions &opts);
|
||||
[[nodiscard]] Result<void> open(std::string_view path); // default OpenOptions
|
||||
|
||||
[[nodiscard]] bool is_open() const noexcept { return fd_ >= 0; }
|
||||
[[nodiscard]] bool preallocated() const noexcept { return preallocated_; }
|
||||
[[nodiscard]] std::string_view path() const noexcept { return path_; }
|
||||
|
||||
// pwrite the whole span at `offset`, looping over short writes and retrying EINTR.
|
||||
// Safe to call concurrently with other write_at()/sync() on the same object as long
|
||||
// as the byte ranges do not overlap — POSIX guarantees each pwrite is atomic for a
|
||||
// regular file, so no lock is taken on the hot path.
|
||||
[[nodiscard]] Result<void> write_at(std::uint64_t offset, ConstByteSpan data);
|
||||
|
||||
// fdatasync. Call on a timer (default 5 s) and on pause — never per write (docs/04 §4).
|
||||
[[nodiscard]] Result<void> sync();
|
||||
|
||||
// POSIX_FADV_DONTNEED on [offset, offset+len): drop already-written pages from the
|
||||
// page cache so a 40 GB ISO does not evict the user's working set. Most effective
|
||||
// after sync(). Best-effort — failures are ignored.
|
||||
void advise_dontneed(std::uint64_t offset, std::uint64_t len) noexcept;
|
||||
|
||||
// ftruncate to `size`: give a chunked download its real size once known, or trim a
|
||||
// preallocated tail that a steal/mirror never filled.
|
||||
[[nodiscard]] Result<void> resize(std::uint64_t size);
|
||||
|
||||
[[nodiscard]] Result<void> close();
|
||||
|
||||
private:
|
||||
void reset() noexcept;
|
||||
|
||||
int fd_ = -1;
|
||||
bool preallocated_ = false;
|
||||
std::string path_;
|
||||
};
|
||||
|
||||
} // namespace vdm::io
|
||||
|
||||
#endif // VDM_IO_SPARSE_FILE_HPP
|
||||
@@ -0,0 +1,69 @@
|
||||
// vdm/io/write_buffer.hpp — per-segment accumulate-and-flush buffer.
|
||||
//
|
||||
// docs/04 §4: curl's write callback appends; the buffer is flushed with a single pwrite
|
||||
// when full or when the segment ends. §8: NO ALLOCATION in the write-callback hot path —
|
||||
// the buffer is preallocated at construction and append() only memcpys.
|
||||
//
|
||||
// Single-threaded. The owning download_task (stage 8) adds the disk-writer-thread handoff
|
||||
// (double buffering) on top; this primitive takes no lock.
|
||||
//
|
||||
// This header compiles standalone.
|
||||
|
||||
#ifndef VDM_IO_WRITE_BUFFER_HPP
|
||||
#define VDM_IO_WRITE_BUFFER_HPP
|
||||
|
||||
#include <cstddef>
|
||||
#include <cstdint>
|
||||
#include <functional>
|
||||
#include <vector>
|
||||
|
||||
#include "vdm/util/bytes.hpp"
|
||||
#include "vdm/util/result.hpp"
|
||||
|
||||
namespace vdm::io {
|
||||
|
||||
class WriteBuffer {
|
||||
public:
|
||||
// Writes `span` durably at absolute file offset `offset`. Must not allocate
|
||||
// (SparseFile::write_at doesn't). Return an error to abort the segment; WriteBuffer
|
||||
// propagates it and keeps the un-flushed bytes so the caller can decide.
|
||||
using FlushFn = std::function<Result<void>(std::uint64_t offset, ConstByteSpan span)>;
|
||||
|
||||
// `capacity` is the effective per-segment buffer_bytes (already clamped by the
|
||||
// segmenter against max_total_buffer_bytes). Must be > 0.
|
||||
WriteBuffer(std::uint64_t start_offset, std::size_t capacity, FlushFn flush);
|
||||
|
||||
WriteBuffer(WriteBuffer &&) noexcept = default;
|
||||
WriteBuffer &operator=(WriteBuffer &&) noexcept = default;
|
||||
WriteBuffer(const WriteBuffer &) = delete;
|
||||
WriteBuffer &operator=(const WriteBuffer &) = delete;
|
||||
|
||||
// Append body bytes. Flushes automatically each time the buffer fills. A chunk at
|
||||
// least `capacity` bytes long, arriving when the buffer is empty, is written straight
|
||||
// through (one extra flush call, still no allocation and no memcpy).
|
||||
[[nodiscard]] Result<void> append(ConstByteSpan span);
|
||||
|
||||
// Flush whatever is buffered right now. Call at segment end and on pause. A no-op
|
||||
// (success) when nothing is pending.
|
||||
[[nodiscard]] Result<void> flush();
|
||||
|
||||
[[nodiscard]] std::size_t capacity() const noexcept { return buf_.size(); }
|
||||
[[nodiscard]] std::size_t pending() const noexcept { return len_; }
|
||||
// File offset the next flush will write at (start + all bytes already flushed).
|
||||
[[nodiscard]] std::uint64_t next_offset() const noexcept { return base_; }
|
||||
// Total bytes handed to append() over this buffer's life.
|
||||
[[nodiscard]] std::uint64_t total_appended() const noexcept { return appended_; }
|
||||
|
||||
private:
|
||||
Result<void> flush_pending();
|
||||
|
||||
std::vector<std::byte> buf_;
|
||||
std::size_t len_ = 0; // bytes currently in buf_
|
||||
std::uint64_t base_ = 0; // file offset of buf_[0]
|
||||
std::uint64_t appended_ = 0;
|
||||
FlushFn flush_;
|
||||
};
|
||||
|
||||
} // namespace vdm::io
|
||||
|
||||
#endif // VDM_IO_WRITE_BUFFER_HPP
|
||||
@@ -0,0 +1,109 @@
|
||||
// vdm/meta/veloxpart.hpp — the `<name>.veloxpart.meta` resume sidecar (docs/04 §5).
|
||||
//
|
||||
// Written next to the part file so a download survives a daemon crash, a reboot, and a
|
||||
// database loss. Little-endian, versioned, CRC-32 over the whole record, fdatasync'd at
|
||||
// segment boundaries.
|
||||
//
|
||||
// The reader is written first and fuzzed (AGENT-CORE §5): this file lives in a
|
||||
// world-writable-ish download directory, so parse_veloxpart() is total on hostile input —
|
||||
// every malformation is a Result error (meta_corrupt / meta_version_unsupported), never a
|
||||
// crash, an over-read, or an unbounded allocation.
|
||||
//
|
||||
// On-disk layout (all integers little-endian):
|
||||
//
|
||||
// magic "VDMP" 4 bytes
|
||||
// version u16 (this build writes/reads kVersion)
|
||||
// flags u16 (bit0: sha256_state present)
|
||||
// total_size u64 (0 = unknown / chunked)
|
||||
// downloaded u64 (sum of segment.completed; a fast read)
|
||||
// url_count u32
|
||||
// url[0] = original, url[1] = effective, url[2..] = mirrors, each length-prefixed
|
||||
// etag length-prefixed UTF-8
|
||||
// last_modified length-prefixed UTF-8
|
||||
// content_type length-prefixed UTF-8
|
||||
// segment_count u32
|
||||
// per segment: start u64, end u64 (INCLUSIVE), completed u64
|
||||
// [flags bit0] sha256_state_len u32, then that many opaque bytes
|
||||
// crc32 u32 (over every byte before this field)
|
||||
//
|
||||
// length-prefixed = u32 length, then that many bytes.
|
||||
//
|
||||
// This header compiles standalone.
|
||||
|
||||
#ifndef VDM_META_VELOXPART_HPP
|
||||
#define VDM_META_VELOXPART_HPP
|
||||
|
||||
#include <cstdint>
|
||||
#include <string>
|
||||
#include <string_view>
|
||||
#include <vector>
|
||||
|
||||
#include "vdm/util/bytes.hpp"
|
||||
#include "vdm/util/result.hpp"
|
||||
|
||||
namespace vdm::meta {
|
||||
|
||||
inline constexpr std::uint16_t kVersion = 1;
|
||||
inline constexpr std::uint16_t kFlagHasShaState = 0x0001;
|
||||
|
||||
// Hard caps the reader enforces so a hostile count/length can't drive allocation or work.
|
||||
inline constexpr std::uint32_t kMaxUrls = 64;
|
||||
inline constexpr std::uint32_t kMaxSegments = 1024; // contract ceiling is 32; headroom
|
||||
inline constexpr std::uint32_t kMaxStringLen = 16 * 1024;
|
||||
inline constexpr std::uint32_t kMaxShaStateLen = 4 * 1024;
|
||||
inline constexpr std::size_t kMaxImageBytes = 256 * 1024; // a real sidecar is < 4 KiB
|
||||
|
||||
struct SegmentRecord {
|
||||
std::uint64_t start = 0;
|
||||
std::uint64_t end = 0; // INCLUSIVE, per contract Segment.endByte / ADR 0010
|
||||
std::uint64_t completed = 0;
|
||||
|
||||
[[nodiscard]] std::uint64_t length() const noexcept {
|
||||
return end >= start ? end - start + 1 : 0;
|
||||
}
|
||||
bool operator==(const SegmentRecord &) const = default;
|
||||
};
|
||||
|
||||
struct VeloxPart {
|
||||
std::uint16_t version = kVersion;
|
||||
std::uint16_t flags = 0;
|
||||
std::uint64_t total_size = 0;
|
||||
std::uint64_t downloaded = 0;
|
||||
std::vector<std::string> urls; // [0]=original, [1]=effective, [2..]=mirrors
|
||||
std::string etag;
|
||||
std::string last_modified;
|
||||
std::string content_type;
|
||||
std::vector<SegmentRecord> segments;
|
||||
std::vector<std::byte> sha256_state;
|
||||
|
||||
[[nodiscard]] std::string_view original_url() const {
|
||||
return urls.empty() ? std::string_view{} : std::string_view(urls[0]);
|
||||
}
|
||||
[[nodiscard]] std::string_view effective_url() const {
|
||||
return urls.size() < 2 ? original_url() : std::string_view(urls[1]);
|
||||
}
|
||||
|
||||
bool operator==(const VeloxPart &) const = default;
|
||||
};
|
||||
|
||||
// Parse a sidecar image. Every failure is a Result error, never a throw or a crash:
|
||||
// meta_corrupt — bad magic, truncation, a count/length past a cap or past
|
||||
// the buffer, trailing bytes, or a CRC mismatch
|
||||
// meta_version_unsupported — magic OK, CRC OK, but version > kVersion
|
||||
[[nodiscard]] Result<VeloxPart> parse_veloxpart(ConstByteSpan image);
|
||||
|
||||
// Serialize. Deterministic: the same VeloxPart always produces the same bytes, so an
|
||||
// unchanged sidecar is not rewritten. The CRC-32 is appended.
|
||||
[[nodiscard]] std::vector<std::byte> serialize_veloxpart(const VeloxPart &vp);
|
||||
|
||||
// File helpers — the sidecar path is `<part file>.veloxpart.meta`.
|
||||
[[nodiscard]] Result<VeloxPart> read_veloxpart_file(std::string_view path);
|
||||
|
||||
// Writes atomically (temp + rename) and, when `fsync`, fdatasync's the file and its
|
||||
// directory before returning — call at every segment-boundary update (docs/04 §5).
|
||||
[[nodiscard]] Result<void> write_veloxpart_file(std::string_view path, const VeloxPart &vp,
|
||||
bool fsync = true);
|
||||
|
||||
} // namespace vdm::meta
|
||||
|
||||
#endif // VDM_META_VELOXPART_HPP
|
||||
@@ -0,0 +1,43 @@
|
||||
// vdm/net/content_disposition.hpp — parse a Content-Disposition header into a filename.
|
||||
//
|
||||
// This is a classic mojibake source (AGENT-CORE §3): RFC 6266 `filename`, RFC 5987
|
||||
// `filename*` ext-values, RFC 2047 encoded-words in the legacy quoted form, and raw
|
||||
// Latin-1 bytes all show up in the wild. The parser is total — hostile input yields a
|
||||
// best-effort or empty result, never a throw or a crash — and has its own test table
|
||||
// (content_disposition_test.cpp) and a fuzz target (tools/fuzz).
|
||||
//
|
||||
// This header compiles standalone.
|
||||
|
||||
#ifndef VDM_NET_CONTENT_DISPOSITION_HPP
|
||||
#define VDM_NET_CONTENT_DISPOSITION_HPP
|
||||
|
||||
#include <string>
|
||||
#include <string_view>
|
||||
|
||||
namespace vdm::net {
|
||||
|
||||
struct ContentDisposition {
|
||||
enum class Type { none, inline_, attachment, form_data, other };
|
||||
|
||||
Type type = Type::none;
|
||||
|
||||
// Best-effort UTF-8 filename: path components stripped, control bytes (incl. NUL) and
|
||||
// edge whitespace removed, or empty when the header carries none. NOT fully sanitized
|
||||
// for the filesystem — that is rules/ (stage 9). `..`, reserved names, and other
|
||||
// printable-but-unsafe content may still be present.
|
||||
std::string filename;
|
||||
|
||||
// The filename came from an RFC 5987 `filename*` ext-value (preferred over a plain
|
||||
// `filename` per RFC 6266 §4.3 when both are present).
|
||||
bool filename_from_ext = false;
|
||||
|
||||
[[nodiscard]] bool is_attachment() const noexcept { return type == Type::attachment; }
|
||||
[[nodiscard]] bool has_filename() const noexcept { return !filename.empty(); }
|
||||
};
|
||||
|
||||
// Parse the value of a Content-Disposition header (everything after the colon).
|
||||
[[nodiscard]] ContentDisposition parse_content_disposition(std::string_view header_value);
|
||||
|
||||
} // namespace vdm::net
|
||||
|
||||
#endif // VDM_NET_CONTENT_DISPOSITION_HPP
|
||||
@@ -0,0 +1,125 @@
|
||||
// vdm/net/http_client.hpp — a libcurl-multi wrapper for the download engine.
|
||||
//
|
||||
// One HttpClient owns a small pool of worker threads, each with its own curl_multi
|
||||
// (curl handles are not thread-safe; a handle lives on exactly one worker for its life).
|
||||
// The engine hands it a Request plus callbacks and gets back a Transfer handle it can
|
||||
// pause / resume / cancel from any thread.
|
||||
//
|
||||
// Layering: this is the ONLY core header that pulls in libcurl, and only in its .cpp —
|
||||
// nothing here exposes a curl type. No JSON / SQL / Qt / RPC (CLAUDE.md §3).
|
||||
//
|
||||
// Callbacks run ON THE WORKER THREAD, one transfer at a time for a given Transfer.
|
||||
// They must not block (that stalls every other transfer on that worker) and must not
|
||||
// call back into this Transfer's pause/resume/cancel re-entrantly — post that work
|
||||
// elsewhere. on_data must not allocate on the hot path (AGENT-CORE); the ring buffer it
|
||||
// writes into is preallocated by the caller (stage 4).
|
||||
//
|
||||
// This header compiles standalone.
|
||||
|
||||
#ifndef VDM_NET_HTTP_CLIENT_HPP
|
||||
#define VDM_NET_HTTP_CLIENT_HPP
|
||||
|
||||
#include <cstddef>
|
||||
#include <cstdint>
|
||||
#include <functional>
|
||||
#include <memory>
|
||||
#include <string>
|
||||
|
||||
#include "vdm/net/http_types.hpp"
|
||||
#include "vdm/util/bytes.hpp"
|
||||
#include "vdm/util/result.hpp"
|
||||
|
||||
namespace vdm::net {
|
||||
|
||||
// What on_data tells the client to do with the transfer after this chunk.
|
||||
enum class DataAction {
|
||||
proceed, // keep receiving
|
||||
pause, // stop receiving; resumes on Transfer::resume()
|
||||
abort, // end the transfer now (finishes with Error::canceled)
|
||||
};
|
||||
|
||||
struct TransferStats {
|
||||
std::uint64_t bytes_received = 0;
|
||||
long http_status = 0;
|
||||
std::string effective_url;
|
||||
// Timings in milliseconds (from CURLINFO_*_TIME_T), 0 if the phase didn't happen.
|
||||
long namelookup_ms = 0;
|
||||
long connect_ms = 0;
|
||||
long appconnect_ms = 0; // TLS handshake done
|
||||
long starttransfer_ms = 0; // first response byte
|
||||
long total_ms = 0;
|
||||
};
|
||||
|
||||
struct TransferCallbacks {
|
||||
// Response headers are in. Called at most once per transfer (a followed redirect's
|
||||
// intermediate headers are not delivered). If the caller only wanted headers (a
|
||||
// probe), return DataAction::abort here.
|
||||
std::function<DataAction(const ResponseHead &)> on_head;
|
||||
|
||||
// A chunk of body bytes. The span is valid only for the duration of the call.
|
||||
std::function<DataAction(ConstByteSpan)> on_data;
|
||||
|
||||
// The transfer ended — success carries stats, failure carries the mapped error
|
||||
// (ErrorInfo.http_status is set for HTTP-status failures). Always called exactly once,
|
||||
// last.
|
||||
std::function<void(Result<TransferStats>)> on_finished;
|
||||
};
|
||||
|
||||
class HttpClient;
|
||||
|
||||
// Lightweight handle to a running transfer. Copyable (shared state). All methods are
|
||||
// safe to call from any thread; they post to the owning worker and return immediately.
|
||||
// Dropping the last handle does NOT cancel — call cancel() for that.
|
||||
class Transfer {
|
||||
public:
|
||||
Transfer() = default;
|
||||
|
||||
[[nodiscard]] std::uint64_t id() const noexcept;
|
||||
[[nodiscard]] bool valid() const noexcept { return static_cast<bool>(state_); }
|
||||
|
||||
void pause(); // no-op if already paused / finished
|
||||
void resume(); // no-op if not paused / finished
|
||||
void cancel(); // idempotent; on_finished fires with Error::canceled
|
||||
|
||||
private:
|
||||
friend class HttpClient;
|
||||
struct State;
|
||||
explicit Transfer(std::shared_ptr<State> s) : state_(std::move(s)) {}
|
||||
std::shared_ptr<State> state_;
|
||||
};
|
||||
|
||||
class HttpClient {
|
||||
public:
|
||||
struct Options {
|
||||
// Worker threads, each with its own curl_multi. New transfers are assigned
|
||||
// round-robin. 0 => pick from hardware_concurrency (min 1, max 4).
|
||||
unsigned workers = 0;
|
||||
// CURLMOPT_MAX_TOTAL_CONNECTIONS per worker (0 = curl default).
|
||||
long max_connections_per_worker = 0;
|
||||
// Shared DNS + TLS-session cache across this client's workers (curl_share).
|
||||
bool share_dns_and_tls = true;
|
||||
};
|
||||
|
||||
HttpClient(); // default Options
|
||||
explicit HttpClient(Options opts);
|
||||
~HttpClient();
|
||||
|
||||
HttpClient(const HttpClient &) = delete;
|
||||
HttpClient &operator=(const HttpClient &) = delete;
|
||||
|
||||
[[nodiscard]] unsigned worker_count() const noexcept;
|
||||
|
||||
// Start a transfer. The Request is consumed (moved). Returns an invalid Transfer and
|
||||
// never calls the callbacks only if the client is shutting down; every other failure
|
||||
// (bad URL, DNS, ...) is delivered through on_finished.
|
||||
Transfer start(Request req, TransferCallbacks cbs);
|
||||
|
||||
private:
|
||||
friend class Transfer; // Transfer posts pause/resume/cancel commands to Impl
|
||||
struct Impl;
|
||||
std::unique_ptr<Impl> impl_;
|
||||
};
|
||||
|
||||
} // namespace vdm::net
|
||||
|
||||
#endif // VDM_NET_HTTP_CLIENT_HPP
|
||||
@@ -0,0 +1,156 @@
|
||||
// vdm/net/http_types.hpp — value types for HTTP requests and responses.
|
||||
//
|
||||
// No libcurl in this header: it is the vocabulary the rest of core/ speaks to the net
|
||||
// layer. http_client.hpp is the only place curl leaks in, and only in its .cpp.
|
||||
//
|
||||
// This header compiles standalone.
|
||||
|
||||
#ifndef VDM_NET_HTTP_TYPES_HPP
|
||||
#define VDM_NET_HTTP_TYPES_HPP
|
||||
|
||||
#include <cstdint>
|
||||
#include <optional>
|
||||
#include <string>
|
||||
#include <string_view>
|
||||
#include <utility>
|
||||
#include <vector>
|
||||
|
||||
namespace vdm::net {
|
||||
|
||||
enum class Method { get, head };
|
||||
|
||||
[[nodiscard]] constexpr std::string_view method_name(Method m) noexcept {
|
||||
return m == Method::head ? "HEAD" : "GET";
|
||||
}
|
||||
|
||||
// A closed byte range [first, last]. `last == kUnbounded` means "to the end of the
|
||||
// resource" (`Range: bytes=first-`). This mirrors HTTP Range semantics — inclusive on
|
||||
// both ends — deliberately: see core/docs/proto-requests-m1.md B3.
|
||||
struct ByteRange {
|
||||
static constexpr std::uint64_t kUnbounded = ~std::uint64_t{0};
|
||||
|
||||
std::uint64_t first = 0;
|
||||
std::uint64_t last = kUnbounded;
|
||||
|
||||
[[nodiscard]] bool bounded() const noexcept { return last != kUnbounded; }
|
||||
[[nodiscard]] std::optional<std::uint64_t> length() const noexcept {
|
||||
if (!bounded())
|
||||
return std::nullopt;
|
||||
return last - first + 1;
|
||||
}
|
||||
// "bytes=100-199" or "bytes=100-"
|
||||
[[nodiscard]] std::string to_header_value() const {
|
||||
std::string v = "bytes=";
|
||||
v += std::to_string(first);
|
||||
v += '-';
|
||||
if (bounded())
|
||||
v += std::to_string(last);
|
||||
return v;
|
||||
}
|
||||
};
|
||||
|
||||
struct HeaderField {
|
||||
std::string name;
|
||||
std::string value;
|
||||
};
|
||||
|
||||
// Case-insensitive view over response headers. Not a multimap for perf — the header set
|
||||
// on a download response is tiny; linear scan is fine and keeps this allocation-light.
|
||||
class HeaderList {
|
||||
public:
|
||||
void add(std::string name, std::string value) {
|
||||
fields_.push_back({std::move(name), std::move(value)});
|
||||
}
|
||||
void clear() noexcept { fields_.clear(); }
|
||||
|
||||
[[nodiscard]] const std::vector<HeaderField> &fields() const noexcept { return fields_; }
|
||||
[[nodiscard]] bool empty() const noexcept { return fields_.empty(); }
|
||||
|
||||
// First value for `name` (ASCII case-insensitive), or nullopt.
|
||||
[[nodiscard]] std::optional<std::string_view> get(std::string_view name) const {
|
||||
for (const auto &f : fields_)
|
||||
if (iequals(f.name, name))
|
||||
return f.value;
|
||||
return std::nullopt;
|
||||
}
|
||||
[[nodiscard]] bool has(std::string_view name) const { return get(name).has_value(); }
|
||||
|
||||
static bool iequals(std::string_view a, std::string_view b) noexcept {
|
||||
if (a.size() != b.size())
|
||||
return false;
|
||||
for (std::size_t i = 0; i < a.size(); ++i)
|
||||
if (lower(a[i]) != lower(b[i]))
|
||||
return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
private:
|
||||
static constexpr char lower(char c) noexcept {
|
||||
return (c >= 'A' && c <= 'Z') ? char(c - 'A' + 'a') : c;
|
||||
}
|
||||
std::vector<HeaderField> fields_;
|
||||
};
|
||||
|
||||
enum class ProxyKind { none, http, socks5, socks5_hostname };
|
||||
|
||||
struct ProxyConfig {
|
||||
ProxyKind kind = ProxyKind::none;
|
||||
std::string host; // host[:port]
|
||||
std::uint16_t port = 0;
|
||||
std::string username; // empty = no proxy auth
|
||||
std::string password;
|
||||
};
|
||||
|
||||
enum class AuthScheme { none, basic, digest, any };
|
||||
|
||||
struct AuthConfig {
|
||||
AuthScheme scheme = AuthScheme::none;
|
||||
std::string username;
|
||||
std::string password;
|
||||
};
|
||||
|
||||
struct Cookie {
|
||||
std::string name;
|
||||
std::string value;
|
||||
};
|
||||
|
||||
// One HTTP request the engine wants performed. Defaults are the well-behaved case.
|
||||
struct Request {
|
||||
std::string url;
|
||||
Method method = Method::get;
|
||||
std::vector<HeaderField> headers; // verbatim; browser UA/Referer/cookies live here
|
||||
std::optional<ByteRange> range;
|
||||
std::vector<Cookie> cookies; // merged into a Cookie: header + curl's jar
|
||||
std::string user_agent; // convenience; also settable via headers
|
||||
std::string referrer;
|
||||
|
||||
ProxyConfig proxy;
|
||||
AuthConfig auth;
|
||||
|
||||
bool follow_redirects = true;
|
||||
long max_redirects = 20;
|
||||
bool accept_encoding = false; // OFF for downloads: a gzip'd body breaks Range math
|
||||
|
||||
// Stall detection: abort if throughput stays under `low_speed_bytes_per_sec` for
|
||||
// `low_speed_secs`. 0 disables. Distinct from an overall deadline (probe sets one).
|
||||
long connect_timeout_ms = 15000;
|
||||
long low_speed_bytes_per_sec = 1024;
|
||||
long low_speed_secs = 30;
|
||||
long overall_timeout_ms = 0; // 0 = none; probe uses a few seconds
|
||||
|
||||
// Coarse download-rate ceiling handed to curl (CURLOPT_MAX_RECV_SPEED_LARGE). The
|
||||
// precise limiter (stage 7) pauses/resumes on top of this. 0 = unlimited.
|
||||
std::uint64_t max_recv_bytes_per_sec = 0;
|
||||
};
|
||||
|
||||
// Delivered once, when response headers are in.
|
||||
struct ResponseHead {
|
||||
long status = 0;
|
||||
std::string effective_url; // after redirects
|
||||
HeaderList headers;
|
||||
std::optional<std::uint64_t> content_length; // from Content-Length, if present & sane
|
||||
};
|
||||
|
||||
} // namespace vdm::net
|
||||
|
||||
#endif // VDM_NET_HTTP_TYPES_HPP
|
||||
@@ -0,0 +1,91 @@
|
||||
// vdm/net/probe.hpp — "what is at this URL?" without downloading it.
|
||||
//
|
||||
// Feeds the File Info dialog (docs/04 §2). HEAD first; a ranged GET `bytes=0-0` follows to
|
||||
// PROVE resumability (a 206 with a matching Content-Range) rather than trust
|
||||
// `Accept-Ranges`, which servers lie about (docs/06 R4). The ranged GET is also the
|
||||
// fallback when HEAD is refused (403/405/501).
|
||||
//
|
||||
// Runs on its own small worker pool, sized outside the segment budget (ADR 0011 §5) so a
|
||||
// burst of probes can't starve transfers and capture.offer's 750 ms path never waits on
|
||||
// one.
|
||||
//
|
||||
// This header compiles standalone.
|
||||
|
||||
#ifndef VDM_NET_PROBE_HPP
|
||||
#define VDM_NET_PROBE_HPP
|
||||
|
||||
#include <cstdint>
|
||||
#include <functional>
|
||||
#include <memory>
|
||||
#include <optional>
|
||||
#include <string>
|
||||
#include <string_view>
|
||||
#include <vector>
|
||||
|
||||
#include "vdm/net/content_disposition.hpp"
|
||||
#include "vdm/net/http_types.hpp"
|
||||
#include "vdm/util/result.hpp"
|
||||
|
||||
namespace vdm::net {
|
||||
|
||||
struct ProbeRequest {
|
||||
std::string url;
|
||||
std::vector<HeaderField> headers; // browser headers, verbatim
|
||||
std::vector<Cookie> cookies;
|
||||
std::string user_agent;
|
||||
std::string referrer;
|
||||
ProxyConfig proxy;
|
||||
AuthConfig auth; // credentials for a re-probe after a 401 (leave scheme == none otherwise)
|
||||
|
||||
long connect_timeout_ms = 15000;
|
||||
long overall_timeout_ms = 25000; // download.probe deadline is 30 s
|
||||
};
|
||||
|
||||
struct ProbeResult {
|
||||
std::string effective_url;
|
||||
std::vector<std::string> redirect_chain; // requested URL first, effective_url last
|
||||
long http_status = 0;
|
||||
|
||||
std::optional<std::uint64_t> total_size; // full-resource size, if known
|
||||
std::string mime; // Content-Type value (params kept)
|
||||
std::string etag;
|
||||
std::string last_modified;
|
||||
|
||||
bool accept_ranges = false; // server advertised Accept-Ranges: bytes
|
||||
bool resumable = false; // PROVEN: ranged GET -> 206 + matching Content-Range,
|
||||
// and a validator (ETag or Last-Modified) is present
|
||||
bool requires_auth = false; // a 401/407 was seen
|
||||
|
||||
// Decoded, path-stripped; NOT filesystem-sanitized (rules/ owns that, stage 9).
|
||||
std::string filename_from_disposition;
|
||||
std::string filename_from_url;
|
||||
ContentDisposition::Type disposition_type = ContentDisposition::Type::none;
|
||||
};
|
||||
|
||||
// Resolution order (docs/04 §2.5): explicit user name -> Content-Disposition -> URL path
|
||||
// segment -> "download.bin". Only a light path/control strip here; rules/ does the
|
||||
// authoritative sanitize, byte cap, and collision handling.
|
||||
[[nodiscard]] std::string suggest_filename(const ProbeResult &r,
|
||||
std::string_view explicit_name = {});
|
||||
|
||||
class Prober {
|
||||
public:
|
||||
// max_concurrent bounds outstanding probe transfers; the rest queue.
|
||||
explicit Prober(unsigned max_concurrent = 4);
|
||||
~Prober();
|
||||
Prober(const Prober &) = delete;
|
||||
Prober &operator=(const Prober &) = delete;
|
||||
|
||||
// Async. `done` runs on an internal worker thread, exactly once. A 401/407 is a
|
||||
// SUCCESS result with requires_auth = true (the GUI collects credentials), not an
|
||||
// error; transport failures and hard HTTP errors (404/410/5xx) are errors.
|
||||
void probe(ProbeRequest req, std::function<void(Result<ProbeResult>)> done);
|
||||
|
||||
private:
|
||||
struct Impl;
|
||||
std::unique_ptr<Impl> impl_;
|
||||
};
|
||||
|
||||
} // namespace vdm::net
|
||||
|
||||
#endif // VDM_NET_PROBE_HPP
|
||||
@@ -0,0 +1,40 @@
|
||||
// vdm/net/url.hpp — a small, total URL splitter.
|
||||
//
|
||||
// Not a full RFC 3986 parser (libcurl does the real fetching); just enough to pull a
|
||||
// filename out of a path and to sanity-check a scheme. Total on hostile input — it has a
|
||||
// fuzz target (tools/fuzz) — never throws, never asserts.
|
||||
//
|
||||
// This header compiles standalone.
|
||||
|
||||
#ifndef VDM_NET_URL_HPP
|
||||
#define VDM_NET_URL_HPP
|
||||
|
||||
#include <cstdint>
|
||||
#include <optional>
|
||||
#include <string>
|
||||
#include <string_view>
|
||||
|
||||
namespace vdm::net {
|
||||
|
||||
struct SplitUrl {
|
||||
std::string scheme; // lowercased, without "://"
|
||||
std::string userinfo; // before '@', if any
|
||||
std::string host; // lowercased; bracketed IPv6 keeps its brackets stripped
|
||||
std::optional<std::uint16_t> port;
|
||||
std::string path; // includes the leading '/', or empty
|
||||
std::string query; // without the '?'
|
||||
std::string fragment; // without the '#'
|
||||
bool valid = false; // scheme + host present and scheme is http/https
|
||||
|
||||
[[nodiscard]] bool is_http() const noexcept { return scheme == "http" || scheme == "https"; }
|
||||
};
|
||||
|
||||
[[nodiscard]] SplitUrl split_url(std::string_view url);
|
||||
|
||||
// The last non-empty path segment, percent-decoded, path components stripped. Empty when
|
||||
// the path has no usable segment (ends in '/', is empty, or is only "/").
|
||||
[[nodiscard]] std::string url_filename(std::string_view url);
|
||||
|
||||
} // namespace vdm::net
|
||||
|
||||
#endif // VDM_NET_URL_HPP
|
||||
@@ -0,0 +1,186 @@
|
||||
// vdm/rate/token_bucket.hpp — a lazily-refilled token bucket, and the global -> queue ->
|
||||
// task limiter hierarchy built on it (docs/04 §6).
|
||||
//
|
||||
// A segment worker calls RateLimiter::acquire(task, n) after receiving n body bytes. If
|
||||
// every applicable level (task, its queue, global) has n tokens, it consumes n from each
|
||||
// and returns 0. Otherwise it consumes nothing and returns how long to wait before
|
||||
// retrying — the worker returns CURL_WRITEFUNC_PAUSE and schedules a curl_easy_pause
|
||||
// resume after that delay (the "precision" layer on top of CURLOPT_MAX_RECV_SPEED_LARGE).
|
||||
//
|
||||
// This header compiles standalone.
|
||||
|
||||
#ifndef VDM_RATE_TOKEN_BUCKET_HPP
|
||||
#define VDM_RATE_TOKEN_BUCKET_HPP
|
||||
|
||||
#include <algorithm>
|
||||
#include <chrono>
|
||||
#include <cstdint>
|
||||
#include <mutex>
|
||||
#include <optional>
|
||||
#include <unordered_map>
|
||||
|
||||
#include "vdm/ids.hpp"
|
||||
|
||||
namespace vdm::rate {
|
||||
|
||||
// rate_bps == 0 means unlimited: consume() always succeeds and never waits.
|
||||
class TokenBucket {
|
||||
public:
|
||||
TokenBucket() = default;
|
||||
// `burst` caps how many tokens accumulate while idle; 0 => 1 second's worth.
|
||||
explicit TokenBucket(std::uint64_t rate_bps, std::uint64_t burst = 0) {
|
||||
set_rate(rate_bps, burst);
|
||||
}
|
||||
|
||||
void set_rate(std::uint64_t rate_bps, std::uint64_t burst = 0) {
|
||||
std::lock_guard lk(mu_);
|
||||
const bool was_unlimited = rate_ == 0;
|
||||
rate_ = rate_bps;
|
||||
cap_ = burst ? burst : rate_bps; // 1 s of burst by default
|
||||
// A freshly-limited bucket starts full: you may transfer at burst speed
|
||||
// immediately, then it throttles (classic token bucket / IDM behaviour). Lowering
|
||||
// an existing limit only clamps down — it never hands out a fresh burst.
|
||||
if (rate_bps > 0 && was_unlimited)
|
||||
tokens_ = cap_;
|
||||
else if (tokens_ > cap_)
|
||||
tokens_ = cap_;
|
||||
last_ = clock::now();
|
||||
}
|
||||
[[nodiscard]] std::uint64_t rate() const {
|
||||
std::lock_guard lk(mu_);
|
||||
return rate_;
|
||||
}
|
||||
|
||||
// Consume `n` if available; otherwise consume nothing. Returns the wait until `n`
|
||||
// tokens *would* be available (0 when it consumed).
|
||||
[[nodiscard]] std::chrono::nanoseconds consume(std::uint64_t n) {
|
||||
std::lock_guard lk(mu_);
|
||||
if (rate_ == 0)
|
||||
return {};
|
||||
refill_locked();
|
||||
if (tokens_ >= n) {
|
||||
tokens_ -= n;
|
||||
return {};
|
||||
}
|
||||
const std::uint64_t deficit = n - tokens_;
|
||||
// ns to earn `deficit` tokens at rate_ bytes/s
|
||||
return std::chrono::nanoseconds{
|
||||
static_cast<std::int64_t>((deficit * 1'000'000'000ull + rate_ - 1) / rate_)};
|
||||
}
|
||||
|
||||
// Two-phase for the hierarchy: check every level, then commit on all or none.
|
||||
[[nodiscard]] std::chrono::nanoseconds peek(std::uint64_t n) {
|
||||
std::lock_guard lk(mu_);
|
||||
if (rate_ == 0)
|
||||
return {};
|
||||
refill_locked();
|
||||
if (tokens_ >= n)
|
||||
return {};
|
||||
const std::uint64_t deficit = n - tokens_;
|
||||
return std::chrono::nanoseconds{
|
||||
static_cast<std::int64_t>((deficit * 1'000'000'000ull + rate_ - 1) / rate_)};
|
||||
}
|
||||
void commit(std::uint64_t n) {
|
||||
std::lock_guard lk(mu_);
|
||||
if (rate_ == 0)
|
||||
return;
|
||||
tokens_ = tokens_ >= n ? tokens_ - n : 0;
|
||||
}
|
||||
|
||||
private:
|
||||
using clock = std::chrono::steady_clock;
|
||||
void refill_locked() {
|
||||
auto now = clock::now();
|
||||
auto dt = std::chrono::duration_cast<std::chrono::nanoseconds>(now - last_).count();
|
||||
if (dt <= 0)
|
||||
return;
|
||||
last_ = now;
|
||||
// added = rate_ * dt / 1e9, guarding overflow for very long idle gaps
|
||||
long double added =
|
||||
static_cast<long double>(rate_) * static_cast<long double>(dt) / 1'000'000'000.0L;
|
||||
std::uint64_t add =
|
||||
added >= static_cast<long double>(cap_) ? cap_ : static_cast<std::uint64_t>(added);
|
||||
tokens_ = tokens_ + add > cap_ ? cap_ : tokens_ + add;
|
||||
}
|
||||
|
||||
mutable std::mutex mu_;
|
||||
std::uint64_t rate_ = 0;
|
||||
std::uint64_t cap_ = 0;
|
||||
std::uint64_t tokens_ = 0;
|
||||
clock::time_point last_ = clock::now();
|
||||
};
|
||||
|
||||
// The hierarchy. All limits default to 0 (unlimited). A task with no queue is limited by
|
||||
// task + global only.
|
||||
class RateLimiter {
|
||||
public:
|
||||
void set_global_limit(std::uint64_t bps) { global_.set_rate(bps); }
|
||||
[[nodiscard]] std::uint64_t global_limit() const { return global_.rate(); }
|
||||
|
||||
void set_queue_limit(QueueId q, std::uint64_t bps) {
|
||||
std::lock_guard lk(mu_);
|
||||
queues_[q].set_rate(bps);
|
||||
}
|
||||
void set_task_limit(TaskId t, std::uint64_t bps) {
|
||||
std::lock_guard lk(mu_);
|
||||
tasks_[t].set_rate(bps);
|
||||
}
|
||||
|
||||
void attach_task(TaskId t, std::optional<QueueId> q) {
|
||||
std::lock_guard lk(mu_);
|
||||
tasks_.try_emplace(t);
|
||||
if (q) {
|
||||
task_queue_[t] = *q;
|
||||
queues_.try_emplace(*q);
|
||||
} else {
|
||||
task_queue_.erase(t);
|
||||
}
|
||||
}
|
||||
void detach_task(TaskId t) {
|
||||
std::lock_guard lk(mu_);
|
||||
tasks_.erase(t);
|
||||
task_queue_.erase(t);
|
||||
}
|
||||
|
||||
// Consume `n` bytes against task, queue and global. 0 => consumed everywhere. > 0 =>
|
||||
// consumed nowhere; wait that long and retry. Held under mu_ for its whole duration
|
||||
// so a concurrent detach_task() can't invalidate the bucket it is using.
|
||||
[[nodiscard]] std::chrono::nanoseconds acquire(TaskId t, std::uint64_t n) {
|
||||
std::lock_guard lk(mu_);
|
||||
TokenBucket *tb = nullptr;
|
||||
TokenBucket *qb = nullptr;
|
||||
if (auto it = tasks_.find(t); it != tasks_.end())
|
||||
tb = &it->second;
|
||||
if (auto qit = task_queue_.find(t); qit != task_queue_.end())
|
||||
if (auto q = queues_.find(qit->second); q != queues_.end())
|
||||
qb = &q->second;
|
||||
|
||||
// peek all, then commit all or none — no level "leaks" tokens on a partial miss.
|
||||
std::chrono::nanoseconds wait{};
|
||||
if (tb)
|
||||
wait = std::max(wait, tb->peek(n));
|
||||
if (qb)
|
||||
wait = std::max(wait, qb->peek(n));
|
||||
wait = std::max(wait, global_.peek(n));
|
||||
if (wait.count() > 0)
|
||||
return wait;
|
||||
|
||||
if (tb)
|
||||
tb->commit(n);
|
||||
if (qb)
|
||||
qb->commit(n);
|
||||
global_.commit(n);
|
||||
return {};
|
||||
}
|
||||
|
||||
private:
|
||||
mutable std::mutex mu_; // guards the maps AND serialises acquire()
|
||||
TokenBucket global_;
|
||||
std::unordered_map<QueueId, TokenBucket> queues_;
|
||||
std::unordered_map<TaskId, TokenBucket> tasks_;
|
||||
std::unordered_map<TaskId, QueueId> task_queue_;
|
||||
};
|
||||
|
||||
} // namespace vdm::rate
|
||||
|
||||
#endif // VDM_RATE_TOKEN_BUCKET_HPP
|
||||
@@ -0,0 +1,137 @@
|
||||
// vdm/segment/budget.hpp — the global segment allocator (ADR 0011).
|
||||
//
|
||||
// One instance per engine. It owns exactly one ceiling — `maxActiveSegments`, in segment
|
||||
// units — and the min-1-before-seconds fairness rule (ADR 0011 §3). DAEMON's scheduler
|
||||
// counts tasks and never touches this except through the read-outs and setters below;
|
||||
// CORE's download_task (stage 8) drives the task-facing half.
|
||||
//
|
||||
// Fairness (two-pass, recomputed on every edge): a guarantee pass gives every task that
|
||||
// wants a slot and holds zero exactly one, in DAEMON's priority order; then a growth pass
|
||||
// round-robins the remainder up to each task's effective cap. A task's target can drop
|
||||
// below what it holds (a lower-priority task shedding for a higher-priority arrival, or a
|
||||
// live `set_max_active_segments` cut) — the task then *yields*: it releases a slot at its
|
||||
// next segment boundary, never mid-segment (ADR 0011 A1). A finishing worker whose target
|
||||
// still covers it *steals* instead (slot-neutral). That steal-vs-yield choice lives in
|
||||
// stage 8, driven by comparing this budget's target to the task's live worker count.
|
||||
//
|
||||
// This header compiles standalone.
|
||||
|
||||
#ifndef VDM_SEGMENT_BUDGET_HPP
|
||||
#define VDM_SEGMENT_BUDGET_HPP
|
||||
|
||||
#include <chrono>
|
||||
#include <condition_variable>
|
||||
#include <cstdint>
|
||||
#include <functional>
|
||||
#include <mutex>
|
||||
#include <optional>
|
||||
#include <span>
|
||||
#include <string>
|
||||
#include <thread>
|
||||
#include <unordered_map>
|
||||
#include <vector>
|
||||
|
||||
#include "vdm/ids.hpp"
|
||||
|
||||
namespace vdm::segment {
|
||||
|
||||
class SegmentBudget {
|
||||
public:
|
||||
struct EngineBudget {
|
||||
std::uint32_t total = 0; // == maxActiveSegments
|
||||
std::uint32_t active = 0; // slots held across all tasks
|
||||
std::uint32_t tasks_starved = 0; // running tasks holding zero slots (ADR 0011 §3.6)
|
||||
bool operator==(const EngineBudget &) const = default;
|
||||
};
|
||||
|
||||
struct Options {
|
||||
std::uint32_t max_active_segments = 32;
|
||||
std::chrono::milliseconds notify_period{250}; // <=4 Hz, per event.task.progress
|
||||
};
|
||||
SegmentBudget(); // default Options
|
||||
explicit SegmentBudget(Options opts);
|
||||
~SegmentBudget();
|
||||
SegmentBudget(const SegmentBudget &) = delete;
|
||||
SegmentBudget &operator=(const SegmentBudget &) = delete;
|
||||
|
||||
// ---- task-facing (download_task, stage 8) -------------------------------------------
|
||||
struct TaskParams {
|
||||
std::string host; // key for the per-host segment cap
|
||||
std::uint32_t per_task_cap = 1; // min(spec.segments ?? maxSegmentsPerDownload, 32)
|
||||
bool resumable = false; // false => effective cap forced to 1
|
||||
};
|
||||
// Called with the new absolute slot target for the task. Runs on a budget thread (or
|
||||
// the caller's, for the edge case) — must not block and must not re-enter the budget
|
||||
// beyond confirm_slot()/release_slot(). The task starts or yields workers to match.
|
||||
using SlotTargetFn = std::function<void(std::uint32_t target)>;
|
||||
|
||||
void register_task(TaskId id, const TaskParams ¶ms, SlotTargetFn on_target);
|
||||
void deregister_task(TaskId id); // pause / complete / cancel
|
||||
|
||||
// How many slots the task could use right now (0 .. per_task_cap): incomplete
|
||||
// segments it has range for. 0 while retry_wait / assembling / verifying / paused —
|
||||
// which is exactly why those states are never counted as starvation.
|
||||
void set_want(TaskId id, std::uint32_t want);
|
||||
|
||||
// A worker actually started on a granted slot. false => the target was cut in the
|
||||
// race and the worker must not start.
|
||||
[[nodiscard]] bool confirm_slot(TaskId id);
|
||||
// A held slot is free: segment complete with no steal, failed, paused, or yielded.
|
||||
void release_slot(TaskId id);
|
||||
|
||||
// ---- DAEMON-facing (sched/) -------------------------------------------------------
|
||||
void set_max_active_segments(std::uint32_t n); // drain-not-kill (ADR 0011 §2)
|
||||
void set_host_segment_cap(std::string host, std::uint32_t cap); // 0 clears
|
||||
void set_task_order(std::span<const TaskId> priority_order); // pushed on change
|
||||
|
||||
[[nodiscard]] EngineBudget budget() const;
|
||||
[[nodiscard]] std::uint32_t segments_active(TaskId id) const;
|
||||
[[nodiscard]] std::vector<TaskId> starved_tasks() const;
|
||||
[[nodiscard]] std::optional<SteadyTime> starved_since(TaskId id) const;
|
||||
void on_budget_changed(std::function<void(EngineBudget)> cb);
|
||||
|
||||
private:
|
||||
struct Task {
|
||||
std::string host;
|
||||
std::uint32_t per_task_cap = 1;
|
||||
bool resumable = false;
|
||||
std::uint32_t want = 0;
|
||||
std::uint32_t held = 0;
|
||||
std::uint32_t target = 0; // last published
|
||||
SlotTargetFn on_target;
|
||||
std::optional<SteadyTime> starved_since;
|
||||
};
|
||||
|
||||
// A unit of deferred work: callbacks are copied out here so the public entry points
|
||||
// can invoke them AFTER dropping mu_.
|
||||
struct Plan {
|
||||
std::vector<std::pair<SlotTargetFn, std::uint32_t>> targets;
|
||||
std::optional<std::pair<std::function<void(EngineBudget)>, EngineBudget>> notify_now;
|
||||
};
|
||||
|
||||
Plan reallocate_locked();
|
||||
static void run(Plan &p);
|
||||
[[nodiscard]] std::uint32_t effective_cap_locked(const Task &t) const;
|
||||
[[nodiscard]] EngineBudget snapshot_locked() const;
|
||||
void notifier_loop(std::stop_token st);
|
||||
|
||||
mutable std::mutex mu_;
|
||||
std::unordered_map<TaskId, Task> tasks_;
|
||||
std::vector<TaskId> order_;
|
||||
std::unordered_map<std::string, std::uint32_t> host_caps_;
|
||||
std::uint32_t max_active_;
|
||||
std::uint32_t active_ = 0;
|
||||
|
||||
std::function<void(EngineBudget)> on_changed_;
|
||||
EngineBudget last_notified_;
|
||||
std::uint32_t last_starved_ = 0;
|
||||
bool dirty_ = false;
|
||||
|
||||
std::chrono::milliseconds notify_period_;
|
||||
std::condition_variable notify_cv_;
|
||||
std::jthread notifier_;
|
||||
};
|
||||
|
||||
} // namespace vdm::segment
|
||||
|
||||
#endif // VDM_SEGMENT_BUDGET_HPP
|
||||
@@ -0,0 +1,178 @@
|
||||
// vdm/segment/segmenter.hpp — per-download range management and dynamic segment stealing.
|
||||
//
|
||||
// docs/04 §3. Owns the split of [0, total_size) for one download: the initial layout, a
|
||||
// split when a slot is granted (`assign_slot`), a *steal* when a worker finishes and may
|
||||
// keep its slot (`on_complete` — take the second half of the largest remaining range),
|
||||
// and a re-split of an orphaned range when a segment fails 3x on the same host with a
|
||||
// mirror available (`on_failed` -> requeue).
|
||||
//
|
||||
// Thread model: one mutex — the segmenter's — *is* "the task lock" (docs/04 §3: the steal
|
||||
// is "atomic under the task lock"). Every method takes it. `advance()` and the per-worker
|
||||
// accessors are called once per buffer flush (a few Hz per segment), not from the curl
|
||||
// write callback, so a lock there is free; the no-lock/no-alloc rule is about that
|
||||
// callback and its ring buffer. Segment fields stay std::atomic so the store type is
|
||||
// trivially relocatable and reads never tear. The store is a std::deque so a steal's
|
||||
// push_back never moves an existing record.
|
||||
//
|
||||
// This header compiles standalone.
|
||||
|
||||
#ifndef VDM_SEGMENT_SEGMENTER_HPP
|
||||
#define VDM_SEGMENT_SEGMENTER_HPP
|
||||
|
||||
#include <atomic>
|
||||
#include <cstdint>
|
||||
#include <deque>
|
||||
#include <mutex>
|
||||
#include <optional>
|
||||
#include <vector>
|
||||
|
||||
namespace vdm::segment {
|
||||
|
||||
inline constexpr std::uint64_t kDefaultMinSegmentBytes = 1u << 20; // 1 MiB (docs/04 §3)
|
||||
inline constexpr std::uint32_t kDefaultSegments = 8;
|
||||
inline constexpr std::uint32_t kMaxSegments = 32;
|
||||
|
||||
enum class SegState : std::uint8_t {
|
||||
idle, // range assigned, no worker connected yet
|
||||
connecting,
|
||||
downloading,
|
||||
stalled, // low-speed; still holds its slot
|
||||
complete,
|
||||
failed, // gave up (orphaned; range requeued or lost to a steal)
|
||||
};
|
||||
|
||||
// A flat, copyable view of one segment record. Ranges are absolute byte offsets,
|
||||
// **inclusive** on both ends (contract Segment.endByte / ADR 0010).
|
||||
struct SegmentView {
|
||||
std::uint32_t index = 0;
|
||||
std::uint64_t start = 0;
|
||||
std::uint64_t end = 0;
|
||||
std::uint64_t completed = 0;
|
||||
SegState state = SegState::idle;
|
||||
std::uint32_t consecutive_failures = 0;
|
||||
|
||||
[[nodiscard]] std::uint64_t length() const noexcept {
|
||||
return end >= start ? end - start + 1 : 0;
|
||||
}
|
||||
[[nodiscard]] std::uint64_t remaining() const noexcept {
|
||||
return length() - (completed < length() ? completed : length());
|
||||
}
|
||||
bool operator==(const SegmentView &) const = default;
|
||||
};
|
||||
|
||||
// One resumed range, as read back from .veloxpart.meta (kept independent of meta/ so this
|
||||
// header stands alone).
|
||||
struct ResumedRange {
|
||||
std::uint64_t start = 0;
|
||||
std::uint64_t end = 0;
|
||||
std::uint64_t completed = 0;
|
||||
};
|
||||
|
||||
enum class FailAction {
|
||||
retry, // same range, backoff (owned by the task/state machine)
|
||||
requeue, // 3x connection failure + a mirror exists: orphan the remaining range and
|
||||
// re-split it; the segment's slot is released
|
||||
};
|
||||
|
||||
class Segmenter {
|
||||
public:
|
||||
// total_size 0 => unknown (chunked): forces a single segment. resumable == false also
|
||||
// forces a single segment (docs/04 §3: "Non-resumable servers -> exactly 1 segment").
|
||||
Segmenter(std::uint64_t total_size, std::uint32_t requested_segments, bool resumable,
|
||||
std::uint64_t min_segment_bytes = kDefaultMinSegmentBytes);
|
||||
|
||||
// Resume: rebuild from a persisted segment table. Ranges must tile [0, total_size)
|
||||
// with no gap or overlap; a malformed table falls back to a single segment.
|
||||
Segmenter(std::uint64_t total_size, std::uint32_t requested_segments,
|
||||
const std::vector<ResumedRange> &resumed, bool resumable,
|
||||
std::uint64_t min_segment_bytes = kDefaultMinSegmentBytes);
|
||||
|
||||
Segmenter(const Segmenter &) = delete;
|
||||
Segmenter &operator=(const Segmenter &) = delete;
|
||||
|
||||
// The count this download would use with an unlimited budget. 1 when non-resumable /
|
||||
// unknown size; otherwise min(requested, floor(total / min_segment_bytes), 32).
|
||||
[[nodiscard]] std::uint32_t target_segment_count() const noexcept { return target_count_; }
|
||||
[[nodiscard]] std::uint64_t total_size() const noexcept { return total_size_; }
|
||||
|
||||
// Give a worker something to download. Called when the budget grants a slot. Prefers
|
||||
// an orphaned range from a requeue; otherwise splits the largest remaining range in
|
||||
// half and hands back the tail. Returns nullopt when the target count is already met
|
||||
// or nothing splits to >= min_segment_bytes. `state` of the returned segment is
|
||||
// `connecting`.
|
||||
[[nodiscard]] std::optional<std::uint32_t> assign_slot();
|
||||
|
||||
// A worker finished its range. If `may_steal` is false the slot is being yielded —
|
||||
// returns nullopt and the caller releases the slot to the budget. If true and a
|
||||
// remaining range splits to >= min_segment_bytes, steals its second half: returns a
|
||||
// NEW segment index for the same worker to continue on (slot-neutral). Otherwise
|
||||
// nullopt (nothing to steal -> release).
|
||||
[[nodiscard]] std::optional<std::uint32_t> on_complete(std::uint32_t idx, bool may_steal);
|
||||
|
||||
// A worker's segment errored. `connection_error` distinguishes a transport failure
|
||||
// (reset/timeout/refused) from an HTTP/content one. Returns requeue only on the 3rd
|
||||
// consecutive connection error when `has_mirror`; on requeue the remaining range is
|
||||
// orphaned for assign_slot() to re-split and the segment is marked failed.
|
||||
FailAction on_failed(std::uint32_t idx, bool connection_error, bool has_mirror);
|
||||
|
||||
// A successful (re)connection resets the consecutive-failure counter.
|
||||
void note_connected(std::uint32_t idx);
|
||||
|
||||
// Progress from the write path. Lock-free. `bytes` is the absolute completed count
|
||||
// within the segment; clamped to the segment length.
|
||||
void advance(std::uint32_t idx, std::uint64_t bytes) noexcept;
|
||||
|
||||
// Per-segment fields for a worker. A worker reads `segment_end` before every write so
|
||||
// a concurrent steal that shrank its range stops it cleanly. `segment_start` never
|
||||
// changes for a given index.
|
||||
[[nodiscard]] std::uint64_t segment_start(std::uint32_t idx) const noexcept;
|
||||
[[nodiscard]] std::uint64_t segment_end(std::uint32_t idx) const noexcept;
|
||||
[[nodiscard]] std::uint64_t segment_completed(std::uint32_t idx) const noexcept;
|
||||
[[nodiscard]] SegState segment_state(std::uint32_t idx) const noexcept;
|
||||
void set_segment_state(std::uint32_t idx, SegState s) noexcept;
|
||||
|
||||
// Hand a segment back to the pool without touching its `completed`: it becomes an
|
||||
// unassigned idle range that the next assign_slot() picks up (used on pause).
|
||||
void release_segment(std::uint32_t idx) noexcept;
|
||||
|
||||
// Sum of bytes done across every segment (active + already complete). Locks.
|
||||
[[nodiscard]] std::uint64_t downloaded() const;
|
||||
[[nodiscard]] bool all_complete() const;
|
||||
|
||||
// Every segment record, for TaskDetail projection and the .veloxpart.meta writer.
|
||||
[[nodiscard]] std::vector<SegmentView> snapshot() const;
|
||||
|
||||
private:
|
||||
struct Seg {
|
||||
std::uint32_t index;
|
||||
std::uint64_t start;
|
||||
std::atomic<std::uint64_t> end;
|
||||
std::atomic<std::uint64_t> completed{0};
|
||||
std::atomic<SegState> state{SegState::idle};
|
||||
std::uint32_t consecutive_failures = 0;
|
||||
bool assigned = false; // a worker holds this segment right now
|
||||
|
||||
Seg(std::uint32_t i, std::uint64_t s, std::uint64_t e) : index(i), start(s), end(e) {}
|
||||
};
|
||||
|
||||
void compute_target(std::uint32_t requested);
|
||||
std::uint32_t add_seg_locked(std::uint64_t start, std::uint64_t end, std::uint64_t completed,
|
||||
SegState state, bool assigned);
|
||||
std::uint32_t split_largest_remaining_locked(); // returns new index, or UINT32_MAX
|
||||
[[nodiscard]] std::uint64_t remaining_of_locked(const Seg &s) const noexcept;
|
||||
[[nodiscard]] std::uint32_t assigned_count_locked() const noexcept;
|
||||
|
||||
mutable std::mutex mu_;
|
||||
std::deque<Seg> segs_;
|
||||
std::vector<ResumedRange> orphans_; // requeued ranges awaiting re-split
|
||||
|
||||
std::uint64_t total_size_;
|
||||
std::uint64_t min_seg_;
|
||||
bool resumable_;
|
||||
std::uint32_t target_count_ = 1;
|
||||
std::uint32_t next_index_ = 0;
|
||||
};
|
||||
|
||||
} // namespace vdm::segment
|
||||
|
||||
#endif // VDM_SEGMENT_SEGMENTER_HPP
|
||||
@@ -0,0 +1,223 @@
|
||||
// vdm/task/download.hpp — the public download API: what DAEMON hands the engine and how
|
||||
// the engine reports back. REVIEW SKETCH (stage 7 pre-work) — value types are final
|
||||
// enough to build against; Engine/DownloadHandle bodies land in stage 8.
|
||||
//
|
||||
// See core/docs/engine-api-m1.md for the threading, lifetime, and pause/resume/cancel
|
||||
// contract that goes with these signatures.
|
||||
//
|
||||
// This header compiles standalone.
|
||||
|
||||
#ifndef VDM_TASK_DOWNLOAD_HPP
|
||||
#define VDM_TASK_DOWNLOAD_HPP
|
||||
|
||||
#include <chrono>
|
||||
#include <cstdint>
|
||||
#include <functional>
|
||||
#include <memory>
|
||||
#include <optional>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vdm/ids.hpp"
|
||||
#include "vdm/net/http_types.hpp"
|
||||
#include "vdm/net/probe.hpp"
|
||||
#include "vdm/segment/segmenter.hpp"
|
||||
#include "vdm/util/error.hpp"
|
||||
#include "vdm/util/result.hpp"
|
||||
|
||||
namespace vdm {
|
||||
class Engine; // owns and fills DownloadHandle (see vdm/engine.hpp)
|
||||
} // namespace vdm
|
||||
|
||||
namespace vdm::task {
|
||||
|
||||
// The task control block. Opaque: defined only in the engine's translation unit. A handle
|
||||
// holds a shared_ptr to one; the engine keeps its own copy so the task outlives a caller
|
||||
// that drops its handle.
|
||||
struct DownloadTaskState;
|
||||
|
||||
// --- input ---------------------------------------------------------------------------
|
||||
|
||||
struct Checksum {
|
||||
enum class Algo { md5, sha1, sha256, sha512 }; // matches the wire Checksum set
|
||||
Algo algo = Algo::sha256;
|
||||
std::string hex; // lower-case, no separators
|
||||
};
|
||||
|
||||
// Everything the engine needs to run ONE download. DAEMON has already run the rules
|
||||
// engine, canonicalised the path, checked it against the allowed roots, resolved the
|
||||
// filename, and created the parent directory — `save_path` is absolute and final and its
|
||||
// directory exists. `<save_path>.veloxpart` and `<save_path>.veloxpart.meta` live beside
|
||||
// it during the transfer; on success the part file is renamed in place. If the directory
|
||||
// is missing at open time the task fails with Error::path_rejected.
|
||||
struct DownloadSpec {
|
||||
std::string url;
|
||||
std::vector<std::string> mirrors; // alternative URLs for the same bytes
|
||||
|
||||
std::vector<net::HeaderField> headers; // the browser's, verbatim
|
||||
std::vector<net::Cookie> cookies;
|
||||
std::string referrer;
|
||||
std::string user_agent;
|
||||
|
||||
std::string save_path; // absolute; the engine never canonicalises or root-checks
|
||||
|
||||
std::optional<std::uint32_t> segments; // requested 1..32; nullopt => engine default
|
||||
std::optional<std::uint64_t> buffer_bytes; // requested per segment; nullopt => default
|
||||
|
||||
net::ProxyConfig proxy;
|
||||
net::AuthConfig auth; // credentials known up front (e.g. from the Secret Service);
|
||||
// leave scheme == none to be prompted on a 401/407
|
||||
|
||||
std::optional<Checksum> checksum; // verified during `verifying`; mismatch => failed
|
||||
|
||||
// DAEMON usually probed already for the File Info dialog. Pass it to skip a second
|
||||
// probe; the engine still revalidates on resume. nullopt => the engine probes.
|
||||
std::optional<net::ProbeResult> probe_hint;
|
||||
|
||||
bool allow_resume = true; // if a valid .veloxpart.meta sits beside save_path, resume
|
||||
// from it; false starts fresh and overwrites
|
||||
std::optional<long> max_retries; // per-segment; nullopt => engine default (10)
|
||||
};
|
||||
|
||||
// --- lifecycle (the CORE-owned subset of the wire TaskState; ADR 0013 §1) -------------
|
||||
|
||||
enum class EngineState {
|
||||
probing,
|
||||
connecting,
|
||||
downloading,
|
||||
paused, // shared with DAEMON; entered by either side, idempotently
|
||||
retry_wait, // the engine's own backoff timer
|
||||
assembling, // no-op rename in M1; a real mux step for HLS/DASH (M4)
|
||||
verifying, // checksum
|
||||
complete, // terminal
|
||||
failed, // terminal
|
||||
cancelled, // terminal; always DAEMON- or user-initiated
|
||||
};
|
||||
|
||||
[[nodiscard]] constexpr bool is_terminal(EngineState s) noexcept {
|
||||
return s == EngineState::complete || s == EngineState::failed || s == EngineState::cancelled;
|
||||
}
|
||||
|
||||
// --- progress ---------------------------------------------------------------------------
|
||||
|
||||
struct SegmentProgress {
|
||||
std::uint32_t index = 0;
|
||||
std::uint64_t start = 0;
|
||||
std::uint64_t end = 0; // inclusive
|
||||
std::uint64_t completed = 0;
|
||||
std::uint64_t speed_bps = 0;
|
||||
segment::SegState state = segment::SegState::idle;
|
||||
};
|
||||
|
||||
struct Progress {
|
||||
std::uint64_t downloaded = 0;
|
||||
std::optional<std::uint64_t> total; // absent for a chunked source until it ends
|
||||
std::uint64_t speed_bps = 0; // aggregate over the last window
|
||||
std::optional<std::uint32_t> eta_seconds;
|
||||
|
||||
std::uint32_t effective_segments = 0; // slots the budget granted (held)
|
||||
std::uint64_t effective_buffer_bytes = 0; // per segment, after the maxTotal clamp
|
||||
std::vector<SegmentProgress> segments;
|
||||
};
|
||||
|
||||
// --- interaction callbacks ----------------------------------------------------------
|
||||
|
||||
// A 401/407. The task has already auto-paused (state -> paused, error == auth_required).
|
||||
// DAEMON collects credentials and calls handle.provide_auth().
|
||||
struct AuthChallenge {
|
||||
std::string host;
|
||||
std::string realm;
|
||||
enum class Scheme { basic, digest, ntlm, negotiate, unknown };
|
||||
Scheme scheme = Scheme::unknown;
|
||||
};
|
||||
|
||||
// The server's copy changed under us (a 200 where a 206 was expected, or an If-Range /
|
||||
// ETag mismatch on resume — docs/04 §5), or the range metadata went stale (416). The
|
||||
// task has auto-paused. DAEMON asks the user and calls handle.decide().
|
||||
struct DecisionRequest {
|
||||
enum class Kind { server_file_changed, range_metadata_stale };
|
||||
Kind kind = Kind::server_file_changed;
|
||||
std::string detail; // human-readable, for the dialog body
|
||||
};
|
||||
|
||||
enum class Decision {
|
||||
restart, // discard the partial file, download again from scratch
|
||||
keep_partial, // trust what is on disk and continue (the user's risk)
|
||||
abort, // give up: the task goes to `failed`
|
||||
};
|
||||
|
||||
struct DownloadOutcome {
|
||||
std::string final_path;
|
||||
std::uint64_t bytes = 0;
|
||||
std::optional<std::string> sha256_hex; // present when a checksum was requested/derived
|
||||
std::chrono::milliseconds elapsed{0};
|
||||
};
|
||||
|
||||
// All callbacks are optional. See core/docs/engine-api-m1.md for the rules; in short:
|
||||
// they arrive on an engine thread, are serialised per task, must not block, and must not
|
||||
// re-enter THIS task's handle synchronously.
|
||||
struct DownloadCallbacks {
|
||||
// Coalesced to <= 4 Hz per task (matches the wire event.task.progress cadence).
|
||||
std::function<void(const Progress &)> on_progress;
|
||||
|
||||
// Every lifecycle transition, including the auto-pauses above (to == paused with a
|
||||
// populated ErrorInfo) and terminals.
|
||||
std::function<void(EngineState from, EngineState to, const std::optional<ErrorInfo> &)>
|
||||
on_state;
|
||||
|
||||
std::function<void(const AuthChallenge &)> on_auth_required;
|
||||
std::function<void(const DecisionRequest &)> on_decision_needed;
|
||||
|
||||
// Fired exactly once, last. Success carries the outcome; failure carries the mapped
|
||||
// ErrorInfo. After it returns the engine makes no further callbacks for this task and
|
||||
// the handle's control methods become no-ops.
|
||||
std::function<void(Result<DownloadOutcome>)> on_finished;
|
||||
};
|
||||
|
||||
// --- the handle -------------------------------------------------------------------------
|
||||
|
||||
// Copyable (shared state). Every method is safe to call from any thread; each posts to
|
||||
// the engine and returns immediately. Dropping the last handle does NOT cancel the task —
|
||||
// call cancel() for that. Bodies land in stage 8.
|
||||
class DownloadHandle {
|
||||
public:
|
||||
DownloadHandle() = default;
|
||||
// The engine builds handles; `DownloadTaskState` is incomplete everywhere else, so
|
||||
// this is effectively engine-only without a friend declaration.
|
||||
explicit DownloadHandle(std::shared_ptr<DownloadTaskState> s) : state_(std::move(s)) {}
|
||||
|
||||
[[nodiscard]] TaskId id() const noexcept;
|
||||
[[nodiscard]] bool valid() const noexcept { return static_cast<bool>(state_); }
|
||||
|
||||
// Idempotent. pause() on an already-paused or terminal task is a no-op (no error);
|
||||
// likewise resume() on a task that is not paused. The resulting state is observed via
|
||||
// on_state / this->state(), never a return value (ADR 0013 §2).
|
||||
void pause();
|
||||
void resume();
|
||||
// Idempotent, terminal. discard_partial also removes the .veloxpart[.meta] files.
|
||||
// Always fires on_state(_, cancelled, nullopt) then on_finished(Err{Error::canceled}),
|
||||
// in that order. `download.cancel` == cancel(false); `download.remove` == cancel(true)
|
||||
// (plus DAEMON's own row/file cleanup).
|
||||
void cancel(bool discard_partial = false);
|
||||
|
||||
// Only act while the task is awaiting the matching input (auto-paused for auth /
|
||||
// decision); otherwise a no-op. `remember` asks DAEMON to persist to the Secret
|
||||
// Service — the engine never stores a credential.
|
||||
void provide_auth(std::string username, std::string password, bool remember);
|
||||
void decide(Decision d);
|
||||
|
||||
// IDM's "Refresh Download Address": swap the URL (e.g. a fresh signed URL) on a live
|
||||
// or paused task without losing progress. Empty `headers` keeps the current ones.
|
||||
void refresh_url(std::string url, std::vector<net::HeaderField> headers = {});
|
||||
|
||||
// Synchronous snapshots — cheap, lock-guarded, safe any time.
|
||||
[[nodiscard]] EngineState state() const;
|
||||
[[nodiscard]] Progress progress() const;
|
||||
|
||||
private:
|
||||
std::shared_ptr<DownloadTaskState> state_;
|
||||
};
|
||||
|
||||
} // namespace vdm::task
|
||||
|
||||
#endif // VDM_TASK_DOWNLOAD_HPP
|
||||
@@ -0,0 +1,48 @@
|
||||
// vdm/util/crc32.hpp — CRC-32 (IEEE 802.3 / zlib polynomial), header-only.
|
||||
//
|
||||
// Used to integrity-check the .veloxpart.meta resume sidecar (docs/04 §5). Standard
|
||||
// reflected CRC-32 with 0xEDB88320, init/xorout 0xFFFFFFFF — byte-compatible with
|
||||
// zlib's crc32() and `cksum -o3` — so the value is reproducible outside this codebase.
|
||||
//
|
||||
// This header compiles standalone.
|
||||
|
||||
#ifndef VDM_UTIL_CRC32_HPP
|
||||
#define VDM_UTIL_CRC32_HPP
|
||||
|
||||
#include <array>
|
||||
#include <cstddef>
|
||||
#include <cstdint>
|
||||
|
||||
#include "vdm/util/bytes.hpp"
|
||||
|
||||
namespace vdm {
|
||||
|
||||
namespace detail {
|
||||
inline constexpr std::array<std::uint32_t, 256> make_crc32_table() {
|
||||
std::array<std::uint32_t, 256> t{};
|
||||
for (std::uint32_t i = 0; i < 256; ++i) {
|
||||
std::uint32_t c = i;
|
||||
for (int k = 0; k < 8; ++k)
|
||||
c = (c & 1u) ? (0xEDB88320u ^ (c >> 1)) : (c >> 1);
|
||||
t[i] = c;
|
||||
}
|
||||
return t;
|
||||
}
|
||||
inline constexpr std::array<std::uint32_t, 256> kCrc32Table = make_crc32_table();
|
||||
} // namespace detail
|
||||
|
||||
// Incremental: pass the previous result back as `seed` to continue over split buffers.
|
||||
[[nodiscard]] inline std::uint32_t crc32_update(std::uint32_t seed, ConstByteSpan data) noexcept {
|
||||
std::uint32_t c = seed ^ 0xFFFFFFFFu;
|
||||
for (std::byte b : data)
|
||||
c = detail::kCrc32Table[(c ^ std::to_integer<std::uint8_t>(b)) & 0xFFu] ^ (c >> 8);
|
||||
return c ^ 0xFFFFFFFFu;
|
||||
}
|
||||
|
||||
[[nodiscard]] inline std::uint32_t crc32(ConstByteSpan data) noexcept {
|
||||
return crc32_update(0u, data);
|
||||
}
|
||||
|
||||
} // namespace vdm
|
||||
|
||||
#endif // VDM_UTIL_CRC32_HPP
|
||||
@@ -0,0 +1,169 @@
|
||||
// vdm/engine.cpp
|
||||
|
||||
#include "vdm/engine.hpp"
|
||||
|
||||
#include <atomic>
|
||||
#include <condition_variable>
|
||||
#include <cstdint>
|
||||
#include <map>
|
||||
#include <mutex>
|
||||
#include <thread>
|
||||
#include <unordered_map>
|
||||
|
||||
#include "task/download_task.hpp"
|
||||
|
||||
namespace vdm {
|
||||
|
||||
struct Engine::Impl : task::TaskHost {
|
||||
explicit Impl(Config c)
|
||||
: cfg_(c),
|
||||
http_(net::HttpClient::Options{.workers = c.http_workers}),
|
||||
prober_(c.probe_pool_size ? c.probe_pool_size : 4),
|
||||
budget_(segment::SegmentBudget::Options{.max_active_segments = c.max_active_segments}) {
|
||||
timer_ = std::jthread([this](std::stop_token st) { timer_loop(st); });
|
||||
}
|
||||
|
||||
~Impl() override {
|
||||
// Quiesce every task first so no callback fires during or after teardown: mark it
|
||||
// retired and cancel its transfers. Then stop the timer thread (a fn already
|
||||
// running holds a shared_ptr and finishes, but seg_finished early-returns on
|
||||
// `retired`). Then drop the registry; http_/prober_/budget_ destruct after.
|
||||
{
|
||||
std::lock_guard lk(reg_mu_);
|
||||
for (auto &[id, t] : tasks_)
|
||||
task::quiesce_task(t);
|
||||
}
|
||||
timer_.request_stop();
|
||||
timer_cv_.notify_all();
|
||||
if (timer_.joinable())
|
||||
timer_.join();
|
||||
{
|
||||
std::lock_guard lk(reg_mu_);
|
||||
tasks_.clear();
|
||||
}
|
||||
}
|
||||
|
||||
// --- TaskHost -------------------------------------------------------------------
|
||||
net::HttpClient &http() override { return http_; }
|
||||
segment::SegmentBudget &budget() override { return budget_; }
|
||||
rate::RateLimiter &limiter() override { return limiter_; }
|
||||
const Config &config() override { return cfg_; }
|
||||
|
||||
task::TimerId schedule(SteadyTime at, std::function<void()> fn) override {
|
||||
task::TimerId id;
|
||||
{
|
||||
std::lock_guard lk(timer_mu_);
|
||||
id = ++next_timer_;
|
||||
timers_.emplace(at, Entry{id, std::move(fn)});
|
||||
}
|
||||
timer_cv_.notify_all();
|
||||
return id;
|
||||
}
|
||||
void cancel_timer(task::TimerId id) override {
|
||||
std::lock_guard lk(timer_mu_);
|
||||
for (auto it = timers_.begin(); it != timers_.end(); ++it)
|
||||
if (it->second.id == id) {
|
||||
timers_.erase(it);
|
||||
return;
|
||||
}
|
||||
}
|
||||
void probe(net::ProbeRequest req, std::function<void(Result<net::ProbeResult>)> done) override {
|
||||
prober_.probe(std::move(req), std::move(done));
|
||||
}
|
||||
void task_retired(TaskId id) override {
|
||||
std::lock_guard lk(reg_mu_);
|
||||
tasks_.erase(id);
|
||||
}
|
||||
|
||||
// --- engine surface ----------------------------------------------------------------
|
||||
task::DownloadHandle start(task::DownloadSpec spec, task::DownloadCallbacks cbs) {
|
||||
TaskId id{++next_id_};
|
||||
auto st = task::create_task(*this, id, std::move(spec), std::move(cbs));
|
||||
{
|
||||
std::lock_guard lk(reg_mu_);
|
||||
tasks_[id] = st;
|
||||
}
|
||||
return task::DownloadHandle(std::move(st));
|
||||
}
|
||||
|
||||
Config cfg_;
|
||||
net::HttpClient http_;
|
||||
net::Prober prober_;
|
||||
segment::SegmentBudget budget_;
|
||||
rate::RateLimiter limiter_;
|
||||
|
||||
std::atomic<std::uint64_t> next_id_{0};
|
||||
|
||||
std::mutex reg_mu_;
|
||||
std::unordered_map<TaskId, std::shared_ptr<task::DownloadTaskState>> tasks_;
|
||||
|
||||
struct Entry {
|
||||
task::TimerId id;
|
||||
std::function<void()> fn;
|
||||
};
|
||||
std::mutex timer_mu_;
|
||||
std::condition_variable timer_cv_;
|
||||
std::multimap<SteadyTime, Entry> timers_;
|
||||
std::atomic<std::uint64_t> next_timer_{0};
|
||||
std::jthread timer_;
|
||||
|
||||
void timer_loop(std::stop_token st) {
|
||||
std::unique_lock lk(timer_mu_);
|
||||
while (!st.stop_requested()) {
|
||||
if (timers_.empty()) {
|
||||
timer_cv_.wait_for(lk, std::chrono::seconds(1));
|
||||
continue;
|
||||
}
|
||||
auto next_at = timers_.begin()->first;
|
||||
if (timer_cv_.wait_until(lk, next_at, [&] {
|
||||
return st.stop_requested() ||
|
||||
(!timers_.empty() && timers_.begin()->first < next_at);
|
||||
})) {
|
||||
continue; // stop, or an earlier timer landed — re-evaluate
|
||||
}
|
||||
// fire everything due
|
||||
std::vector<std::function<void()>> due;
|
||||
auto now = std::chrono::steady_clock::now();
|
||||
for (auto it = timers_.begin(); it != timers_.end() && it->first <= now;)
|
||||
due.push_back(std::move(it->second.fn)), it = timers_.erase(it);
|
||||
lk.unlock();
|
||||
for (auto &fn : due)
|
||||
fn();
|
||||
lk.lock();
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
// --- Engine ---------------------------------------------------------------------------
|
||||
|
||||
Engine::Engine() : Engine(Config{}) {}
|
||||
Engine::Engine(Config cfg) : impl_(std::make_unique<Impl>(cfg)) {}
|
||||
Engine::~Engine() = default;
|
||||
|
||||
task::DownloadHandle Engine::start(task::DownloadSpec spec, task::DownloadCallbacks cbs) {
|
||||
return impl_->start(std::move(spec), std::move(cbs));
|
||||
}
|
||||
|
||||
segment::SegmentBudget &Engine::segment_budget() noexcept {
|
||||
return impl_->budget_;
|
||||
}
|
||||
rate::RateLimiter &Engine::rate_limiter() noexcept {
|
||||
return impl_->limiter_;
|
||||
}
|
||||
|
||||
void Engine::set_default_segments(std::uint32_t n) {
|
||||
impl_->cfg_.default_segments = n ? n : 1;
|
||||
}
|
||||
void Engine::set_default_buffer_bytes(std::uint64_t b) {
|
||||
impl_->cfg_.default_buffer_bytes = b;
|
||||
}
|
||||
void Engine::set_max_total_buffer_bytes(std::uint64_t b) {
|
||||
impl_->cfg_.max_total_buffer_bytes = b;
|
||||
}
|
||||
void Engine::set_probe_pool_size(std::uint32_t) { /* prober pool is fixed at construction in M1 */ }
|
||||
|
||||
void Engine::probe(net::ProbeRequest req, std::function<void(Result<net::ProbeResult>)> done) {
|
||||
impl_->prober_.probe(std::move(req), std::move(done));
|
||||
}
|
||||
|
||||
} // namespace vdm
|
||||
@@ -0,0 +1,186 @@
|
||||
// vdm/io/sparse_file.cpp
|
||||
|
||||
#include "vdm/io/sparse_file.hpp"
|
||||
|
||||
#include <fcntl.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#include <cerrno>
|
||||
#include <cstring>
|
||||
#include <utility>
|
||||
|
||||
namespace vdm::io {
|
||||
namespace {
|
||||
|
||||
Error errno_to_error(int e) noexcept {
|
||||
switch (e) {
|
||||
case ENOSPC:
|
||||
case EDQUOT:
|
||||
return Error::disk_full;
|
||||
case EACCES:
|
||||
case EPERM:
|
||||
case EROFS:
|
||||
return Error::permission_denied;
|
||||
case ENOENT:
|
||||
case ENOTDIR:
|
||||
case EISDIR:
|
||||
case ENAMETOOLONG:
|
||||
case ELOOP:
|
||||
return Error::path_rejected;
|
||||
default:
|
||||
return Error::io_error;
|
||||
}
|
||||
}
|
||||
|
||||
ErrorInfo sys_error(std::string_view what, int e) {
|
||||
return ErrorInfo(errno_to_error(e), std::string(what) + ": " + std::strerror(e));
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
SparseFile::~SparseFile() {
|
||||
if (fd_ >= 0)
|
||||
::close(fd_);
|
||||
}
|
||||
|
||||
SparseFile::SparseFile(SparseFile &&o) noexcept
|
||||
: fd_(std::exchange(o.fd_, -1)),
|
||||
preallocated_(std::exchange(o.preallocated_, false)),
|
||||
path_(std::move(o.path_)) {}
|
||||
|
||||
SparseFile &SparseFile::operator=(SparseFile &&o) noexcept {
|
||||
if (this != &o) {
|
||||
if (fd_ >= 0)
|
||||
::close(fd_);
|
||||
fd_ = std::exchange(o.fd_, -1);
|
||||
preallocated_ = std::exchange(o.preallocated_, false);
|
||||
path_ = std::move(o.path_);
|
||||
}
|
||||
return *this;
|
||||
}
|
||||
|
||||
void SparseFile::reset() noexcept {
|
||||
fd_ = -1;
|
||||
preallocated_ = false;
|
||||
path_.clear();
|
||||
}
|
||||
|
||||
Result<void> SparseFile::open(std::string_view path) {
|
||||
return open(path, OpenOptions{});
|
||||
}
|
||||
|
||||
Result<void> SparseFile::open(std::string_view path, const OpenOptions &opts) {
|
||||
if (fd_ >= 0)
|
||||
return ErrorInfo(Error::internal, "SparseFile already open");
|
||||
|
||||
std::string p(path);
|
||||
// O_NOFOLLOW: the final component of a download target must never be a symlink, on
|
||||
// create or on resume. DAEMON canonicalises the path and checks it against the allowed
|
||||
// roots before start(), but a symlink swapped in afterwards would redirect our writes
|
||||
// outside those roots (daemon/docs/safepath-adversarial.md leans on this open closing
|
||||
// that TOCTOU window). A symlinked leaf fails here with ELOOP -> Error::path_rejected.
|
||||
int flags = O_WRONLY | O_CREAT | O_CLOEXEC | O_NOFOLLOW;
|
||||
if (opts.truncate_existing)
|
||||
flags |= O_TRUNC;
|
||||
|
||||
int fd = ::open(p.c_str(), flags, 0644);
|
||||
if (fd < 0)
|
||||
return sys_error("open " + p, errno);
|
||||
|
||||
bool prealloc = false;
|
||||
if (opts.total_size > 0) {
|
||||
if (opts.preallocate) {
|
||||
// posix_fallocate returns the error number directly and does not set errno.
|
||||
int rc = ::posix_fallocate(fd, 0, static_cast<off_t>(opts.total_size));
|
||||
if (rc == 0) {
|
||||
prealloc = true;
|
||||
} else if (rc == EOPNOTSUPP || rc == ENOSYS || rc == EINVAL) {
|
||||
if (::ftruncate(fd, static_cast<off_t>(opts.total_size)) != 0) {
|
||||
int e = errno;
|
||||
::close(fd);
|
||||
return sys_error("ftruncate " + p, e);
|
||||
}
|
||||
} else {
|
||||
::close(fd);
|
||||
return sys_error("posix_fallocate " + p, rc);
|
||||
}
|
||||
} else if (!opts.truncate_existing) {
|
||||
// Resuming: make sure the file is at least total_size so pwrite offsets land.
|
||||
if (::ftruncate(fd, static_cast<off_t>(opts.total_size)) != 0) {
|
||||
int e = errno;
|
||||
::close(fd);
|
||||
return sys_error("ftruncate " + p, e);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fd_ = fd;
|
||||
preallocated_ = prealloc;
|
||||
path_ = std::move(p);
|
||||
return ok();
|
||||
}
|
||||
|
||||
Result<void> SparseFile::write_at(std::uint64_t offset, ConstByteSpan data) {
|
||||
if (fd_ < 0)
|
||||
return ErrorInfo(Error::internal, "write_at on a closed SparseFile");
|
||||
|
||||
const std::byte *p = data.data();
|
||||
std::size_t remaining = data.size();
|
||||
off_t pos = static_cast<off_t>(offset);
|
||||
|
||||
while (remaining > 0) {
|
||||
ssize_t n = ::pwrite(fd_, p, remaining, pos);
|
||||
if (n < 0) {
|
||||
if (errno == EINTR)
|
||||
continue;
|
||||
return sys_error("pwrite", errno);
|
||||
}
|
||||
if (n == 0)
|
||||
return ErrorInfo(Error::io_error, "pwrite returned 0");
|
||||
p += n;
|
||||
pos += n;
|
||||
remaining -= static_cast<std::size_t>(n);
|
||||
}
|
||||
return ok();
|
||||
}
|
||||
|
||||
Result<void> SparseFile::sync() {
|
||||
if (fd_ < 0)
|
||||
return ErrorInfo(Error::internal, "sync on a closed SparseFile");
|
||||
while (::fdatasync(fd_) != 0) {
|
||||
if (errno == EINTR)
|
||||
continue;
|
||||
return sys_error("fdatasync", errno);
|
||||
}
|
||||
return ok();
|
||||
}
|
||||
|
||||
void SparseFile::advise_dontneed(std::uint64_t offset, std::uint64_t len) noexcept {
|
||||
if (fd_ < 0 || len == 0)
|
||||
return;
|
||||
::posix_fadvise(fd_, static_cast<off_t>(offset), static_cast<off_t>(len), POSIX_FADV_DONTNEED);
|
||||
}
|
||||
|
||||
Result<void> SparseFile::resize(std::uint64_t size) {
|
||||
if (fd_ < 0)
|
||||
return ErrorInfo(Error::internal, "resize on a closed SparseFile");
|
||||
while (::ftruncate(fd_, static_cast<off_t>(size)) != 0) {
|
||||
if (errno == EINTR)
|
||||
continue;
|
||||
return sys_error("ftruncate", errno);
|
||||
}
|
||||
return ok();
|
||||
}
|
||||
|
||||
Result<void> SparseFile::close() {
|
||||
if (fd_ < 0)
|
||||
return ok();
|
||||
int fd = std::exchange(fd_, -1);
|
||||
int rc = ::close(fd);
|
||||
reset();
|
||||
if (rc != 0)
|
||||
return sys_error("close", errno);
|
||||
return ok();
|
||||
}
|
||||
|
||||
} // namespace vdm::io
|
||||
@@ -0,0 +1,57 @@
|
||||
// vdm/io/write_buffer.cpp
|
||||
|
||||
#include "vdm/io/write_buffer.hpp"
|
||||
|
||||
#include <algorithm>
|
||||
#include <cassert>
|
||||
#include <cstring>
|
||||
#include <utility>
|
||||
|
||||
namespace vdm::io {
|
||||
|
||||
WriteBuffer::WriteBuffer(std::uint64_t start_offset, std::size_t capacity, FlushFn flush)
|
||||
: buf_(capacity), base_(start_offset), flush_(std::move(flush)) {
|
||||
assert(capacity > 0 && "WriteBuffer capacity must be > 0");
|
||||
}
|
||||
|
||||
Result<void> WriteBuffer::flush_pending() {
|
||||
if (len_ == 0)
|
||||
return ok();
|
||||
VDM_TRY(flush_(base_, ConstByteSpan(buf_.data(), len_)));
|
||||
base_ += len_;
|
||||
len_ = 0;
|
||||
return ok();
|
||||
}
|
||||
|
||||
Result<void> WriteBuffer::append(ConstByteSpan span) {
|
||||
// On an error return, next_offset() still reflects exactly what is durable; buffered
|
||||
// (non-durable) bytes and any unconsumed tail of `span` are the caller's to abandon —
|
||||
// the segment aborts and resumes/restarts from next_offset().
|
||||
while (!span.empty()) {
|
||||
// Buffer empty and the incoming chunk fills at least a whole buffer: write it
|
||||
// straight through — no memcpy, no allocation, just an extra flush call.
|
||||
if (len_ == 0 && span.size() >= buf_.size()) {
|
||||
VDM_TRY(flush_(base_, span));
|
||||
base_ += span.size();
|
||||
appended_ += span.size();
|
||||
return ok();
|
||||
}
|
||||
|
||||
const std::size_t room = buf_.size() - len_;
|
||||
const std::size_t n = std::min(span.size(), room);
|
||||
std::memcpy(buf_.data() + len_, span.data(), n);
|
||||
len_ += n;
|
||||
appended_ += n;
|
||||
span = span.subspan(n);
|
||||
|
||||
if (len_ == buf_.size())
|
||||
VDM_TRY(flush_pending());
|
||||
}
|
||||
return ok();
|
||||
}
|
||||
|
||||
Result<void> WriteBuffer::flush() {
|
||||
return flush_pending();
|
||||
}
|
||||
|
||||
} // namespace vdm::io
|
||||
@@ -0,0 +1,311 @@
|
||||
// vdm/meta/veloxpart.cpp
|
||||
//
|
||||
// Reader first (AGENT-CORE §5). parse_veloxpart() is the attacker-facing surface; it is
|
||||
// total on any byte string.
|
||||
|
||||
#include "vdm/meta/veloxpart.hpp"
|
||||
|
||||
#include <fcntl.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#include <cerrno>
|
||||
#include <cstring>
|
||||
#include <string>
|
||||
#include <utility>
|
||||
|
||||
#include "vdm/util/crc32.hpp"
|
||||
|
||||
namespace vdm::meta {
|
||||
namespace {
|
||||
|
||||
// magic(4)+ver(2)+flags(2)+total(8)+downloaded(8)+url_count(4)
|
||||
// +etag_len(4)+lm_len(4)+ct_len(4)+seg_count(4)+crc(4)
|
||||
constexpr std::size_t kMinImageBytes = 52;
|
||||
constexpr char kMagic[4] = {'V', 'D', 'M', 'P'};
|
||||
|
||||
Error errno_to_error(int e) noexcept {
|
||||
switch (e) {
|
||||
case ENOSPC:
|
||||
case EDQUOT:
|
||||
return Error::disk_full;
|
||||
case EACCES:
|
||||
case EPERM:
|
||||
case EROFS:
|
||||
return Error::permission_denied;
|
||||
case ENOENT:
|
||||
case ENOTDIR:
|
||||
case EISDIR:
|
||||
case ENAMETOOLONG:
|
||||
case ELOOP:
|
||||
return Error::path_rejected;
|
||||
default:
|
||||
return Error::io_error;
|
||||
}
|
||||
}
|
||||
|
||||
ErrorInfo sys_error(std::string_view what, int e) {
|
||||
return ErrorInfo(errno_to_error(e), std::string(what) + ": " + std::strerror(e));
|
||||
}
|
||||
|
||||
ErrorInfo corrupt(std::string_view where) {
|
||||
return ErrorInfo(Error::meta_corrupt, std::string(".veloxpart.meta: ") + std::string(where));
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
// ---------------------------------------------------------------------------------------
|
||||
// Reader
|
||||
|
||||
Result<VeloxPart> parse_veloxpart(ConstByteSpan image) {
|
||||
if (image.size() > kMaxImageBytes)
|
||||
return corrupt("image exceeds cap");
|
||||
if (image.size() < kMinImageBytes)
|
||||
return corrupt("image shorter than the header");
|
||||
|
||||
// CRC over everything but the trailing u32 — reject before interpreting any field.
|
||||
const ConstByteSpan body = image.first(image.size() - 4);
|
||||
const std::uint32_t want = load_le<std::uint32_t>(image.subspan(image.size() - 4));
|
||||
if (crc32(body) != want)
|
||||
return corrupt("crc32 mismatch");
|
||||
|
||||
ByteReader r(body);
|
||||
|
||||
if (as_chars(r.raw(4)) != std::string_view(kMagic, 4))
|
||||
return corrupt("bad magic");
|
||||
|
||||
VeloxPart vp;
|
||||
vp.version = r.u16();
|
||||
vp.flags = r.u16();
|
||||
if (vp.version > kVersion)
|
||||
return ErrorInfo(Error::meta_version_unsupported,
|
||||
".veloxpart.meta: version " + std::to_string(vp.version) +
|
||||
" > supported " + std::to_string(kVersion));
|
||||
|
||||
vp.total_size = r.u64();
|
||||
vp.downloaded = r.u64();
|
||||
|
||||
const std::uint32_t url_count = r.u32();
|
||||
if (url_count > kMaxUrls)
|
||||
return corrupt("url_count past cap");
|
||||
if (static_cast<std::uint64_t>(url_count) * 4 > r.remaining())
|
||||
return corrupt("url_count");
|
||||
vp.urls.reserve(url_count);
|
||||
for (std::uint32_t i = 0; i < url_count; ++i) {
|
||||
std::string_view s = r.lp_string();
|
||||
if (r.overran() || s.size() > kMaxStringLen)
|
||||
return corrupt("url");
|
||||
vp.urls.emplace_back(s);
|
||||
}
|
||||
|
||||
auto read_str = [&](std::string &dst, std::string_view what) -> Result<void> {
|
||||
std::string_view s = r.lp_string();
|
||||
if (r.overran() || s.size() > kMaxStringLen)
|
||||
return corrupt(what);
|
||||
dst.assign(s);
|
||||
return ok();
|
||||
};
|
||||
VDM_TRY(read_str(vp.etag, "etag"));
|
||||
VDM_TRY(read_str(vp.last_modified, "last_modified"));
|
||||
VDM_TRY(read_str(vp.content_type, "content_type"));
|
||||
|
||||
const std::uint32_t seg_count = r.u32();
|
||||
if (seg_count > kMaxSegments)
|
||||
return corrupt("segment_count past cap");
|
||||
if (static_cast<std::uint64_t>(seg_count) * 24 > r.remaining())
|
||||
return corrupt("segment_count");
|
||||
vp.segments.reserve(seg_count);
|
||||
for (std::uint32_t i = 0; i < seg_count; ++i) {
|
||||
SegmentRecord s;
|
||||
s.start = r.u64();
|
||||
s.end = r.u64();
|
||||
s.completed = r.u64();
|
||||
if (r.overran())
|
||||
return corrupt("segment record");
|
||||
if (s.end >= s.start && s.completed > s.end - s.start + 1)
|
||||
return corrupt("segment.completed exceeds its range");
|
||||
vp.segments.push_back(s);
|
||||
}
|
||||
|
||||
if (vp.flags & kFlagHasShaState) {
|
||||
const std::uint32_t n = r.u32();
|
||||
if (r.overran() || n > kMaxShaStateLen)
|
||||
return corrupt("sha256_state length");
|
||||
ConstByteSpan blob = r.raw(n);
|
||||
if (r.overran())
|
||||
return corrupt("sha256_state body");
|
||||
vp.sha256_state.assign(blob.begin(), blob.end());
|
||||
}
|
||||
|
||||
if (r.overran())
|
||||
return corrupt("truncated");
|
||||
if (r.remaining() != 0)
|
||||
return corrupt("trailing bytes after the record");
|
||||
|
||||
return vp;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------------------
|
||||
// Writer
|
||||
|
||||
std::vector<std::byte> serialize_veloxpart(const VeloxPart &vp) {
|
||||
std::vector<std::byte> out;
|
||||
std::size_t est = kMinImageBytes + vp.segments.size() * 24 + vp.sha256_state.size() + 64;
|
||||
for (const auto &u : vp.urls)
|
||||
est += 4 + u.size();
|
||||
est += vp.etag.size() + vp.last_modified.size() + vp.content_type.size();
|
||||
out.reserve(est);
|
||||
|
||||
auto put_bytes = [&](const void *p, std::size_t n) {
|
||||
const auto *b = static_cast<const std::byte *>(p);
|
||||
out.insert(out.end(), b, b + n);
|
||||
};
|
||||
auto put_u16 = [&](std::uint16_t v) {
|
||||
std::byte t[2];
|
||||
store_le<std::uint16_t>(t, v);
|
||||
put_bytes(t, 2);
|
||||
};
|
||||
auto put_u32 = [&](std::uint32_t v) {
|
||||
std::byte t[4];
|
||||
store_le<std::uint32_t>(t, v);
|
||||
put_bytes(t, 4);
|
||||
};
|
||||
auto put_u64 = [&](std::uint64_t v) {
|
||||
std::byte t[8];
|
||||
store_le<std::uint64_t>(t, v);
|
||||
put_bytes(t, 8);
|
||||
};
|
||||
auto put_str = [&](std::string_view s) {
|
||||
put_u32(static_cast<std::uint32_t>(s.size()));
|
||||
put_bytes(s.data(), s.size());
|
||||
};
|
||||
|
||||
// Normalise the sha-state flag to match the payload so a round-trip is exact.
|
||||
std::uint16_t flags = vp.flags;
|
||||
if (vp.sha256_state.empty())
|
||||
flags &= static_cast<std::uint16_t>(~kFlagHasShaState);
|
||||
else
|
||||
flags |= kFlagHasShaState;
|
||||
|
||||
put_bytes(kMagic, 4);
|
||||
put_u16(vp.version);
|
||||
put_u16(flags);
|
||||
put_u64(vp.total_size);
|
||||
put_u64(vp.downloaded);
|
||||
put_u32(static_cast<std::uint32_t>(vp.urls.size()));
|
||||
for (const auto &u : vp.urls)
|
||||
put_str(u);
|
||||
put_str(vp.etag);
|
||||
put_str(vp.last_modified);
|
||||
put_str(vp.content_type);
|
||||
put_u32(static_cast<std::uint32_t>(vp.segments.size()));
|
||||
for (const auto &s : vp.segments) {
|
||||
put_u64(s.start);
|
||||
put_u64(s.end);
|
||||
put_u64(s.completed);
|
||||
}
|
||||
if (!vp.sha256_state.empty()) {
|
||||
put_u32(static_cast<std::uint32_t>(vp.sha256_state.size()));
|
||||
put_bytes(vp.sha256_state.data(), vp.sha256_state.size());
|
||||
}
|
||||
|
||||
put_u32(crc32(ConstByteSpan(out.data(), out.size())));
|
||||
return out;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------------------
|
||||
// File helpers
|
||||
|
||||
Result<VeloxPart> read_veloxpart_file(std::string_view path) {
|
||||
std::string p(path);
|
||||
int fd = ::open(p.c_str(), O_RDONLY | O_CLOEXEC);
|
||||
if (fd < 0)
|
||||
return sys_error("open " + p, errno);
|
||||
|
||||
std::vector<std::byte> buf;
|
||||
buf.resize(kMaxImageBytes + 1);
|
||||
std::size_t total = 0;
|
||||
for (;;) {
|
||||
ssize_t n = ::read(fd, buf.data() + total, buf.size() - total);
|
||||
if (n < 0) {
|
||||
if (errno == EINTR)
|
||||
continue;
|
||||
int e = errno;
|
||||
::close(fd);
|
||||
return sys_error("read " + p, e);
|
||||
}
|
||||
if (n == 0)
|
||||
break;
|
||||
total += static_cast<std::size_t>(n);
|
||||
if (total > kMaxImageBytes) {
|
||||
::close(fd);
|
||||
return corrupt("sidecar file exceeds cap");
|
||||
}
|
||||
}
|
||||
::close(fd);
|
||||
buf.resize(total);
|
||||
return parse_veloxpart(ConstByteSpan(buf.data(), buf.size()));
|
||||
}
|
||||
|
||||
Result<void> write_veloxpart_file(std::string_view path, const VeloxPart &vp, bool fsync) {
|
||||
std::string p(path);
|
||||
std::string tmp = p + ".tmp";
|
||||
std::vector<std::byte> image = serialize_veloxpart(vp);
|
||||
|
||||
int fd = ::open(tmp.c_str(), O_WRONLY | O_CREAT | O_TRUNC | O_CLOEXEC, 0644);
|
||||
if (fd < 0)
|
||||
return sys_error("open " + tmp, errno);
|
||||
|
||||
const std::byte *pd = image.data();
|
||||
std::size_t remaining = image.size();
|
||||
while (remaining > 0) {
|
||||
ssize_t n = ::write(fd, pd, remaining);
|
||||
if (n < 0) {
|
||||
if (errno == EINTR)
|
||||
continue;
|
||||
int e = errno;
|
||||
::close(fd);
|
||||
::unlink(tmp.c_str());
|
||||
return sys_error("write " + tmp, e);
|
||||
}
|
||||
pd += n;
|
||||
remaining -= static_cast<std::size_t>(n);
|
||||
}
|
||||
|
||||
if (fsync) {
|
||||
while (::fdatasync(fd) != 0) {
|
||||
if (errno == EINTR)
|
||||
continue;
|
||||
int e = errno;
|
||||
::close(fd);
|
||||
::unlink(tmp.c_str());
|
||||
return sys_error("fdatasync " + tmp, e);
|
||||
}
|
||||
}
|
||||
if (::close(fd) != 0) {
|
||||
int e = errno;
|
||||
::unlink(tmp.c_str());
|
||||
return sys_error("close " + tmp, e);
|
||||
}
|
||||
|
||||
if (::rename(tmp.c_str(), p.c_str()) != 0) {
|
||||
int e = errno;
|
||||
::unlink(tmp.c_str());
|
||||
return sys_error("rename " + tmp + " -> " + p, e);
|
||||
}
|
||||
|
||||
if (fsync) {
|
||||
// fsync the directory so the rename itself is durable.
|
||||
std::string dir = p.substr(0, p.find_last_of('/'));
|
||||
if (dir.empty() || dir == p)
|
||||
dir = ".";
|
||||
int dfd = ::open(dir.c_str(), O_RDONLY | O_DIRECTORY | O_CLOEXEC);
|
||||
if (dfd >= 0) {
|
||||
while (::fsync(dfd) != 0 && errno == EINTR) {
|
||||
}
|
||||
::close(dfd);
|
||||
}
|
||||
}
|
||||
return ok();
|
||||
}
|
||||
|
||||
} // namespace vdm::meta
|
||||
@@ -0,0 +1,292 @@
|
||||
// vdm/net/content_disposition.cpp
|
||||
|
||||
#include "vdm/net/content_disposition.hpp"
|
||||
|
||||
#include <algorithm>
|
||||
#include <string>
|
||||
#include <utility>
|
||||
#include <vector>
|
||||
|
||||
#include "net/text_codec.hpp"
|
||||
|
||||
namespace vdm::net {
|
||||
namespace {
|
||||
|
||||
using detail::decode_rfc2047;
|
||||
using detail::latin1_to_utf8;
|
||||
using detail::percent_decode;
|
||||
using detail::to_utf8_best_effort;
|
||||
|
||||
std::string_view trim(std::string_view s) {
|
||||
while (!s.empty() && (s.front() == ' ' || s.front() == '\t'))
|
||||
s.remove_prefix(1);
|
||||
while (!s.empty() &&
|
||||
(s.back() == ' ' || s.back() == '\t' || s.back() == '\r' || s.back() == '\n'))
|
||||
s.remove_suffix(1);
|
||||
return s;
|
||||
}
|
||||
|
||||
std::string ascii_lower(std::string_view s) {
|
||||
std::string r(s);
|
||||
for (char &c : r)
|
||||
if (c >= 'A' && c <= 'Z')
|
||||
c = static_cast<char>(c - 'A' + 'a');
|
||||
return r;
|
||||
}
|
||||
|
||||
// Split "type; a=b; c*=d; e=\"f;g\"" into the type and a param list, honouring quoted
|
||||
// strings (a ';' inside quotes is not a separator). For quoted values the value stored is
|
||||
// the *raw* inner text (quotes removed, escapes NOT yet resolved) with `quoted = true`;
|
||||
// callers resolve escapes and strip path components together (order matters — see
|
||||
// unquote_strip). Keys are lowercased.
|
||||
struct Param {
|
||||
std::string key;
|
||||
std::string value;
|
||||
bool quoted = false;
|
||||
};
|
||||
struct Params {
|
||||
std::string type;
|
||||
std::vector<Param> kv;
|
||||
|
||||
[[nodiscard]] const Param *find(std::string_view key) const {
|
||||
for (const auto &p : kv)
|
||||
if (p.key == key)
|
||||
return &p;
|
||||
return nullptr;
|
||||
}
|
||||
};
|
||||
|
||||
Params tokenize(std::string_view h) {
|
||||
Params out;
|
||||
std::size_t i = 0;
|
||||
const std::size_t n = h.size();
|
||||
|
||||
auto read_segment = [&]() -> std::string_view {
|
||||
std::size_t start = i;
|
||||
bool in_q = false;
|
||||
for (; i < n; ++i) {
|
||||
char c = h[i];
|
||||
if (c == '"') {
|
||||
in_q = !in_q;
|
||||
} else if (c == '\\' && in_q && i + 1 < n) {
|
||||
++i; // skip escaped char
|
||||
} else if (c == ';' && !in_q) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
std::string_view seg = h.substr(start, i - start);
|
||||
if (i < n)
|
||||
++i; // consume ';'
|
||||
return seg;
|
||||
};
|
||||
|
||||
out.type = ascii_lower(trim(read_segment()));
|
||||
|
||||
while (i < n) {
|
||||
std::string_view seg = trim(read_segment());
|
||||
if (seg.empty())
|
||||
continue;
|
||||
auto eq = seg.find('=');
|
||||
if (eq == std::string_view::npos) {
|
||||
out.kv.emplace_back(ascii_lower(seg), std::string{});
|
||||
continue;
|
||||
}
|
||||
std::string key = ascii_lower(trim(seg.substr(0, eq)));
|
||||
std::string_view rawval = trim(seg.substr(eq + 1));
|
||||
|
||||
Param param;
|
||||
param.key = std::move(key);
|
||||
if (rawval.size() >= 2 && rawval.front() == '"') {
|
||||
std::string_view inner = rawval.substr(1);
|
||||
auto close = inner.rfind('"');
|
||||
if (close != std::string_view::npos)
|
||||
inner = inner.substr(0, close);
|
||||
param.value.assign(inner); // raw, escapes unresolved
|
||||
param.quoted = true;
|
||||
} else {
|
||||
param.value.assign(rawval);
|
||||
}
|
||||
out.kv.push_back(std::move(param));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// Drop C0 control bytes and DEL, then trim edge whitespace. NUL and control characters
|
||||
// are never a legitimate part of a filename and are a classic truncation/spoofing vector,
|
||||
// so the decode layer strips them even though rules/ (stage 9) owns the authoritative
|
||||
// sanitize. `..` and other "unsafe but printable" content is left for rules/.
|
||||
std::string sanitize_leaf(std::string s) {
|
||||
std::string out;
|
||||
out.reserve(s.size());
|
||||
for (unsigned char c : s)
|
||||
if (c >= 0x20 && c != 0x7F)
|
||||
out.push_back(static_cast<char>(c));
|
||||
std::string_view v = trim(out);
|
||||
return std::string(v);
|
||||
}
|
||||
|
||||
std::string strip_path(std::string s) {
|
||||
auto slash = s.find_last_of("/\\");
|
||||
if (slash != std::string::npos)
|
||||
s.erase(0, slash + 1);
|
||||
return s;
|
||||
}
|
||||
|
||||
// Resolve ONLY the `\"` escape (needed so a quote can appear mid-name). Every other
|
||||
// backslash is kept literal and later treated as a path separator by strip_path — real
|
||||
// Windows paths in the wild use `\` unescaped, and path-traversal defence matters more
|
||||
// than supporting the vanishingly rare filename with a literal backslash.
|
||||
std::string unescape_dquote(std::string_view raw) {
|
||||
std::string out;
|
||||
out.reserve(raw.size());
|
||||
for (std::size_t i = 0; i < raw.size(); ++i) {
|
||||
if (raw[i] == '\\' && i + 1 < raw.size() && raw[i + 1] == '"') {
|
||||
out.push_back('"');
|
||||
++i;
|
||||
} else {
|
||||
out.push_back(raw[i]);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// Decode an RFC 5987 ext-value: charset'lang'pct-encoded-octets
|
||||
std::string decode_ext_value(std::string_view v) {
|
||||
auto q1 = v.find('\'');
|
||||
if (q1 == std::string_view::npos)
|
||||
return percent_decode(v); // malformed: best effort
|
||||
auto q2 = v.find('\'', q1 + 1);
|
||||
if (q2 == std::string_view::npos)
|
||||
return percent_decode(v.substr(q1 + 1));
|
||||
|
||||
std::string_view charset = v.substr(0, q1);
|
||||
std::string_view enc = v.substr(q2 + 1);
|
||||
std::string bytes = percent_decode(enc);
|
||||
|
||||
std::string cs = ascii_lower(charset);
|
||||
if (cs == "iso-8859-1" || cs == "latin1")
|
||||
return latin1_to_utf8(bytes);
|
||||
return to_utf8_best_effort(bytes); // utf-8 or unknown -> best effort
|
||||
}
|
||||
|
||||
// Reassemble RFC 2231 continuations: name*0*, name*1, name*2* ... in order.
|
||||
std::string join_continuations(const Params &p, std::string_view base, bool &is_ext) {
|
||||
std::vector<std::pair<int, std::string>> parts;
|
||||
is_ext = false;
|
||||
for (const auto ¶m : p.kv) {
|
||||
const std::string &k = param.key;
|
||||
const std::string &val = param.value;
|
||||
if (k.size() <= base.size() + 1 || k.compare(0, base.size(), base) != 0)
|
||||
continue;
|
||||
if (k[base.size()] != '*')
|
||||
continue;
|
||||
std::string_view rest(k);
|
||||
rest.remove_prefix(base.size() + 1); // after "base*"
|
||||
bool star = false;
|
||||
if (!rest.empty() && rest.back() == '*') {
|
||||
star = true;
|
||||
rest.remove_suffix(1);
|
||||
}
|
||||
int idx = 0;
|
||||
for (char c : rest) {
|
||||
if (c < '0' || c > '9') {
|
||||
idx = -1;
|
||||
break;
|
||||
}
|
||||
idx = idx * 10 + (c - '0');
|
||||
}
|
||||
if (idx < 0)
|
||||
continue;
|
||||
if (star)
|
||||
is_ext = true;
|
||||
parts.emplace_back(idx, val);
|
||||
}
|
||||
if (parts.empty())
|
||||
return {};
|
||||
std::sort(parts.begin(), parts.end(),
|
||||
[](const auto &a, const auto &b) { return a.first < b.first; });
|
||||
|
||||
// Piece 0 (if star-form) carries charset'lang' prefix; later pieces are raw
|
||||
// percent-encoded. Concatenate the percent-encoded text then decode once.
|
||||
std::string charset_prefix;
|
||||
std::string enc;
|
||||
bool first = true;
|
||||
for (auto &[idx, val] : parts) {
|
||||
if (first && is_ext) {
|
||||
auto q1 = val.find('\'');
|
||||
auto q2 = (q1 == std::string::npos) ? std::string::npos : val.find('\'', q1 + 1);
|
||||
if (q2 != std::string::npos) {
|
||||
charset_prefix = val.substr(0, q2 + 1);
|
||||
enc += val.substr(q2 + 1);
|
||||
} else {
|
||||
enc += val;
|
||||
}
|
||||
} else {
|
||||
enc += val;
|
||||
}
|
||||
first = false;
|
||||
}
|
||||
if (is_ext)
|
||||
return decode_ext_value(charset_prefix + enc);
|
||||
return to_utf8_best_effort(percent_decode(enc));
|
||||
}
|
||||
|
||||
ContentDisposition::Type classify(std::string_view t) {
|
||||
if (t == "inline")
|
||||
return ContentDisposition::Type::inline_;
|
||||
if (t == "attachment")
|
||||
return ContentDisposition::Type::attachment;
|
||||
if (t == "form-data")
|
||||
return ContentDisposition::Type::form_data;
|
||||
if (t.empty())
|
||||
return ContentDisposition::Type::none;
|
||||
return ContentDisposition::Type::other;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
ContentDisposition parse_content_disposition(std::string_view header_value) {
|
||||
ContentDisposition cd;
|
||||
header_value = trim(header_value);
|
||||
if (header_value.empty())
|
||||
return cd;
|
||||
|
||||
Params p = tokenize(header_value);
|
||||
cd.type = classify(p.type);
|
||||
|
||||
// RFC 6266 §4.3: prefer filename* over filename.
|
||||
std::string ext_name;
|
||||
bool ext_is_ext = false;
|
||||
if (const Param *fstar = p.find("filename*")) {
|
||||
ext_name = decode_ext_value(fstar->value);
|
||||
ext_is_ext = true;
|
||||
} else {
|
||||
std::string joined = join_continuations(p, "filename", ext_is_ext);
|
||||
if (!joined.empty())
|
||||
ext_name = std::move(joined);
|
||||
}
|
||||
|
||||
std::string plain_name;
|
||||
if (const Param *f = p.find("filename")) {
|
||||
// Resolve `\"`, then decode legacy encoded-words if present — BEFORE stripping
|
||||
// path components, since a base64 payload can legitimately contain '/'.
|
||||
std::string raw = f->quoted ? unescape_dquote(f->value) : f->value;
|
||||
bool had_ew = false;
|
||||
std::string decoded = decode_rfc2047(raw, &had_ew);
|
||||
plain_name = had_ew ? std::move(decoded) : to_utf8_best_effort(raw);
|
||||
}
|
||||
|
||||
if (!ext_name.empty()) {
|
||||
cd.filename = strip_path(std::move(ext_name));
|
||||
cd.filename_from_ext = ext_is_ext;
|
||||
} else if (!plain_name.empty()) {
|
||||
cd.filename = strip_path(std::move(plain_name));
|
||||
cd.filename_from_ext = false;
|
||||
}
|
||||
|
||||
// Drop control bytes (incl. NUL) and edge whitespace the decoders may have produced.
|
||||
cd.filename = sanitize_leaf(std::move(cd.filename));
|
||||
return cd;
|
||||
}
|
||||
|
||||
} // namespace vdm::net
|
||||
@@ -0,0 +1,89 @@
|
||||
// vdm/net/curl_error.cpp
|
||||
|
||||
#include "net/curl_error.hpp"
|
||||
|
||||
namespace vdm::net::detail {
|
||||
|
||||
Error error_from_curl(CURLcode code, long http_status) noexcept {
|
||||
// Transport-level failures first — these override any status.
|
||||
switch (code) {
|
||||
case CURLE_OK:
|
||||
break;
|
||||
case CURLE_COULDNT_RESOLVE_PROXY:
|
||||
case CURLE_COULDNT_RESOLVE_HOST:
|
||||
return Error::resolve_failed;
|
||||
case CURLE_COULDNT_CONNECT:
|
||||
case CURLE_INTERFACE_FAILED:
|
||||
return Error::connect_failed;
|
||||
case CURLE_OPERATION_TIMEDOUT:
|
||||
return Error::timeout;
|
||||
case CURLE_TOO_MANY_REDIRECTS:
|
||||
return Error::too_many_redirects;
|
||||
case CURLE_PEER_FAILED_VERIFICATION:
|
||||
case CURLE_SSL_CONNECT_ERROR:
|
||||
case CURLE_SSL_CERTPROBLEM:
|
||||
case CURLE_SSL_CIPHER:
|
||||
case CURLE_SSL_CACERT_BADFILE:
|
||||
case CURLE_SSL_ISSUER_ERROR:
|
||||
case CURLE_SSL_PINNEDPUBKEYNOTMATCH:
|
||||
case CURLE_SSL_INVALIDCERTSTATUS:
|
||||
return Error::tls_failed;
|
||||
case CURLE_GOT_NOTHING:
|
||||
case CURLE_RECV_ERROR:
|
||||
case CURLE_SEND_ERROR:
|
||||
case CURLE_PARTIAL_FILE:
|
||||
case CURLE_HTTP2:
|
||||
case CURLE_HTTP2_STREAM:
|
||||
return Error::connection_reset;
|
||||
case CURLE_WRITE_ERROR:
|
||||
// Our write callback returned short — the sink (disk) failed or we're
|
||||
// cancelling. The caller distinguishes; default to io_error.
|
||||
return Error::io_error;
|
||||
case CURLE_LOGIN_DENIED:
|
||||
return Error::auth_required;
|
||||
case CURLE_UNSUPPORTED_PROTOCOL:
|
||||
case CURLE_URL_MALFORMAT:
|
||||
return Error::unsupported_url_scheme;
|
||||
case CURLE_ABORTED_BY_CALLBACK:
|
||||
return Error::canceled;
|
||||
default:
|
||||
// Fall through to status-based classification, else generic.
|
||||
break;
|
||||
}
|
||||
|
||||
// HTTP status classification (also reached on CURLE_OK).
|
||||
if (http_status >= 400) {
|
||||
switch (http_status) {
|
||||
case 401:
|
||||
case 407:
|
||||
return Error::auth_required;
|
||||
case 403:
|
||||
return Error::forbidden;
|
||||
case 404:
|
||||
return Error::not_found;
|
||||
case 410:
|
||||
return Error::gone;
|
||||
case 416:
|
||||
return Error::range_not_satisfiable;
|
||||
default:
|
||||
return http_status >= 500 ? Error::http_server_error : Error::http_client_error;
|
||||
}
|
||||
}
|
||||
|
||||
if (code != CURLE_OK)
|
||||
return Error::internal;
|
||||
return Error::ok;
|
||||
}
|
||||
|
||||
ErrorInfo make_error(CURLcode code, long http_status, const char *curl_msg) {
|
||||
Error e = error_from_curl(code, http_status);
|
||||
std::string ctx;
|
||||
if (curl_msg && *curl_msg)
|
||||
ctx = curl_msg;
|
||||
else if (code != CURLE_OK)
|
||||
ctx = curl_easy_strerror(code);
|
||||
ErrorInfo info(e, std::move(ctx), static_cast<int>(http_status));
|
||||
return info;
|
||||
}
|
||||
|
||||
} // namespace vdm::net::detail
|
||||
@@ -0,0 +1,24 @@
|
||||
// vdm/net/curl_error.hpp — internal: CURLcode -> vdm::Error. Not a public header.
|
||||
|
||||
#ifndef VDM_NET_CURL_ERROR_HPP
|
||||
#define VDM_NET_CURL_ERROR_HPP
|
||||
|
||||
#include <curl/curl.h>
|
||||
|
||||
#include <string>
|
||||
|
||||
#include "vdm/util/error.hpp"
|
||||
|
||||
namespace vdm::net::detail {
|
||||
|
||||
// Map a libcurl transfer result to the engine taxonomy. `http_status` (0 if none) lets
|
||||
// the HTTP-status errors (403/404/416/...) be classified here too; pass it from
|
||||
// CURLINFO_RESPONSE_CODE. `CURLE_OK` with a >= 400 status still yields an error.
|
||||
[[nodiscard]] Error error_from_curl(CURLcode code, long http_status) noexcept;
|
||||
|
||||
// A human-readable ErrorInfo, folding in curl's own message and the status.
|
||||
[[nodiscard]] ErrorInfo make_error(CURLcode code, long http_status, const char *curl_msg = nullptr);
|
||||
|
||||
} // namespace vdm::net::detail
|
||||
|
||||
#endif // VDM_NET_CURL_ERROR_HPP
|
||||
@@ -0,0 +1,577 @@
|
||||
// vdm/net/http_client.cpp — libcurl multi implementation.
|
||||
//
|
||||
// Threading model: one Worker == one std::jthread + one CURLM. An easy handle is created,
|
||||
// used, paused, and destroyed only on its Worker's thread. Public calls (start / pause /
|
||||
// resume / cancel) just enqueue a Command and curl_multi_wakeup() the worker.
|
||||
|
||||
#include "vdm/net/http_client.hpp"
|
||||
|
||||
#include <curl/curl.h>
|
||||
|
||||
#include <algorithm>
|
||||
#include <atomic>
|
||||
#include <charconv>
|
||||
#include <deque>
|
||||
#include <mutex>
|
||||
#include <string_view>
|
||||
#include <thread>
|
||||
#include <vector>
|
||||
|
||||
#include "net/curl_error.hpp"
|
||||
#include "vdm/util/log.hpp"
|
||||
|
||||
namespace vdm::net {
|
||||
namespace {
|
||||
|
||||
struct CurlGlobal {
|
||||
CurlGlobal() {
|
||||
if (curl_global_init(CURL_GLOBAL_ALL) != CURLE_OK)
|
||||
VDM_LOG_ERROR("net", "curl_global_init failed");
|
||||
}
|
||||
~CurlGlobal() { curl_global_cleanup(); }
|
||||
};
|
||||
void ensure_curl_global() {
|
||||
static CurlGlobal g;
|
||||
(void)g;
|
||||
}
|
||||
|
||||
std::uint64_t next_id() {
|
||||
static std::atomic<std::uint64_t> counter{0};
|
||||
return ++counter;
|
||||
}
|
||||
|
||||
bool parse_header_line(std::string_view line, std::string &name, std::string &value) {
|
||||
while (!line.empty() && (line.back() == '\r' || line.back() == '\n'))
|
||||
line.remove_suffix(1);
|
||||
if (line.empty())
|
||||
return false;
|
||||
auto colon = line.find(':');
|
||||
if (colon == std::string_view::npos)
|
||||
return false;
|
||||
name.assign(line.substr(0, colon));
|
||||
auto v = line.substr(colon + 1);
|
||||
while (!v.empty() && (v.front() == ' ' || v.front() == '\t'))
|
||||
v.remove_prefix(1);
|
||||
value.assign(v);
|
||||
return true;
|
||||
}
|
||||
|
||||
// "HTTP/1.1 206 Partial Content" -> 206; 0 on parse failure.
|
||||
long status_from_line(std::string_view line) {
|
||||
auto sp = line.find(' ');
|
||||
if (sp == std::string_view::npos)
|
||||
return 0;
|
||||
auto rest = line.substr(sp + 1);
|
||||
long code = 0;
|
||||
auto [p, ec] = std::from_chars(rest.data(), rest.data() + rest.size(), code);
|
||||
(void)p;
|
||||
return ec == std::errc{} ? code : 0;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
// --- Transfer::State -----------------------------------------------------------------
|
||||
struct Transfer::State {
|
||||
enum class Stop { none, head_complete, aborted };
|
||||
|
||||
std::uint64_t id = 0;
|
||||
struct HttpClient::Impl *client = nullptr;
|
||||
unsigned worker_index = 0;
|
||||
|
||||
Request req;
|
||||
TransferCallbacks cbs;
|
||||
|
||||
// Worker-thread-owned.
|
||||
CURL *easy = nullptr;
|
||||
curl_slist *header_slist = nullptr;
|
||||
std::string range_value;
|
||||
std::string cookie_value;
|
||||
|
||||
ResponseHead head;
|
||||
long line_status = 0; // status from the most recent HTTP/ line
|
||||
bool head_delivered = false;
|
||||
|
||||
std::atomic<bool> pause_requested{false};
|
||||
std::atomic<Stop> stop{Stop::none};
|
||||
bool curl_paused = false;
|
||||
|
||||
std::uint64_t bytes_received = 0;
|
||||
bool finished = false;
|
||||
};
|
||||
|
||||
// --- Impl -------------------------------------------------------------------------
|
||||
struct HttpClient::Impl {
|
||||
enum class CmdKind { add, pause, resume, cancel };
|
||||
struct Command {
|
||||
CmdKind kind;
|
||||
std::shared_ptr<Transfer::State> state;
|
||||
};
|
||||
|
||||
struct Worker {
|
||||
CURLM *multi = nullptr;
|
||||
std::mutex mu;
|
||||
std::deque<Command> queue;
|
||||
std::vector<std::shared_ptr<Transfer::State>> live;
|
||||
std::jthread thread;
|
||||
};
|
||||
|
||||
explicit Impl(Options o) : opts(o) {
|
||||
ensure_curl_global();
|
||||
|
||||
unsigned n = opts.workers;
|
||||
if (n == 0) {
|
||||
unsigned hw = std::thread::hardware_concurrency();
|
||||
n = std::clamp<unsigned>(hw ? hw : 1, 1, 4);
|
||||
}
|
||||
|
||||
if (opts.share_dns_and_tls) {
|
||||
share = curl_share_init();
|
||||
if (share) {
|
||||
curl_share_setopt(share, CURLSHOPT_LOCKFUNC, &Impl::share_lock);
|
||||
curl_share_setopt(share, CURLSHOPT_UNLOCKFUNC, &Impl::share_unlock);
|
||||
curl_share_setopt(share, CURLSHOPT_USERDATA, this);
|
||||
curl_share_setopt(share, CURLSHOPT_SHARE, CURL_LOCK_DATA_DNS);
|
||||
curl_share_setopt(share, CURLSHOPT_SHARE, CURL_LOCK_DATA_SSL_SESSION);
|
||||
}
|
||||
}
|
||||
|
||||
workers.reserve(n);
|
||||
for (unsigned i = 0; i < n; ++i) {
|
||||
auto w = std::make_unique<Worker>();
|
||||
w->multi = curl_multi_init();
|
||||
if (opts.max_connections_per_worker > 0)
|
||||
curl_multi_setopt(w->multi, CURLMOPT_MAX_TOTAL_CONNECTIONS,
|
||||
opts.max_connections_per_worker);
|
||||
Worker *raw = w.get();
|
||||
w->thread = std::jthread([this, raw](std::stop_token st) { run(*raw, st); });
|
||||
workers.push_back(std::move(w));
|
||||
}
|
||||
}
|
||||
|
||||
~Impl() {
|
||||
stopping.store(true);
|
||||
for (auto &w : workers) {
|
||||
w->thread.request_stop();
|
||||
if (w->multi)
|
||||
curl_multi_wakeup(w->multi);
|
||||
}
|
||||
for (auto &w : workers)
|
||||
if (w->thread.joinable())
|
||||
w->thread.join();
|
||||
for (auto &w : workers)
|
||||
if (w->multi)
|
||||
curl_multi_cleanup(w->multi);
|
||||
if (share)
|
||||
curl_share_cleanup(share);
|
||||
}
|
||||
|
||||
Options opts;
|
||||
std::vector<std::unique_ptr<Worker>> workers;
|
||||
CURLSH *share = nullptr;
|
||||
std::mutex share_mu[CURL_LOCK_DATA_LAST];
|
||||
std::atomic<unsigned> rr{0};
|
||||
std::atomic<bool> stopping{false};
|
||||
|
||||
static void share_lock(CURL *, curl_lock_data data, curl_lock_access, void *userp) {
|
||||
static_cast<Impl *>(userp)->share_mu[data].lock();
|
||||
}
|
||||
static void share_unlock(CURL *, curl_lock_data data, void *userp) {
|
||||
static_cast<Impl *>(userp)->share_mu[data].unlock();
|
||||
}
|
||||
|
||||
void enqueue(unsigned wi, Command cmd) {
|
||||
Worker &w = *workers[wi];
|
||||
{
|
||||
std::lock_guard lk(w.mu);
|
||||
w.queue.push_back(std::move(cmd));
|
||||
}
|
||||
curl_multi_wakeup(w.multi);
|
||||
}
|
||||
|
||||
// ---- curl C callbacks ----
|
||||
static std::size_t header_cb(char *buf, std::size_t size, std::size_t n, void *userp) {
|
||||
auto *st = static_cast<Transfer::State *>(userp);
|
||||
const std::size_t total = size * n;
|
||||
std::string_view line(buf, total);
|
||||
|
||||
if (line.starts_with("HTTP/")) {
|
||||
// A new status line after we already delivered a 401/407 means libcurl's
|
||||
// CURLAUTH_ANY handshake just resent with credentials: let the head of this
|
||||
// second response be delivered too, so callers see the real (2xx/4xx) status
|
||||
// rather than the challenge. Redirects never reach here delivered — their head
|
||||
// is suppressed below — so this only fires for the auth resend.
|
||||
if (st->head_delivered && (st->line_status == 401 || st->line_status == 407))
|
||||
st->head_delivered = false;
|
||||
st->line_status = status_from_line(line);
|
||||
st->head.headers.clear(); // keep only the final response's headers
|
||||
return total;
|
||||
}
|
||||
if (line == "\r\n" || line == "\n") {
|
||||
const bool redirect =
|
||||
st->req.follow_redirects && st->line_status >= 300 && st->line_status < 400;
|
||||
if (!redirect)
|
||||
deliver_head(st);
|
||||
return total;
|
||||
}
|
||||
std::string name, value;
|
||||
if (parse_header_line(line, name, value))
|
||||
st->head.headers.add(std::move(name), std::move(value));
|
||||
return total;
|
||||
}
|
||||
|
||||
static void deliver_head(Transfer::State *st) {
|
||||
if (st->head_delivered)
|
||||
return;
|
||||
st->head_delivered = true;
|
||||
|
||||
long code = 0;
|
||||
curl_easy_getinfo(st->easy, CURLINFO_RESPONSE_CODE, &code);
|
||||
st->head.status = code ? code : st->line_status;
|
||||
char *eff = nullptr;
|
||||
if (curl_easy_getinfo(st->easy, CURLINFO_EFFECTIVE_URL, &eff) == CURLE_OK && eff)
|
||||
st->head.effective_url = eff;
|
||||
curl_off_t clen = -1;
|
||||
if (curl_easy_getinfo(st->easy, CURLINFO_CONTENT_LENGTH_DOWNLOAD_T, &clen) == CURLE_OK &&
|
||||
clen >= 0)
|
||||
st->head.content_length = static_cast<std::uint64_t>(clen);
|
||||
|
||||
if (st->cbs.on_head) {
|
||||
DataAction a = st->cbs.on_head(st->head);
|
||||
if (a == DataAction::abort)
|
||||
st->stop.store(Transfer::State::Stop::head_complete);
|
||||
else if (a == DataAction::pause)
|
||||
st->pause_requested.store(true);
|
||||
}
|
||||
}
|
||||
|
||||
static std::size_t write_cb(char *ptr, std::size_t size, std::size_t n, void *userp) {
|
||||
auto *st = static_cast<Transfer::State *>(userp);
|
||||
const std::size_t total = size * n;
|
||||
|
||||
if (!st->head_delivered)
|
||||
deliver_head(st);
|
||||
if (st->stop.load() != Transfer::State::Stop::none)
|
||||
return 0; // -> CURLE_WRITE_ERROR
|
||||
if (st->pause_requested.load()) {
|
||||
st->curl_paused = true;
|
||||
return CURL_WRITEFUNC_PAUSE;
|
||||
}
|
||||
|
||||
if (total && st->cbs.on_data) {
|
||||
ConstByteSpan span(reinterpret_cast<const std::byte *>(ptr), total);
|
||||
DataAction a = st->cbs.on_data(span);
|
||||
if (a == DataAction::abort) {
|
||||
st->stop.store(Transfer::State::Stop::aborted);
|
||||
return 0;
|
||||
}
|
||||
if (a == DataAction::pause) {
|
||||
st->pause_requested.store(true);
|
||||
st->curl_paused = true;
|
||||
return CURL_WRITEFUNC_PAUSE;
|
||||
}
|
||||
}
|
||||
st->bytes_received += total;
|
||||
return total;
|
||||
}
|
||||
|
||||
// ---- worker thread ----
|
||||
void run(Worker &w, std::stop_token stok) {
|
||||
while (!stok.stop_requested()) {
|
||||
drain_commands(w);
|
||||
int running = 0;
|
||||
curl_multi_perform(w.multi, &running);
|
||||
reap(w);
|
||||
if (stok.stop_requested())
|
||||
break;
|
||||
int numfds = 0;
|
||||
curl_multi_poll(w.multi, nullptr, 0, 1000, &numfds);
|
||||
}
|
||||
shutdown_worker(w);
|
||||
}
|
||||
|
||||
void drain_commands(Worker &w) {
|
||||
std::deque<Command> local;
|
||||
{
|
||||
std::lock_guard lk(w.mu);
|
||||
local.swap(w.queue);
|
||||
}
|
||||
for (auto &cmd : local) {
|
||||
auto &st = cmd.state;
|
||||
switch (cmd.kind) {
|
||||
case CmdKind::add:
|
||||
attach(w, st);
|
||||
break;
|
||||
case CmdKind::pause:
|
||||
st->pause_requested.store(true);
|
||||
if (st->easy && !st->curl_paused) {
|
||||
curl_easy_pause(st->easy, CURLPAUSE_RECV);
|
||||
st->curl_paused = true;
|
||||
}
|
||||
break;
|
||||
case CmdKind::resume:
|
||||
st->pause_requested.store(false);
|
||||
if (st->easy && st->curl_paused) {
|
||||
st->curl_paused = false;
|
||||
curl_easy_pause(st->easy, CURLPAUSE_CONT);
|
||||
}
|
||||
break;
|
||||
case CmdKind::cancel:
|
||||
st->stop.store(Transfer::State::Stop::aborted);
|
||||
if (st->easy && st->curl_paused) {
|
||||
st->curl_paused = false;
|
||||
curl_easy_pause(st->easy, CURLPAUSE_CONT); // let write_cb return 0
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
void attach(Worker &w, std::shared_ptr<Transfer::State> st) {
|
||||
if (stopping.load()) {
|
||||
complete(st, ErrorInfo(Error::canceled, "client shutting down"));
|
||||
return;
|
||||
}
|
||||
CURL *e = curl_easy_init();
|
||||
if (!e) {
|
||||
complete(st, ErrorInfo(Error::internal, "curl_easy_init"));
|
||||
return;
|
||||
}
|
||||
st->easy = e;
|
||||
const Request &r = st->req;
|
||||
|
||||
curl_easy_setopt(e, CURLOPT_URL, r.url.c_str());
|
||||
curl_easy_setopt(e, CURLOPT_PRIVATE, st.get());
|
||||
curl_easy_setopt(e, CURLOPT_NOSIGNAL, 1L);
|
||||
curl_easy_setopt(e, CURLOPT_NOPROGRESS, 1L);
|
||||
curl_easy_setopt(e, CURLOPT_HEADERFUNCTION, &Impl::header_cb);
|
||||
curl_easy_setopt(e, CURLOPT_HEADERDATA, st.get());
|
||||
curl_easy_setopt(e, CURLOPT_WRITEFUNCTION, &Impl::write_cb);
|
||||
curl_easy_setopt(e, CURLOPT_WRITEDATA, st.get());
|
||||
curl_easy_setopt(e, CURLOPT_TCP_KEEPALIVE, 1L);
|
||||
if (share)
|
||||
curl_easy_setopt(e, CURLOPT_SHARE, share);
|
||||
|
||||
if (r.method == Method::head)
|
||||
curl_easy_setopt(e, CURLOPT_NOBODY, 1L);
|
||||
|
||||
if (r.follow_redirects) {
|
||||
curl_easy_setopt(e, CURLOPT_FOLLOWLOCATION, 1L);
|
||||
curl_easy_setopt(e, CURLOPT_MAXREDIRS, r.max_redirects);
|
||||
}
|
||||
curl_easy_setopt(e, CURLOPT_ACCEPT_ENCODING, r.accept_encoding ? "" : nullptr);
|
||||
|
||||
if (r.range) {
|
||||
std::string v = r.range->to_header_value(); // "bytes=first-last"
|
||||
std::string_view sv = v;
|
||||
if (sv.starts_with("bytes="))
|
||||
sv.remove_prefix(6);
|
||||
st->range_value.assign(sv);
|
||||
curl_easy_setopt(e, CURLOPT_RANGE, st->range_value.c_str());
|
||||
}
|
||||
|
||||
curl_easy_setopt(e, CURLOPT_CONNECTTIMEOUT_MS, static_cast<long>(r.connect_timeout_ms));
|
||||
if (r.overall_timeout_ms > 0)
|
||||
curl_easy_setopt(e, CURLOPT_TIMEOUT_MS, static_cast<long>(r.overall_timeout_ms));
|
||||
if (r.low_speed_bytes_per_sec > 0 && r.low_speed_secs > 0) {
|
||||
curl_easy_setopt(e, CURLOPT_LOW_SPEED_LIMIT, r.low_speed_bytes_per_sec);
|
||||
curl_easy_setopt(e, CURLOPT_LOW_SPEED_TIME, r.low_speed_secs);
|
||||
}
|
||||
if (r.max_recv_bytes_per_sec > 0)
|
||||
curl_easy_setopt(e, CURLOPT_MAX_RECV_SPEED_LARGE,
|
||||
static_cast<curl_off_t>(r.max_recv_bytes_per_sec));
|
||||
|
||||
if (!r.user_agent.empty())
|
||||
curl_easy_setopt(e, CURLOPT_USERAGENT, r.user_agent.c_str());
|
||||
if (!r.referrer.empty())
|
||||
curl_easy_setopt(e, CURLOPT_REFERER, r.referrer.c_str());
|
||||
|
||||
if (r.proxy.kind != ProxyKind::none) {
|
||||
curl_easy_setopt(e, CURLOPT_PROXY, r.proxy.host.c_str());
|
||||
if (r.proxy.port)
|
||||
curl_easy_setopt(e, CURLOPT_PROXYPORT, long(r.proxy.port));
|
||||
long pt = CURLPROXY_HTTP;
|
||||
if (r.proxy.kind == ProxyKind::socks5)
|
||||
pt = CURLPROXY_SOCKS5;
|
||||
else if (r.proxy.kind == ProxyKind::socks5_hostname)
|
||||
pt = CURLPROXY_SOCKS5_HOSTNAME;
|
||||
curl_easy_setopt(e, CURLOPT_PROXYTYPE, pt);
|
||||
if (!r.proxy.username.empty()) {
|
||||
std::string up = r.proxy.username + ":" + r.proxy.password;
|
||||
curl_easy_setopt(e, CURLOPT_PROXYUSERPWD, up.c_str());
|
||||
}
|
||||
}
|
||||
|
||||
if (r.auth.scheme != AuthScheme::none) {
|
||||
long m = CURLAUTH_ANY;
|
||||
if (r.auth.scheme == AuthScheme::basic)
|
||||
m = CURLAUTH_BASIC;
|
||||
else if (r.auth.scheme == AuthScheme::digest)
|
||||
m = CURLAUTH_DIGEST;
|
||||
curl_easy_setopt(e, CURLOPT_HTTPAUTH, m);
|
||||
std::string up = r.auth.username + ":" + r.auth.password;
|
||||
curl_easy_setopt(e, CURLOPT_USERPWD, up.c_str());
|
||||
}
|
||||
|
||||
if (!r.cookies.empty()) {
|
||||
for (const auto &c : r.cookies) {
|
||||
if (!st->cookie_value.empty())
|
||||
st->cookie_value += "; ";
|
||||
st->cookie_value += c.name + "=" + c.value;
|
||||
}
|
||||
curl_easy_setopt(e, CURLOPT_COOKIE, st->cookie_value.c_str());
|
||||
}
|
||||
|
||||
for (const auto &h : r.headers) {
|
||||
std::string joined = h.name + ": " + h.value;
|
||||
st->header_slist = curl_slist_append(st->header_slist, joined.c_str());
|
||||
}
|
||||
if (st->header_slist)
|
||||
curl_easy_setopt(e, CURLOPT_HTTPHEADER, st->header_slist);
|
||||
|
||||
CURLMcode mc = curl_multi_add_handle(w.multi, e);
|
||||
if (mc != CURLM_OK) {
|
||||
curl_easy_cleanup(e);
|
||||
st->easy = nullptr;
|
||||
complete(st, ErrorInfo(Error::internal, curl_multi_strerror(mc)));
|
||||
return;
|
||||
}
|
||||
w.live.push_back(std::move(st));
|
||||
}
|
||||
|
||||
void reap(Worker &w) {
|
||||
CURLMsg *msg = nullptr;
|
||||
int inq = 0;
|
||||
while ((msg = curl_multi_info_read(w.multi, &inq)) != nullptr) {
|
||||
if (msg->msg != CURLMSG_DONE)
|
||||
continue;
|
||||
CURL *e = msg->easy_handle;
|
||||
const CURLcode res = msg->data.result;
|
||||
|
||||
Transfer::State *raw = nullptr;
|
||||
curl_easy_getinfo(e, CURLINFO_PRIVATE, &raw);
|
||||
long code = 0;
|
||||
curl_easy_getinfo(e, CURLINFO_RESPONSE_CODE, &code);
|
||||
|
||||
TransferStats stats;
|
||||
stats.http_status = code;
|
||||
gather_timings(e, stats);
|
||||
|
||||
auto it = std::find_if(w.live.begin(), w.live.end(),
|
||||
[raw](const auto &s) { return s.get() == raw; });
|
||||
std::shared_ptr<Transfer::State> st = (it != w.live.end()) ? *it : nullptr;
|
||||
|
||||
curl_multi_remove_handle(w.multi, e);
|
||||
curl_easy_cleanup(e);
|
||||
if (st) {
|
||||
st->easy = nullptr;
|
||||
if (st->header_slist) {
|
||||
curl_slist_free_all(st->header_slist);
|
||||
st->header_slist = nullptr;
|
||||
}
|
||||
}
|
||||
if (it != w.live.end())
|
||||
w.live.erase(it);
|
||||
if (!st)
|
||||
continue;
|
||||
|
||||
using Stop = Transfer::State::Stop;
|
||||
const Stop stop = st->stop.load();
|
||||
|
||||
if (stop == Stop::aborted) {
|
||||
complete(st, ErrorInfo(Error::canceled));
|
||||
} else if (stop == Stop::head_complete) {
|
||||
// A probe: on_head asked to stop. Headers were the goal -> success, even
|
||||
// though a ranged GET body-stop surfaces as CURLE_WRITE_ERROR.
|
||||
stats.bytes_received = st->bytes_received;
|
||||
stats.effective_url = st->head.effective_url;
|
||||
complete(st, std::move(stats));
|
||||
} else if (res == CURLE_OK && code < 400) {
|
||||
stats.bytes_received = st->bytes_received;
|
||||
stats.effective_url = st->head.effective_url;
|
||||
complete(st, std::move(stats));
|
||||
} else {
|
||||
complete(st, detail::make_error(res, code));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
static void gather_timings(CURL *e, TransferStats &s) {
|
||||
auto us_to_ms = [](curl_off_t us) { return us > 0 ? static_cast<long>(us / 1000) : 0L; };
|
||||
curl_off_t t = 0;
|
||||
if (curl_easy_getinfo(e, CURLINFO_NAMELOOKUP_TIME_T, &t) == CURLE_OK)
|
||||
s.namelookup_ms = us_to_ms(t);
|
||||
if (curl_easy_getinfo(e, CURLINFO_CONNECT_TIME_T, &t) == CURLE_OK)
|
||||
s.connect_ms = us_to_ms(t);
|
||||
if (curl_easy_getinfo(e, CURLINFO_APPCONNECT_TIME_T, &t) == CURLE_OK)
|
||||
s.appconnect_ms = us_to_ms(t);
|
||||
if (curl_easy_getinfo(e, CURLINFO_STARTTRANSFER_TIME_T, &t) == CURLE_OK)
|
||||
s.starttransfer_ms = us_to_ms(t);
|
||||
if (curl_easy_getinfo(e, CURLINFO_TOTAL_TIME_T, &t) == CURLE_OK)
|
||||
s.total_ms = us_to_ms(t);
|
||||
}
|
||||
|
||||
void complete(const std::shared_ptr<Transfer::State> &st, Result<TransferStats> r) {
|
||||
if (st->finished)
|
||||
return;
|
||||
st->finished = true;
|
||||
if (st->cbs.on_finished)
|
||||
st->cbs.on_finished(std::move(r));
|
||||
}
|
||||
|
||||
void shutdown_worker(Worker &w) {
|
||||
for (auto &st : w.live) {
|
||||
if (st->easy) {
|
||||
curl_multi_remove_handle(w.multi, st->easy);
|
||||
curl_easy_cleanup(st->easy);
|
||||
st->easy = nullptr;
|
||||
}
|
||||
if (st->header_slist) {
|
||||
curl_slist_free_all(st->header_slist);
|
||||
st->header_slist = nullptr;
|
||||
}
|
||||
complete(st, ErrorInfo(Error::canceled, "client shutting down"));
|
||||
}
|
||||
w.live.clear();
|
||||
}
|
||||
};
|
||||
|
||||
// --- Transfer -------------------------------------------------------------------
|
||||
std::uint64_t Transfer::id() const noexcept {
|
||||
return state_ ? state_->id : 0;
|
||||
}
|
||||
|
||||
void Transfer::pause() {
|
||||
if (state_ && state_->client)
|
||||
state_->client->enqueue(state_->worker_index, {HttpClient::Impl::CmdKind::pause, state_});
|
||||
}
|
||||
void Transfer::resume() {
|
||||
if (state_ && state_->client)
|
||||
state_->client->enqueue(state_->worker_index, {HttpClient::Impl::CmdKind::resume, state_});
|
||||
}
|
||||
void Transfer::cancel() {
|
||||
if (state_ && state_->client)
|
||||
state_->client->enqueue(state_->worker_index, {HttpClient::Impl::CmdKind::cancel, state_});
|
||||
}
|
||||
|
||||
// --- HttpClient ---------------------------------------------------------------
|
||||
HttpClient::HttpClient() : HttpClient(Options{}) {}
|
||||
HttpClient::HttpClient(Options opts) : impl_(std::make_unique<Impl>(opts)) {}
|
||||
HttpClient::~HttpClient() = default;
|
||||
|
||||
unsigned HttpClient::worker_count() const noexcept {
|
||||
return impl_ ? static_cast<unsigned>(impl_->workers.size()) : 0;
|
||||
}
|
||||
|
||||
Transfer HttpClient::start(Request req, TransferCallbacks cbs) {
|
||||
auto st = std::make_shared<Transfer::State>();
|
||||
st->id = next_id();
|
||||
st->client = impl_.get();
|
||||
st->req = std::move(req);
|
||||
st->cbs = std::move(cbs);
|
||||
st->worker_index = impl_->workers.empty() ? 0 : impl_->rr.fetch_add(1) % impl_->workers.size();
|
||||
|
||||
impl_->enqueue(st->worker_index, {Impl::CmdKind::add, st});
|
||||
return Transfer(st);
|
||||
}
|
||||
|
||||
} // namespace vdm::net
|
||||
@@ -0,0 +1,360 @@
|
||||
// vdm/net/probe.cpp
|
||||
|
||||
#include "vdm/net/probe.hpp"
|
||||
|
||||
#include <charconv>
|
||||
#include <deque>
|
||||
#include <mutex>
|
||||
|
||||
#include "net/curl_error.hpp"
|
||||
#include "vdm/net/http_client.hpp"
|
||||
#include "vdm/net/url.hpp"
|
||||
|
||||
namespace vdm::net {
|
||||
namespace {
|
||||
|
||||
std::string_view trim(std::string_view s) {
|
||||
while (!s.empty() && (s.front() == ' ' || s.front() == '\t'))
|
||||
s.remove_prefix(1);
|
||||
while (!s.empty() && (s.back() == ' ' || s.back() == '\t'))
|
||||
s.remove_suffix(1);
|
||||
return s;
|
||||
}
|
||||
|
||||
bool iequals(std::string_view a, std::string_view b) {
|
||||
return HeaderList::iequals(a, b);
|
||||
}
|
||||
|
||||
// "bytes 0-0/12345" -> 12345 ; "bytes 0-0/*" or malformed -> nullopt
|
||||
std::optional<std::uint64_t> total_from_content_range(std::string_view v) {
|
||||
auto slash = v.find('/');
|
||||
if (slash == std::string_view::npos)
|
||||
return std::nullopt;
|
||||
std::string_view tail = trim(v.substr(slash + 1));
|
||||
if (tail.empty() || tail == "*")
|
||||
return std::nullopt;
|
||||
std::uint64_t n = 0;
|
||||
auto [p, ec] = std::from_chars(tail.data(), tail.data() + tail.size(), n);
|
||||
(void)p;
|
||||
if (ec != std::errc{})
|
||||
return std::nullopt;
|
||||
return n;
|
||||
}
|
||||
|
||||
std::string light_sanitize(std::string_view in) {
|
||||
std::string out;
|
||||
out.reserve(in.size());
|
||||
for (unsigned char c : in) {
|
||||
if (c == '/' || c == '\\' || c == 0) {
|
||||
out.push_back('_');
|
||||
} else if (c >= 0x20 || (c & 0x80)) { // keep printable ASCII + all UTF-8 bytes
|
||||
out.push_back(static_cast<char>(c));
|
||||
}
|
||||
}
|
||||
while (!out.empty() && (out.back() == '.' || out.back() == ' '))
|
||||
out.pop_back();
|
||||
if (out == "." || out == "..")
|
||||
out.clear();
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
std::string suggest_filename(const ProbeResult &r, std::string_view explicit_name) {
|
||||
std::string cand;
|
||||
if (!explicit_name.empty())
|
||||
cand = light_sanitize(explicit_name);
|
||||
if (cand.empty() && !r.filename_from_disposition.empty())
|
||||
cand = light_sanitize(r.filename_from_disposition);
|
||||
if (cand.empty() && !r.filename_from_url.empty())
|
||||
cand = light_sanitize(r.filename_from_url);
|
||||
if (cand.empty())
|
||||
cand = "download.bin";
|
||||
return cand;
|
||||
}
|
||||
|
||||
// --- Prober::Impl ---------------------------------------------------------------------
|
||||
struct Prober::Impl {
|
||||
struct Job {
|
||||
ProbeRequest req;
|
||||
std::function<void(Result<ProbeResult>)> done;
|
||||
};
|
||||
|
||||
struct P {
|
||||
Impl *self = nullptr;
|
||||
Job job;
|
||||
ProbeResult result;
|
||||
bool had_head_ok = false;
|
||||
bool delivered = false;
|
||||
};
|
||||
|
||||
explicit Impl(unsigned max_concurrent)
|
||||
: max_(max_concurrent ? max_concurrent : 1), client_(HttpClient::Options{.workers = 2}) {}
|
||||
|
||||
unsigned max_;
|
||||
HttpClient client_;
|
||||
std::mutex mu_;
|
||||
unsigned inflight_ = 0;
|
||||
std::deque<Job> pending_;
|
||||
|
||||
void submit(Job j) {
|
||||
{
|
||||
std::lock_guard lk(mu_);
|
||||
if (inflight_ >= max_) {
|
||||
pending_.push_back(std::move(j));
|
||||
return;
|
||||
}
|
||||
++inflight_;
|
||||
}
|
||||
start(std::move(j));
|
||||
}
|
||||
|
||||
void finish_one() {
|
||||
Job next;
|
||||
bool have_next = false;
|
||||
{
|
||||
std::lock_guard lk(mu_);
|
||||
--inflight_;
|
||||
if (!pending_.empty()) {
|
||||
next = std::move(pending_.front());
|
||||
pending_.pop_front();
|
||||
++inflight_;
|
||||
have_next = true;
|
||||
}
|
||||
}
|
||||
if (have_next)
|
||||
start(std::move(next));
|
||||
}
|
||||
|
||||
Request base_request(const ProbeRequest &pr) {
|
||||
Request r;
|
||||
r.url = pr.url;
|
||||
r.headers = pr.headers;
|
||||
r.cookies = pr.cookies;
|
||||
r.user_agent = pr.user_agent;
|
||||
r.referrer = pr.referrer;
|
||||
r.proxy = pr.proxy;
|
||||
r.auth = pr.auth;
|
||||
r.follow_redirects = true;
|
||||
r.accept_encoding = false;
|
||||
r.connect_timeout_ms = pr.connect_timeout_ms;
|
||||
r.overall_timeout_ms = pr.overall_timeout_ms;
|
||||
r.low_speed_bytes_per_sec = 0; // probes are tiny; no stall detector
|
||||
r.low_speed_secs = 0;
|
||||
return r;
|
||||
}
|
||||
|
||||
void start(Job j) {
|
||||
auto p = std::make_shared<P>();
|
||||
p->self = this;
|
||||
p->job = std::move(j);
|
||||
p->result.effective_url = p->job.req.url;
|
||||
|
||||
Request req = base_request(p->job.req);
|
||||
req.method = Method::head;
|
||||
|
||||
TransferCallbacks cbs;
|
||||
cbs.on_head = [p](const ResponseHead &h) {
|
||||
absorb_head(p->result, h);
|
||||
return head_action(p, h);
|
||||
};
|
||||
cbs.on_data = [](ConstByteSpan) { return DataAction::abort; };
|
||||
cbs.on_finished = [p](Result<TransferStats> r) { on_head_done(p, std::move(r)); };
|
||||
|
||||
// The worker keeps Transfer::State alive; the callbacks keep `p` alive. Storing
|
||||
// the Transfer in `p` would make a p -> Transfer -> State -> cbs -> p cycle.
|
||||
client_.start(std::move(req), std::move(cbs));
|
||||
}
|
||||
|
||||
// We want no body from a probe, so the head callback normally aborts after headers.
|
||||
// The exception: a 401/407 when we were handed credentials — libcurl's CURLAUTH_ANY
|
||||
// has to see that response before it resends with Authorization, so let this one
|
||||
// through (a HEAD has no body; the ranged GET's is a single byte). The final status
|
||||
// then lands on the next header block.
|
||||
static DataAction head_action(const std::shared_ptr<P> &p, const ResponseHead &h) {
|
||||
if ((h.status == 401 || h.status == 407) && p->job.req.auth.scheme != AuthScheme::none)
|
||||
return DataAction::proceed;
|
||||
return DataAction::abort;
|
||||
}
|
||||
|
||||
static void absorb_head(ProbeResult &res, const ResponseHead &h) {
|
||||
if (h.status)
|
||||
res.http_status = h.status;
|
||||
if (!h.effective_url.empty())
|
||||
res.effective_url = h.effective_url;
|
||||
refresh_common(res, h);
|
||||
if (h.content_length && !res.total_size)
|
||||
res.total_size = h.content_length;
|
||||
}
|
||||
|
||||
static void refresh_common(ProbeResult &res, const ResponseHead &h) {
|
||||
if (auto v = h.headers.get("Content-Type"); v && res.mime.empty()) {
|
||||
std::string_view mv = *v;
|
||||
mv = mv.substr(0, mv.find(';'));
|
||||
res.mime.assign(trim(mv));
|
||||
}
|
||||
if (auto v = h.headers.get("ETag"); v && res.etag.empty())
|
||||
res.etag.assign(*v);
|
||||
if (auto v = h.headers.get("Last-Modified"); v && res.last_modified.empty())
|
||||
res.last_modified.assign(*v);
|
||||
if (auto v = h.headers.get("Accept-Ranges")) {
|
||||
if (iequals(trim(*v), "bytes"))
|
||||
res.accept_ranges = true;
|
||||
}
|
||||
if (auto v = h.headers.get("Content-Disposition");
|
||||
v && res.filename_from_disposition.empty()) {
|
||||
ContentDisposition cd = parse_content_disposition(*v);
|
||||
res.disposition_type = cd.type;
|
||||
if (cd.has_filename())
|
||||
res.filename_from_disposition = cd.filename;
|
||||
}
|
||||
}
|
||||
|
||||
static bool has_validator(const ProbeResult &r) {
|
||||
return !r.etag.empty() || !r.last_modified.empty();
|
||||
}
|
||||
|
||||
static void on_head_done(std::shared_ptr<P> p, Result<TransferStats> r) {
|
||||
const long status = p->result.http_status;
|
||||
|
||||
if (status == 0) { // never got headers -> transport failure
|
||||
deliver(p, r.has_value() ? Result<ProbeResult>(ErrorInfo(Error::probe_failed))
|
||||
: Result<ProbeResult>(std::move(r).error()));
|
||||
return;
|
||||
}
|
||||
p->had_head_ok = (status >= 200 && status < 300);
|
||||
|
||||
if (status == 401 || status == 407) {
|
||||
p->result.requires_auth = true;
|
||||
finalize_and_deliver(p);
|
||||
return;
|
||||
}
|
||||
if (status == 403 || status == 405 || status == 501) {
|
||||
start_range_get(p); // HEAD refused; the ranged GET is now the primary probe
|
||||
return;
|
||||
}
|
||||
if (status >= 400) {
|
||||
deliver(p, ErrorInfo(detail::error_from_curl(CURLE_OK, status), "probe HEAD",
|
||||
static_cast<int>(status)));
|
||||
return;
|
||||
}
|
||||
// 2xx. Prove resumability with a ranged GET unless the server said "none".
|
||||
if (auto ar = p->result.accept_ranges; !ar) {
|
||||
// Accept-Ranges absent or not "bytes": still try one ranged GET — servers
|
||||
// that support ranges without advertising are common (docs/06 R4).
|
||||
}
|
||||
start_range_get(p);
|
||||
}
|
||||
|
||||
static void start_range_get(const std::shared_ptr<P> &p) {
|
||||
Request req = p->self->base_request(p->job.req);
|
||||
req.method = Method::get;
|
||||
req.range = ByteRange{0, 0};
|
||||
|
||||
TransferCallbacks cbs;
|
||||
cbs.on_head = [p](const ResponseHead &h) {
|
||||
absorb_range_head(p->result, h);
|
||||
return head_action(p, h); // abort after headers, except a 401/407 with creds
|
||||
};
|
||||
cbs.on_data = [](ConstByteSpan) { return DataAction::abort; };
|
||||
cbs.on_finished = [p](Result<TransferStats> r) { on_range_done(p, std::move(r)); };
|
||||
|
||||
p->self->client_.start(std::move(req), std::move(cbs));
|
||||
}
|
||||
|
||||
static void absorb_range_head(ProbeResult &res, const ResponseHead &h) {
|
||||
if (h.status)
|
||||
res.http_status = h.status;
|
||||
if (!h.effective_url.empty())
|
||||
res.effective_url = h.effective_url;
|
||||
refresh_common(res, h);
|
||||
|
||||
if (h.status == 206) {
|
||||
res.accept_ranges = true; // proven, not just advertised
|
||||
if (auto cr = h.headers.get("Content-Range")) {
|
||||
if (auto total = total_from_content_range(*cr))
|
||||
res.total_size = total;
|
||||
}
|
||||
} else if (h.status == 200) {
|
||||
if (h.content_length)
|
||||
res.total_size = h.content_length;
|
||||
}
|
||||
}
|
||||
|
||||
static void on_range_done(std::shared_ptr<P> p, Result<TransferStats> r) {
|
||||
const long status = p->result.http_status;
|
||||
|
||||
if (status == 0) { // range GET died at transport level
|
||||
if (p->had_head_ok) {
|
||||
p->result.resumable = false;
|
||||
finalize_and_deliver(p);
|
||||
} else {
|
||||
deliver(p, r.has_value() ? Result<ProbeResult>(ErrorInfo(Error::probe_failed))
|
||||
: Result<ProbeResult>(std::move(r).error()));
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (status == 401 || status == 407) {
|
||||
p->result.requires_auth = true;
|
||||
finalize_and_deliver(p);
|
||||
return;
|
||||
}
|
||||
if (status == 206) {
|
||||
p->result.resumable = has_validator(p->result);
|
||||
finalize_and_deliver(p);
|
||||
return;
|
||||
}
|
||||
if (status == 200 || status == 416) {
|
||||
p->result.resumable = false;
|
||||
finalize_and_deliver(p);
|
||||
return;
|
||||
}
|
||||
if (status >= 400) {
|
||||
if (p->had_head_ok) {
|
||||
p->result.resumable = false;
|
||||
finalize_and_deliver(p);
|
||||
} else {
|
||||
deliver(p, ErrorInfo(detail::error_from_curl(CURLE_OK, status), "probe ranged GET",
|
||||
static_cast<int>(status)));
|
||||
}
|
||||
return;
|
||||
}
|
||||
p->result.resumable = false;
|
||||
finalize_and_deliver(p);
|
||||
}
|
||||
|
||||
static void finalize_and_deliver(const std::shared_ptr<P> &p) {
|
||||
ProbeResult &res = p->result;
|
||||
if (res.effective_url.empty())
|
||||
res.effective_url = p->job.req.url;
|
||||
res.filename_from_url = url_filename(res.effective_url);
|
||||
if (res.filename_from_url.empty() && res.effective_url != p->job.req.url)
|
||||
res.filename_from_url = url_filename(p->job.req.url);
|
||||
|
||||
res.redirect_chain.clear();
|
||||
res.redirect_chain.push_back(p->job.req.url);
|
||||
if (res.effective_url != p->job.req.url)
|
||||
res.redirect_chain.push_back(res.effective_url);
|
||||
|
||||
deliver(p, ProbeResult(res));
|
||||
}
|
||||
|
||||
static void deliver(const std::shared_ptr<P> &p, Result<ProbeResult> out) {
|
||||
if (p->delivered)
|
||||
return;
|
||||
p->delivered = true;
|
||||
Impl *self = p->self;
|
||||
p->job.done(std::move(out));
|
||||
self->finish_one();
|
||||
}
|
||||
};
|
||||
|
||||
// --- Prober -----------------------------------------------------------------------
|
||||
Prober::Prober(unsigned max_concurrent) : impl_(std::make_unique<Impl>(max_concurrent)) {}
|
||||
Prober::~Prober() = default;
|
||||
|
||||
void Prober::probe(ProbeRequest req, std::function<void(Result<ProbeResult>)> done) {
|
||||
impl_->submit(Impl::Job{std::move(req), std::move(done)});
|
||||
}
|
||||
|
||||
} // namespace vdm::net
|
||||
@@ -0,0 +1,262 @@
|
||||
// vdm/net/text_codec.cpp
|
||||
|
||||
#include "net/text_codec.hpp"
|
||||
|
||||
#include <array>
|
||||
#include <cctype>
|
||||
|
||||
namespace vdm::net::detail {
|
||||
namespace {
|
||||
|
||||
int hex_val(char c) noexcept {
|
||||
if (c >= '0' && c <= '9')
|
||||
return c - '0';
|
||||
if (c >= 'a' && c <= 'f')
|
||||
return c - 'a' + 10;
|
||||
if (c >= 'A' && c <= 'F')
|
||||
return c - 'A' + 10;
|
||||
return -1;
|
||||
}
|
||||
|
||||
void append_utf8(std::string &out, std::uint32_t cp) {
|
||||
if (cp <= 0x7F) {
|
||||
out.push_back(static_cast<char>(cp));
|
||||
} else if (cp <= 0x7FF) {
|
||||
out.push_back(static_cast<char>(0xC0 | (cp >> 6)));
|
||||
out.push_back(static_cast<char>(0x80 | (cp & 0x3F)));
|
||||
} else if (cp <= 0xFFFF) {
|
||||
out.push_back(static_cast<char>(0xE0 | (cp >> 12)));
|
||||
out.push_back(static_cast<char>(0x80 | ((cp >> 6) & 0x3F)));
|
||||
out.push_back(static_cast<char>(0x80 | (cp & 0x3F)));
|
||||
} else {
|
||||
out.push_back(static_cast<char>(0xF0 | (cp >> 18)));
|
||||
out.push_back(static_cast<char>(0x80 | ((cp >> 12) & 0x3F)));
|
||||
out.push_back(static_cast<char>(0x80 | ((cp >> 6) & 0x3F)));
|
||||
out.push_back(static_cast<char>(0x80 | (cp & 0x3F)));
|
||||
}
|
||||
}
|
||||
|
||||
bool charset_is(std::string_view cs, std::string_view want) {
|
||||
if (cs.size() != want.size())
|
||||
return false;
|
||||
for (std::size_t i = 0; i < cs.size(); ++i) {
|
||||
char a = cs[i], b = want[i];
|
||||
if (a >= 'A' && a <= 'Z')
|
||||
a = static_cast<char>(a - 'A' + 'a');
|
||||
if (b >= 'A' && b <= 'Z')
|
||||
b = static_cast<char>(b - 'A' + 'a');
|
||||
if (a != b)
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
bool is_utf8(std::string_view cs) {
|
||||
return charset_is(cs, "utf-8") || charset_is(cs, "utf8");
|
||||
}
|
||||
bool is_latin1(std::string_view cs) {
|
||||
return charset_is(cs, "iso-8859-1") || charset_is(cs, "latin1") ||
|
||||
charset_is(cs, "iso8859-1") || charset_is(cs, "windows-1252");
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
std::string percent_decode(std::string_view in, bool plus_as_space) {
|
||||
std::string out;
|
||||
out.reserve(in.size());
|
||||
for (std::size_t i = 0; i < in.size(); ++i) {
|
||||
char c = in[i];
|
||||
if (c == '%' && i + 2 < in.size()) {
|
||||
int hi = hex_val(in[i + 1]);
|
||||
int lo = hex_val(in[i + 2]);
|
||||
if (hi >= 0 && lo >= 0) {
|
||||
out.push_back(static_cast<char>((hi << 4) | lo));
|
||||
i += 2;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
if (c == '+' && plus_as_space) {
|
||||
out.push_back(' ');
|
||||
continue;
|
||||
}
|
||||
out.push_back(c);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
bool is_valid_utf8(std::string_view s) noexcept {
|
||||
std::size_t i = 0;
|
||||
const std::size_t n = s.size();
|
||||
auto cont = [&](std::size_t k) {
|
||||
return k < n && (static_cast<unsigned char>(s[k]) & 0xC0) == 0x80;
|
||||
};
|
||||
while (i < n) {
|
||||
unsigned char c = static_cast<unsigned char>(s[i]);
|
||||
if (c < 0x80) {
|
||||
++i;
|
||||
} else if ((c & 0xE0) == 0xC0) {
|
||||
if (!cont(i + 1))
|
||||
return false;
|
||||
std::uint32_t cp = (c & 0x1F) << 6 | (static_cast<unsigned char>(s[i + 1]) & 0x3F);
|
||||
if (cp < 0x80)
|
||||
return false; // overlong
|
||||
i += 2;
|
||||
} else if ((c & 0xF0) == 0xE0) {
|
||||
if (!cont(i + 1) || !cont(i + 2))
|
||||
return false;
|
||||
std::uint32_t cp = (c & 0x0F) << 12 |
|
||||
(static_cast<unsigned char>(s[i + 1]) & 0x3F) << 6 |
|
||||
(static_cast<unsigned char>(s[i + 2]) & 0x3F);
|
||||
if (cp < 0x800 || (cp >= 0xD800 && cp <= 0xDFFF))
|
||||
return false;
|
||||
i += 3;
|
||||
} else if ((c & 0xF8) == 0xF0) {
|
||||
if (!cont(i + 1) || !cont(i + 2) || !cont(i + 3))
|
||||
return false;
|
||||
std::uint32_t cp = (c & 0x07) << 18 |
|
||||
(static_cast<unsigned char>(s[i + 1]) & 0x3F) << 12 |
|
||||
(static_cast<unsigned char>(s[i + 2]) & 0x3F) << 6 |
|
||||
(static_cast<unsigned char>(s[i + 3]) & 0x3F);
|
||||
if (cp < 0x10000 || cp > 0x10FFFF)
|
||||
return false;
|
||||
i += 4;
|
||||
} else {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
std::string latin1_to_utf8(std::string_view s) {
|
||||
std::string out;
|
||||
out.reserve(s.size() + s.size() / 2);
|
||||
for (char ch : s)
|
||||
append_utf8(out, static_cast<unsigned char>(ch));
|
||||
return out;
|
||||
}
|
||||
|
||||
std::string to_utf8_best_effort(std::string_view s) {
|
||||
return is_valid_utf8(s) ? std::string(s) : latin1_to_utf8(s);
|
||||
}
|
||||
|
||||
std::string base64_decode(std::string_view in) {
|
||||
auto val = [](char c) -> int {
|
||||
if (c >= 'A' && c <= 'Z')
|
||||
return c - 'A';
|
||||
if (c >= 'a' && c <= 'z')
|
||||
return c - 'a' + 26;
|
||||
if (c >= '0' && c <= '9')
|
||||
return c - '0' + 52;
|
||||
if (c == '+')
|
||||
return 62;
|
||||
if (c == '/')
|
||||
return 63;
|
||||
return -1;
|
||||
};
|
||||
std::string out;
|
||||
out.reserve(in.size() / 4 * 3 + 3);
|
||||
std::uint32_t acc = 0;
|
||||
int bits = 0;
|
||||
for (char c : in) {
|
||||
if (c == '=' || c == '\r' || c == '\n' || c == ' ' || c == '\t')
|
||||
continue;
|
||||
int v = val(c);
|
||||
if (v < 0)
|
||||
continue; // skip stray bytes
|
||||
acc = (acc << 6) | static_cast<std::uint32_t>(v);
|
||||
bits += 6;
|
||||
if (bits >= 8) {
|
||||
bits -= 8;
|
||||
out.push_back(static_cast<char>((acc >> bits) & 0xFF));
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
std::string decode_rfc2047(std::string_view in, bool *had_encoded_word) {
|
||||
if (had_encoded_word)
|
||||
*had_encoded_word = false;
|
||||
std::string out;
|
||||
out.reserve(in.size());
|
||||
std::size_t i = 0;
|
||||
const std::size_t n = in.size();
|
||||
|
||||
while (i < n) {
|
||||
auto start = in.find("=?", i);
|
||||
if (start == std::string_view::npos) {
|
||||
out.append(in.substr(i));
|
||||
break;
|
||||
}
|
||||
out.append(in.substr(i, start - i));
|
||||
|
||||
// =?charset?enc?text?=
|
||||
auto q1 = in.find('?', start + 2);
|
||||
if (q1 == std::string_view::npos) {
|
||||
out.append(in.substr(start));
|
||||
break;
|
||||
}
|
||||
auto q2 = in.find('?', q1 + 1);
|
||||
if (q2 == std::string_view::npos || q2 != q1 + 2) {
|
||||
out.append("=?");
|
||||
i = start + 2;
|
||||
continue;
|
||||
}
|
||||
auto end = in.find("?=", q2 + 1);
|
||||
if (end == std::string_view::npos) {
|
||||
out.append(in.substr(start));
|
||||
break;
|
||||
}
|
||||
|
||||
std::string_view charset = in.substr(start + 2, q1 - (start + 2));
|
||||
char enc = in[q1 + 1];
|
||||
std::string_view text = in.substr(q2 + 1, end - (q2 + 1));
|
||||
|
||||
std::string bytes;
|
||||
if (enc == 'B' || enc == 'b') {
|
||||
bytes = base64_decode(text);
|
||||
} else if (enc == 'Q' || enc == 'q') {
|
||||
for (std::size_t k = 0; k < text.size(); ++k) {
|
||||
char c = text[k];
|
||||
if (c == '_') {
|
||||
bytes.push_back(' ');
|
||||
} else if (c == '=' && k + 2 < text.size()) {
|
||||
int hi = hex_val(text[k + 1]);
|
||||
int lo = hex_val(text[k + 2]);
|
||||
if (hi >= 0 && lo >= 0) {
|
||||
bytes.push_back(static_cast<char>((hi << 4) | lo));
|
||||
k += 2;
|
||||
} else {
|
||||
bytes.push_back(c);
|
||||
}
|
||||
} else {
|
||||
bytes.push_back(c);
|
||||
}
|
||||
}
|
||||
} else {
|
||||
out.append(in.substr(start, end + 2 - start)); // unknown encoding: verbatim
|
||||
i = end + 2;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (is_utf8(charset))
|
||||
out.append(to_utf8_best_effort(bytes));
|
||||
else if (is_latin1(charset))
|
||||
out.append(latin1_to_utf8(bytes));
|
||||
else
|
||||
out.append(to_utf8_best_effort(bytes));
|
||||
|
||||
if (had_encoded_word)
|
||||
*had_encoded_word = true;
|
||||
i = end + 2;
|
||||
|
||||
// RFC 2047: whitespace between adjacent encoded words is elided.
|
||||
std::size_t j = i;
|
||||
while (j < n && (in[j] == ' ' || in[j] == '\t'))
|
||||
++j;
|
||||
if (j < n && in.compare(j, 2, "=?") == 0)
|
||||
i = j;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace vdm::net::detail
|
||||
@@ -0,0 +1,40 @@
|
||||
// vdm/net/text_codec.hpp — internal: small byte/text codecs for header parsing.
|
||||
//
|
||||
// Not a public header. Everything here is pure, allocation-bounded, and total (no throw,
|
||||
// no assert on input): the inputs come off the wire from untrusted servers.
|
||||
|
||||
#ifndef VDM_NET_TEXT_CODEC_HPP
|
||||
#define VDM_NET_TEXT_CODEC_HPP
|
||||
|
||||
#include <cstdint>
|
||||
#include <string>
|
||||
#include <string_view>
|
||||
|
||||
namespace vdm::net::detail {
|
||||
|
||||
// Percent-decode ("%XX"). A stray '%' or a non-hex digit after it is emitted literally.
|
||||
// `plus_as_space` handles application/x-www-form-urlencoded style; off for URL paths.
|
||||
[[nodiscard]] std::string percent_decode(std::string_view in, bool plus_as_space = false);
|
||||
|
||||
// True if `s` is well-formed UTF-8 (no overlong forms, no surrogates, no > U+10FFFF).
|
||||
[[nodiscard]] bool is_valid_utf8(std::string_view s) noexcept;
|
||||
|
||||
// Reinterpret each byte as a Latin-1 (ISO-8859-1) code point and re-encode as UTF-8.
|
||||
[[nodiscard]] std::string latin1_to_utf8(std::string_view s);
|
||||
|
||||
// Decode standard base64 (RFC 4648, '+' '/', optional '=' padding). Whitespace is
|
||||
// skipped. Invalid trailing bits are dropped. Returns the decoded bytes.
|
||||
[[nodiscard]] std::string base64_decode(std::string_view in);
|
||||
|
||||
// Decode RFC 2047 "encoded-word" runs: =?charset?B?..?= / =?charset?Q?..?=. Text outside
|
||||
// encoded words is passed through. Only UTF-8 and ISO-8859-1/Latin-1 charsets are
|
||||
// transcoded; anything else is passed through as-is (best effort). `had_encoded_word`
|
||||
// reports whether at least one well-formed word was found.
|
||||
[[nodiscard]] std::string decode_rfc2047(std::string_view in, bool *had_encoded_word = nullptr);
|
||||
|
||||
// If `s` is valid UTF-8, return it unchanged; otherwise treat it as Latin-1 and transcode.
|
||||
[[nodiscard]] std::string to_utf8_best_effort(std::string_view s);
|
||||
|
||||
} // namespace vdm::net::detail
|
||||
|
||||
#endif // VDM_NET_TEXT_CODEC_HPP
|
||||
@@ -0,0 +1,126 @@
|
||||
// vdm/net/url.cpp
|
||||
|
||||
#include "vdm/net/url.hpp"
|
||||
|
||||
#include <charconv>
|
||||
|
||||
#include "net/text_codec.hpp"
|
||||
|
||||
namespace vdm::net {
|
||||
namespace {
|
||||
|
||||
std::string ascii_lower(std::string_view s) {
|
||||
std::string r(s);
|
||||
for (char &c : r)
|
||||
if (c >= 'A' && c <= 'Z')
|
||||
c = static_cast<char>(c - 'A' + 'a');
|
||||
return r;
|
||||
}
|
||||
|
||||
bool valid_scheme(std::string_view s) {
|
||||
if (s.empty())
|
||||
return false;
|
||||
if (!((s[0] >= 'a' && s[0] <= 'z') || (s[0] >= 'A' && s[0] <= 'Z')))
|
||||
return false;
|
||||
for (char c : s) {
|
||||
bool ok = (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') ||
|
||||
c == '+' || c == '-' || c == '.';
|
||||
if (!ok)
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
SplitUrl split_url(std::string_view url) {
|
||||
SplitUrl out;
|
||||
|
||||
auto scheme_end = url.find("://");
|
||||
if (scheme_end == std::string_view::npos)
|
||||
return out;
|
||||
std::string_view scheme = url.substr(0, scheme_end);
|
||||
if (!valid_scheme(scheme))
|
||||
return out;
|
||||
out.scheme = ascii_lower(scheme);
|
||||
|
||||
std::string_view rest = url.substr(scheme_end + 3);
|
||||
|
||||
// authority ends at the first '/', '?' or '#'
|
||||
std::size_t auth_end = rest.size();
|
||||
for (std::size_t i = 0; i < rest.size(); ++i) {
|
||||
char c = rest[i];
|
||||
if (c == '/' || c == '?' || c == '#') {
|
||||
auth_end = i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
std::string_view authority = rest.substr(0, auth_end);
|
||||
std::string_view tail = rest.substr(auth_end);
|
||||
|
||||
if (auto at = authority.rfind('@'); at != std::string_view::npos) {
|
||||
out.userinfo.assign(authority.substr(0, at));
|
||||
authority = authority.substr(at + 1);
|
||||
}
|
||||
|
||||
std::string_view host = authority;
|
||||
std::string_view port;
|
||||
if (!authority.empty() && authority.front() == '[') {
|
||||
auto close = authority.find(']');
|
||||
if (close != std::string_view::npos) {
|
||||
host = authority.substr(1, close - 1); // strip brackets
|
||||
if (close + 1 < authority.size() && authority[close + 1] == ':')
|
||||
port = authority.substr(close + 2);
|
||||
}
|
||||
} else if (auto colon = authority.rfind(':'); colon != std::string_view::npos) {
|
||||
host = authority.substr(0, colon);
|
||||
port = authority.substr(colon + 1);
|
||||
}
|
||||
out.host = ascii_lower(host);
|
||||
|
||||
if (!port.empty()) {
|
||||
unsigned v = 0;
|
||||
auto [p, ec] = std::from_chars(port.data(), port.data() + port.size(), v);
|
||||
(void)p;
|
||||
if (ec == std::errc{} && v > 0 && v <= 65535)
|
||||
out.port = static_cast<std::uint16_t>(v);
|
||||
}
|
||||
|
||||
// tail = path [ '?' query ] [ '#' fragment ]
|
||||
std::string_view path_and_rest = tail;
|
||||
if (auto hash = path_and_rest.find('#'); hash != std::string_view::npos) {
|
||||
out.fragment.assign(path_and_rest.substr(hash + 1));
|
||||
path_and_rest = path_and_rest.substr(0, hash);
|
||||
}
|
||||
if (auto q = path_and_rest.find('?'); q != std::string_view::npos) {
|
||||
out.query.assign(path_and_rest.substr(q + 1));
|
||||
path_and_rest = path_and_rest.substr(0, q);
|
||||
}
|
||||
out.path.assign(path_and_rest);
|
||||
|
||||
out.valid = !out.host.empty() && out.is_http();
|
||||
return out;
|
||||
}
|
||||
|
||||
std::string url_filename(std::string_view url) {
|
||||
SplitUrl u = split_url(url);
|
||||
std::string_view path = u.path;
|
||||
if (path.empty())
|
||||
return {};
|
||||
|
||||
auto slash = path.find_last_of('/');
|
||||
std::string_view seg = (slash == std::string_view::npos) ? path : path.substr(slash + 1);
|
||||
if (seg.empty())
|
||||
return {};
|
||||
|
||||
std::string name = detail::percent_decode(seg);
|
||||
// Guard against a decoded segment that reintroduces a separator or NUL.
|
||||
for (char &c : name)
|
||||
if (c == '/' || c == '\\' || c == '\0')
|
||||
c = '_';
|
||||
if (name == "." || name == "..")
|
||||
return {};
|
||||
return name;
|
||||
}
|
||||
|
||||
} // namespace vdm::net
|
||||
@@ -0,0 +1,299 @@
|
||||
// vdm/segment/budget.cpp
|
||||
|
||||
#include "vdm/segment/budget.hpp"
|
||||
|
||||
#include <algorithm>
|
||||
|
||||
namespace vdm::segment {
|
||||
|
||||
SegmentBudget::SegmentBudget() : SegmentBudget(Options{}) {}
|
||||
|
||||
SegmentBudget::SegmentBudget(Options opts)
|
||||
: max_active_(opts.max_active_segments ? opts.max_active_segments : 1),
|
||||
notify_period_(opts.notify_period) {
|
||||
notifier_ = std::jthread([this](std::stop_token st) { notifier_loop(st); });
|
||||
}
|
||||
|
||||
SegmentBudget::~SegmentBudget() {
|
||||
notifier_.request_stop();
|
||||
notify_cv_.notify_all();
|
||||
}
|
||||
|
||||
// --- allocation -----------------------------------------------------------------------
|
||||
|
||||
std::uint32_t SegmentBudget::effective_cap_locked(const Task &t) const {
|
||||
std::uint32_t base = t.resumable ? t.per_task_cap : 1;
|
||||
base = std::clamp<std::uint32_t>(base, 1, 32);
|
||||
if (auto it = host_caps_.find(t.host); it != host_caps_.end() && it->second > 0)
|
||||
base = std::min(base, it->second);
|
||||
return base;
|
||||
}
|
||||
|
||||
SegmentBudget::EngineBudget SegmentBudget::snapshot_locked() const {
|
||||
std::uint32_t starved = 0;
|
||||
for (const auto &[id, t] : tasks_)
|
||||
if (t.want >= 1 && t.held == 0)
|
||||
++starved;
|
||||
return EngineBudget{max_active_, active_, starved};
|
||||
}
|
||||
|
||||
// The two-pass fairness allocation. Recomputes every task's target from scratch (so a
|
||||
// live cap cut naturally produces target < held -> yield), diffs against the last
|
||||
// published target, and collects the callbacks to fire once mu_ is released.
|
||||
SegmentBudget::Plan SegmentBudget::reallocate_locked() {
|
||||
// Priority order: DAEMON's list first, then any registered task not in it (defensive;
|
||||
// "a running task absent from the list sorts last").
|
||||
std::vector<TaskId> order;
|
||||
order.reserve(tasks_.size());
|
||||
for (TaskId id : order_)
|
||||
if (tasks_.count(id))
|
||||
order.push_back(id);
|
||||
for (const auto &[id, _] : tasks_)
|
||||
if (std::find(order.begin(), order.end(), id) == order.end())
|
||||
order.push_back(id);
|
||||
|
||||
std::unordered_map<TaskId, std::uint32_t> target;
|
||||
target.reserve(order.size());
|
||||
std::uint32_t pool = max_active_;
|
||||
|
||||
auto capped_want = [&](TaskId id) {
|
||||
const Task &t = tasks_.at(id);
|
||||
return std::min(t.want, effective_cap_locked(t));
|
||||
};
|
||||
|
||||
// Guarantee pass: one slot each, in priority order, to anyone who wants one.
|
||||
for (TaskId id : order) {
|
||||
if (pool == 0)
|
||||
break;
|
||||
if (capped_want(id) >= 1) {
|
||||
target[id] = 1;
|
||||
--pool;
|
||||
}
|
||||
}
|
||||
// Growth pass: round-robin the remainder, up to each task's effective cap.
|
||||
while (pool > 0) {
|
||||
bool granted = false;
|
||||
for (TaskId id : order) {
|
||||
if (pool == 0)
|
||||
break;
|
||||
std::uint32_t &tv = target[id];
|
||||
if (tv < capped_want(id)) {
|
||||
++tv;
|
||||
--pool;
|
||||
granted = true;
|
||||
}
|
||||
}
|
||||
if (!granted)
|
||||
break;
|
||||
}
|
||||
|
||||
const SteadyTime now = std::chrono::steady_clock::now();
|
||||
Plan plan;
|
||||
for (auto &[id, t] : tasks_) {
|
||||
std::uint32_t nt = target.count(id) ? target[id] : 0;
|
||||
if (nt != t.target) {
|
||||
t.target = nt;
|
||||
if (t.on_target)
|
||||
plan.targets.emplace_back(t.on_target, nt);
|
||||
}
|
||||
// starvation timestamp bookkeeping
|
||||
bool starved_now = t.want >= 1 && t.held == 0;
|
||||
if (starved_now && !t.starved_since)
|
||||
t.starved_since = now;
|
||||
if (!starved_now)
|
||||
t.starved_since.reset();
|
||||
}
|
||||
|
||||
EngineBudget eb = snapshot_locked();
|
||||
bool starved_edge = (eb.tasks_starved == 0) != (last_starved_ == 0);
|
||||
if (eb != last_notified_)
|
||||
dirty_ = true;
|
||||
last_starved_ = eb.tasks_starved;
|
||||
|
||||
if (starved_edge && on_changed_) {
|
||||
plan.notify_now = std::make_pair(on_changed_, eb);
|
||||
last_notified_ = eb;
|
||||
dirty_ = false;
|
||||
}
|
||||
if (dirty_)
|
||||
notify_cv_.notify_one();
|
||||
return plan;
|
||||
}
|
||||
|
||||
void SegmentBudget::run(Plan &p) {
|
||||
for (auto &[fn, n] : p.targets)
|
||||
if (fn)
|
||||
fn(n);
|
||||
if (p.notify_now && p.notify_now->first)
|
||||
p.notify_now->first(p.notify_now->second);
|
||||
}
|
||||
|
||||
// --- task-facing --------------------------------------------------------------------
|
||||
|
||||
void SegmentBudget::register_task(TaskId id, const TaskParams ¶ms, SlotTargetFn on_target) {
|
||||
Plan plan;
|
||||
{
|
||||
std::lock_guard lk(mu_);
|
||||
Task t;
|
||||
t.host = params.host;
|
||||
t.per_task_cap = params.per_task_cap ? params.per_task_cap : 1;
|
||||
t.resumable = params.resumable;
|
||||
t.on_target = std::move(on_target);
|
||||
tasks_[id] = std::move(t);
|
||||
plan = reallocate_locked();
|
||||
}
|
||||
run(plan);
|
||||
}
|
||||
|
||||
void SegmentBudget::deregister_task(TaskId id) {
|
||||
Plan plan;
|
||||
{
|
||||
std::lock_guard lk(mu_);
|
||||
auto it = tasks_.find(id);
|
||||
if (it == tasks_.end())
|
||||
return;
|
||||
active_ -= it->second.held;
|
||||
tasks_.erase(it);
|
||||
plan = reallocate_locked();
|
||||
}
|
||||
run(plan);
|
||||
}
|
||||
|
||||
void SegmentBudget::set_want(TaskId id, std::uint32_t want) {
|
||||
Plan plan;
|
||||
{
|
||||
std::lock_guard lk(mu_);
|
||||
auto it = tasks_.find(id);
|
||||
if (it == tasks_.end())
|
||||
return;
|
||||
if (it->second.want == want)
|
||||
return;
|
||||
it->second.want = want;
|
||||
plan = reallocate_locked();
|
||||
}
|
||||
run(plan);
|
||||
}
|
||||
|
||||
bool SegmentBudget::confirm_slot(TaskId id) {
|
||||
std::lock_guard lk(mu_);
|
||||
auto it = tasks_.find(id);
|
||||
if (it == tasks_.end())
|
||||
return false;
|
||||
Task &t = it->second;
|
||||
if (t.held >= t.target)
|
||||
return false; // target was cut in the race
|
||||
++t.held;
|
||||
++active_;
|
||||
if (snapshot_locked() != last_notified_) {
|
||||
dirty_ = true;
|
||||
notify_cv_.notify_one();
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
void SegmentBudget::release_slot(TaskId id) {
|
||||
Plan plan;
|
||||
{
|
||||
std::lock_guard lk(mu_);
|
||||
auto it = tasks_.find(id);
|
||||
if (it == tasks_.end() || it->second.held == 0)
|
||||
return;
|
||||
--it->second.held;
|
||||
--active_;
|
||||
plan = reallocate_locked();
|
||||
}
|
||||
run(plan);
|
||||
}
|
||||
|
||||
// --- DAEMON-facing ---------------------------------------------------------------------
|
||||
|
||||
void SegmentBudget::set_max_active_segments(std::uint32_t n) {
|
||||
Plan plan;
|
||||
{
|
||||
std::lock_guard lk(mu_);
|
||||
n = n ? n : 1;
|
||||
if (n == max_active_)
|
||||
return;
|
||||
max_active_ = n;
|
||||
plan = reallocate_locked();
|
||||
}
|
||||
run(plan);
|
||||
}
|
||||
|
||||
void SegmentBudget::set_host_segment_cap(std::string host, std::uint32_t cap) {
|
||||
Plan plan;
|
||||
{
|
||||
std::lock_guard lk(mu_);
|
||||
if (cap == 0)
|
||||
host_caps_.erase(host);
|
||||
else
|
||||
host_caps_[std::move(host)] = cap;
|
||||
plan = reallocate_locked();
|
||||
}
|
||||
run(plan);
|
||||
}
|
||||
|
||||
void SegmentBudget::set_task_order(std::span<const TaskId> priority_order) {
|
||||
Plan plan;
|
||||
{
|
||||
std::lock_guard lk(mu_);
|
||||
order_.assign(priority_order.begin(), priority_order.end());
|
||||
plan = reallocate_locked();
|
||||
}
|
||||
run(plan);
|
||||
}
|
||||
|
||||
SegmentBudget::EngineBudget SegmentBudget::budget() const {
|
||||
std::lock_guard lk(mu_);
|
||||
return snapshot_locked();
|
||||
}
|
||||
|
||||
std::uint32_t SegmentBudget::segments_active(TaskId id) const {
|
||||
std::lock_guard lk(mu_);
|
||||
auto it = tasks_.find(id);
|
||||
return it == tasks_.end() ? 0 : it->second.held;
|
||||
}
|
||||
|
||||
std::vector<TaskId> SegmentBudget::starved_tasks() const {
|
||||
std::lock_guard lk(mu_);
|
||||
std::vector<TaskId> out;
|
||||
for (const auto &[id, t] : tasks_)
|
||||
if (t.want >= 1 && t.held == 0)
|
||||
out.push_back(id);
|
||||
return out;
|
||||
}
|
||||
|
||||
std::optional<SteadyTime> SegmentBudget::starved_since(TaskId id) const {
|
||||
std::lock_guard lk(mu_);
|
||||
auto it = tasks_.find(id);
|
||||
return it == tasks_.end() ? std::nullopt : it->second.starved_since;
|
||||
}
|
||||
|
||||
void SegmentBudget::on_budget_changed(std::function<void(EngineBudget)> cb) {
|
||||
std::lock_guard lk(mu_);
|
||||
on_changed_ = std::move(cb);
|
||||
}
|
||||
|
||||
// --- notifier thread: coalesced <=4 Hz -----------------------------------------------
|
||||
|
||||
void SegmentBudget::notifier_loop(std::stop_token st) {
|
||||
std::unique_lock lk(mu_);
|
||||
while (!st.stop_requested()) {
|
||||
notify_cv_.wait_for(lk, notify_period_, [&] { return dirty_ || st.stop_requested(); });
|
||||
if (st.stop_requested())
|
||||
break;
|
||||
if (!dirty_)
|
||||
continue;
|
||||
EngineBudget eb = snapshot_locked();
|
||||
auto cb = on_changed_;
|
||||
last_notified_ = eb;
|
||||
last_starved_ = eb.tasks_starved;
|
||||
dirty_ = false;
|
||||
lk.unlock();
|
||||
if (cb)
|
||||
cb(eb);
|
||||
lk.lock();
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace vdm::segment
|
||||
@@ -0,0 +1,321 @@
|
||||
// vdm/segment/segmenter.cpp
|
||||
|
||||
#include "vdm/segment/segmenter.hpp"
|
||||
|
||||
#include <algorithm>
|
||||
#include <limits>
|
||||
|
||||
namespace vdm::segment {
|
||||
namespace {
|
||||
constexpr std::uint32_t kNoIndex = std::numeric_limits<std::uint32_t>::max();
|
||||
constexpr std::uint64_t kU64Max = std::numeric_limits<std::uint64_t>::max();
|
||||
|
||||
bool is_live(SegState s) noexcept {
|
||||
return s == SegState::idle || s == SegState::connecting || s == SegState::downloading ||
|
||||
s == SegState::stalled;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
// Seg holds std::atomics, so it is neither copyable nor movable — every insertion is an
|
||||
// emplace_back that constructs it in place, followed by stores. This helper centralises
|
||||
// that. Caller holds mu_.
|
||||
std::uint32_t Segmenter::add_seg_locked(std::uint64_t start, std::uint64_t end,
|
||||
std::uint64_t completed, SegState state, bool assigned) {
|
||||
segs_.emplace_back(next_index_++, start, end);
|
||||
Seg &s = segs_.back();
|
||||
s.completed.store(completed);
|
||||
s.state.store(state);
|
||||
s.assigned = assigned;
|
||||
return s.index;
|
||||
}
|
||||
|
||||
// --- construction ---------------------------------------------------------------------
|
||||
|
||||
Segmenter::Segmenter(std::uint64_t total_size, std::uint32_t requested_segments, bool resumable,
|
||||
std::uint64_t min_segment_bytes)
|
||||
: total_size_(total_size),
|
||||
min_seg_(min_segment_bytes ? min_segment_bytes : 1),
|
||||
resumable_(resumable) {
|
||||
compute_target(requested_segments);
|
||||
}
|
||||
|
||||
Segmenter::Segmenter(std::uint64_t total_size, std::uint32_t requested_segments,
|
||||
const std::vector<ResumedRange> &resumed, bool resumable,
|
||||
std::uint64_t min_segment_bytes)
|
||||
: total_size_(total_size),
|
||||
min_seg_(min_segment_bytes ? min_segment_bytes : 1),
|
||||
resumable_(resumable) {
|
||||
compute_target(requested_segments);
|
||||
|
||||
// Validate the resumed table tiles [0, total_size) exactly.
|
||||
bool ok = resumable_ && total_size_ > 0 && !resumed.empty();
|
||||
if (ok) {
|
||||
std::vector<ResumedRange> sorted = resumed;
|
||||
std::sort(sorted.begin(), sorted.end(),
|
||||
[](const auto &a, const auto &b) { return a.start < b.start; });
|
||||
std::uint64_t cursor = 0;
|
||||
for (const auto &r : sorted) {
|
||||
if (r.start != cursor || r.end < r.start || r.completed > r.end - r.start + 1) {
|
||||
ok = false;
|
||||
break;
|
||||
}
|
||||
cursor = r.end + 1;
|
||||
}
|
||||
if (ok && cursor != total_size_)
|
||||
ok = false;
|
||||
if (ok) {
|
||||
for (const auto &r : sorted) {
|
||||
bool done = r.completed == r.end - r.start + 1;
|
||||
add_seg_locked(r.start, r.end, r.completed,
|
||||
done ? SegState::complete : SegState::idle, false);
|
||||
}
|
||||
std::uint32_t incomplete = 0;
|
||||
for (const auto &s : segs_)
|
||||
if (s.state.load() != SegState::complete)
|
||||
++incomplete;
|
||||
target_count_ = std::clamp<std::uint32_t>(std::max(incomplete, 1u), 1, kMaxSegments);
|
||||
return;
|
||||
}
|
||||
}
|
||||
// Fall back to a fresh single/target layout (segs created lazily by assign_slot()).
|
||||
segs_.clear();
|
||||
}
|
||||
|
||||
void Segmenter::compute_target(std::uint32_t requested) {
|
||||
if (!resumable_ || total_size_ == 0) {
|
||||
target_count_ = 1;
|
||||
return;
|
||||
}
|
||||
std::uint64_t by_size = total_size_ / min_seg_;
|
||||
if (by_size == 0)
|
||||
by_size = 1;
|
||||
std::uint64_t t = requested == 0 ? kDefaultSegments : requested;
|
||||
t = std::min<std::uint64_t>(t, by_size);
|
||||
target_count_ = std::clamp<std::uint32_t>(static_cast<std::uint32_t>(t), 1, kMaxSegments);
|
||||
}
|
||||
|
||||
// --- helpers (mu_ held) --------------------------------------------------------------
|
||||
|
||||
std::uint64_t Segmenter::remaining_of_locked(const Seg &s) const noexcept {
|
||||
std::uint64_t end = s.end.load();
|
||||
std::uint64_t done = s.start + s.completed.load();
|
||||
return done > end ? 0 : end - done + 1;
|
||||
}
|
||||
|
||||
std::uint32_t Segmenter::assigned_count_locked() const noexcept {
|
||||
std::uint32_t n = 0;
|
||||
for (const auto &s : segs_)
|
||||
if (s.assigned)
|
||||
++n;
|
||||
return n;
|
||||
}
|
||||
|
||||
// Split the largest remaining range; hand back its second half as a new segment.
|
||||
std::uint32_t Segmenter::split_largest_remaining_locked() {
|
||||
Seg *victim = nullptr;
|
||||
std::uint64_t best = 0;
|
||||
for (auto &s : segs_) {
|
||||
if (!s.assigned || !is_live(s.state.load()))
|
||||
continue;
|
||||
std::uint64_t rem = remaining_of_locked(s);
|
||||
if (rem > best) {
|
||||
best = rem;
|
||||
victim = &s;
|
||||
}
|
||||
}
|
||||
if (!victim || best < 2 * min_seg_)
|
||||
return kNoIndex;
|
||||
|
||||
const std::uint64_t v_end = victim->end.load();
|
||||
const std::uint64_t half = best / 2; // >= min_seg_ since best >= 2*min
|
||||
const std::uint64_t mid = v_end - half; // victim keeps [start, mid]
|
||||
const std::uint64_t cur = victim->start + victim->completed.load();
|
||||
if (mid < cur || mid - cur + 1 < min_seg_)
|
||||
return kNoIndex; // victim would be too small
|
||||
|
||||
victim->end.store(mid); // the victim's worker reads end before each write and stops here
|
||||
return add_seg_locked(mid + 1, v_end, 0, SegState::idle, false);
|
||||
}
|
||||
|
||||
// --- public: structural (take the lock) --------------------------------------------
|
||||
|
||||
std::optional<std::uint32_t> Segmenter::assign_slot() {
|
||||
std::lock_guard lk(mu_);
|
||||
|
||||
if (!orphans_.empty()) {
|
||||
ResumedRange o = orphans_.front();
|
||||
orphans_.erase(orphans_.begin());
|
||||
return add_seg_locked(o.start, o.end, o.completed, SegState::connecting, true);
|
||||
}
|
||||
|
||||
if (assigned_count_locked() >= target_count_)
|
||||
return std::nullopt;
|
||||
|
||||
if (segs_.empty()) {
|
||||
std::uint64_t end = total_size_ > 0 ? total_size_ - 1 : kU64Max - 1;
|
||||
return add_seg_locked(0, end, 0, SegState::connecting, true);
|
||||
}
|
||||
|
||||
// Some resumed segments may be unassigned idle ranges — hand one out before splitting.
|
||||
for (auto &s : segs_) {
|
||||
if (!s.assigned && s.state.load() == SegState::idle) {
|
||||
s.assigned = true;
|
||||
s.state.store(SegState::connecting);
|
||||
return s.index;
|
||||
}
|
||||
}
|
||||
|
||||
std::uint32_t idx = split_largest_remaining_locked();
|
||||
if (idx == kNoIndex)
|
||||
return std::nullopt;
|
||||
segs_[idx].assigned = true;
|
||||
segs_[idx].state.store(SegState::connecting);
|
||||
return idx;
|
||||
}
|
||||
|
||||
std::optional<std::uint32_t> Segmenter::on_complete(std::uint32_t idx, bool may_steal) {
|
||||
std::lock_guard lk(mu_);
|
||||
if (idx >= segs_.size())
|
||||
return std::nullopt;
|
||||
Seg &seg = segs_[idx];
|
||||
seg.completed.store(seg.end.load() - seg.start + 1);
|
||||
seg.state.store(SegState::complete);
|
||||
seg.assigned = false;
|
||||
|
||||
if (!may_steal)
|
||||
return std::nullopt; // yielding the slot
|
||||
|
||||
if (!orphans_.empty()) {
|
||||
ResumedRange o = orphans_.front();
|
||||
orphans_.erase(orphans_.begin());
|
||||
return add_seg_locked(o.start, o.end, o.completed, SegState::connecting, true);
|
||||
}
|
||||
|
||||
std::uint32_t new_idx = split_largest_remaining_locked();
|
||||
if (new_idx == kNoIndex)
|
||||
return std::nullopt; // nothing to steal -> release the slot
|
||||
segs_[new_idx].assigned = true;
|
||||
segs_[new_idx].state.store(SegState::connecting);
|
||||
return new_idx;
|
||||
}
|
||||
|
||||
FailAction Segmenter::on_failed(std::uint32_t idx, bool connection_error, bool has_mirror) {
|
||||
std::lock_guard lk(mu_);
|
||||
if (idx >= segs_.size())
|
||||
return FailAction::retry;
|
||||
Seg &seg = segs_[idx];
|
||||
++seg.consecutive_failures;
|
||||
|
||||
if (connection_error && seg.consecutive_failures >= 3 && has_mirror) {
|
||||
std::uint64_t cur = seg.start + seg.completed.load();
|
||||
std::uint64_t end = seg.end.load();
|
||||
if (cur <= end)
|
||||
orphans_.push_back({cur, end, 0});
|
||||
seg.state.store(SegState::failed);
|
||||
seg.assigned = false;
|
||||
return FailAction::requeue;
|
||||
}
|
||||
return FailAction::retry;
|
||||
}
|
||||
|
||||
void Segmenter::note_connected(std::uint32_t idx) {
|
||||
std::lock_guard lk(mu_);
|
||||
if (idx < segs_.size())
|
||||
segs_[idx].consecutive_failures = 0;
|
||||
}
|
||||
|
||||
// --- public: per-worker accessors ----------------------------------------------
|
||||
//
|
||||
// These take mu_. They are called from the write path once per buffer flush (a few per
|
||||
// second per segment), not from the curl write callback — the no-lock/no-alloc rule is
|
||||
// about that callback and its ring buffer, not about progress bookkeeping. The segment
|
||||
// fields are still std::atomic so a reader that already holds a stable reference sees a
|
||||
// torn-free value, and so the deque element type is safe to relocate-free.
|
||||
|
||||
void Segmenter::advance(std::uint32_t idx, std::uint64_t bytes) noexcept {
|
||||
std::lock_guard lk(mu_);
|
||||
if (idx >= segs_.size())
|
||||
return;
|
||||
Seg &s = segs_[idx];
|
||||
std::uint64_t len = s.end.load() - s.start + 1;
|
||||
s.completed.store(bytes < len ? bytes : len);
|
||||
}
|
||||
|
||||
std::uint64_t Segmenter::segment_start(std::uint32_t idx) const noexcept {
|
||||
std::lock_guard lk(mu_);
|
||||
return idx < segs_.size() ? segs_[idx].start : 0;
|
||||
}
|
||||
std::uint64_t Segmenter::segment_end(std::uint32_t idx) const noexcept {
|
||||
std::lock_guard lk(mu_);
|
||||
return idx < segs_.size() ? segs_[idx].end.load() : 0;
|
||||
}
|
||||
std::uint64_t Segmenter::segment_completed(std::uint32_t idx) const noexcept {
|
||||
std::lock_guard lk(mu_);
|
||||
return idx < segs_.size() ? segs_[idx].completed.load() : 0;
|
||||
}
|
||||
SegState Segmenter::segment_state(std::uint32_t idx) const noexcept {
|
||||
std::lock_guard lk(mu_);
|
||||
return idx < segs_.size() ? segs_[idx].state.load() : SegState::failed;
|
||||
}
|
||||
void Segmenter::set_segment_state(std::uint32_t idx, SegState st) noexcept {
|
||||
std::lock_guard lk(mu_);
|
||||
if (idx < segs_.size())
|
||||
segs_[idx].state.store(st);
|
||||
}
|
||||
|
||||
void Segmenter::release_segment(std::uint32_t idx) noexcept {
|
||||
std::lock_guard lk(mu_);
|
||||
if (idx >= segs_.size())
|
||||
return;
|
||||
segs_[idx].assigned = false;
|
||||
if (segs_[idx].state.load() != SegState::complete)
|
||||
segs_[idx].state.store(SegState::idle);
|
||||
}
|
||||
|
||||
// --- public: queries (take the lock) ----------------------------------------------
|
||||
|
||||
std::uint64_t Segmenter::downloaded() const {
|
||||
std::lock_guard lk(mu_);
|
||||
std::uint64_t sum = 0;
|
||||
for (const auto &s : segs_)
|
||||
sum += s.completed.load();
|
||||
return sum;
|
||||
}
|
||||
|
||||
bool Segmenter::all_complete() const {
|
||||
std::lock_guard lk(mu_);
|
||||
if (segs_.empty())
|
||||
return false;
|
||||
if (total_size_ == 0)
|
||||
return segs_.front().state.load() == SegState::complete;
|
||||
if (!orphans_.empty())
|
||||
return false;
|
||||
|
||||
std::vector<std::pair<std::uint64_t, std::uint64_t>> done; // [start, start+completed)
|
||||
for (const auto &s : segs_) {
|
||||
std::uint64_t c = s.completed.load();
|
||||
if (c > 0)
|
||||
done.emplace_back(s.start, s.start + c);
|
||||
}
|
||||
std::sort(done.begin(), done.end());
|
||||
std::uint64_t cursor = 0;
|
||||
for (auto [a, b] : done) {
|
||||
if (a > cursor)
|
||||
return false; // gap
|
||||
if (b > cursor)
|
||||
cursor = b;
|
||||
}
|
||||
return cursor >= total_size_;
|
||||
}
|
||||
|
||||
std::vector<SegmentView> Segmenter::snapshot() const {
|
||||
std::lock_guard lk(mu_);
|
||||
std::vector<SegmentView> out;
|
||||
out.reserve(segs_.size());
|
||||
for (const auto &s : segs_)
|
||||
out.push_back(SegmentView{s.index, s.start, s.end.load(), s.completed.load(),
|
||||
s.state.load(), s.consecutive_failures});
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace vdm::segment
|
||||
@@ -0,0 +1,89 @@
|
||||
// vdm/task/digest.cpp
|
||||
|
||||
#include "task/digest.hpp"
|
||||
|
||||
#include <fcntl.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#include <array>
|
||||
#include <cerrno>
|
||||
#include <cstring>
|
||||
|
||||
#include <openssl/evp.h>
|
||||
|
||||
namespace vdm::task {
|
||||
namespace {
|
||||
|
||||
const EVP_MD *md_for(Checksum::Algo a) {
|
||||
switch (a) {
|
||||
case Checksum::Algo::md5:
|
||||
return EVP_md5();
|
||||
case Checksum::Algo::sha1:
|
||||
return EVP_sha1();
|
||||
case Checksum::Algo::sha256:
|
||||
return EVP_sha256();
|
||||
case Checksum::Algo::sha512:
|
||||
return EVP_sha512();
|
||||
}
|
||||
return EVP_sha256();
|
||||
}
|
||||
|
||||
std::string to_hex(const unsigned char *p, unsigned n) {
|
||||
static const char *h = "0123456789abcdef";
|
||||
std::string s;
|
||||
s.reserve(n * 2);
|
||||
for (unsigned i = 0; i < n; ++i) {
|
||||
s.push_back(h[p[i] >> 4]);
|
||||
s.push_back(h[p[i] & 0xF]);
|
||||
}
|
||||
return s;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
Result<std::string> hash_file(std::string_view path, Checksum::Algo algo) {
|
||||
std::string p(path);
|
||||
int fd = ::open(p.c_str(), O_RDONLY | O_CLOEXEC);
|
||||
if (fd < 0)
|
||||
return ErrorInfo(Error::path_rejected,
|
||||
std::string("open ") + p + ": " + std::strerror(errno));
|
||||
|
||||
EVP_MD_CTX *ctx = EVP_MD_CTX_new();
|
||||
if (!ctx) {
|
||||
::close(fd);
|
||||
return ErrorInfo(Error::internal, "EVP_MD_CTX_new");
|
||||
}
|
||||
auto fail = [&](Error e, std::string msg) {
|
||||
EVP_MD_CTX_free(ctx);
|
||||
::close(fd);
|
||||
return Result<std::string>(ErrorInfo(e, std::move(msg)));
|
||||
};
|
||||
|
||||
if (EVP_DigestInit_ex(ctx, md_for(algo), nullptr) != 1)
|
||||
return fail(Error::internal, "EVP_DigestInit_ex");
|
||||
|
||||
std::array<unsigned char, 256 * 1024> buf{};
|
||||
for (;;) {
|
||||
ssize_t n = ::read(fd, buf.data(), buf.size());
|
||||
if (n < 0) {
|
||||
if (errno == EINTR)
|
||||
continue;
|
||||
return fail(Error::io_error, std::string("read: ") + std::strerror(errno));
|
||||
}
|
||||
if (n == 0)
|
||||
break;
|
||||
if (EVP_DigestUpdate(ctx, buf.data(), static_cast<std::size_t>(n)) != 1)
|
||||
return fail(Error::internal, "EVP_DigestUpdate");
|
||||
}
|
||||
|
||||
unsigned char out[EVP_MAX_MD_SIZE];
|
||||
unsigned out_len = 0;
|
||||
if (EVP_DigestFinal_ex(ctx, out, &out_len) != 1)
|
||||
return fail(Error::internal, "EVP_DigestFinal_ex");
|
||||
|
||||
EVP_MD_CTX_free(ctx);
|
||||
::close(fd);
|
||||
return to_hex(out, out_len);
|
||||
}
|
||||
|
||||
} // namespace vdm::task
|
||||
@@ -0,0 +1,20 @@
|
||||
// vdm/task/digest.hpp — internal: hash a finished file for checksum verification.
|
||||
|
||||
#ifndef VDM_TASK_DIGEST_HPP
|
||||
#define VDM_TASK_DIGEST_HPP
|
||||
|
||||
#include <string>
|
||||
#include <string_view>
|
||||
|
||||
#include "vdm/task/download.hpp"
|
||||
#include "vdm/util/result.hpp"
|
||||
|
||||
namespace vdm::task {
|
||||
|
||||
// Stream `path` through the digest and return it lower-case hex. io_error on a read
|
||||
// failure, path_rejected if the file can't be opened.
|
||||
[[nodiscard]] Result<std::string> hash_file(std::string_view path, Checksum::Algo algo);
|
||||
|
||||
} // namespace vdm::task
|
||||
|
||||
#endif // VDM_TASK_DIGEST_HPP
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,58 @@
|
||||
// vdm/task/download_task.hpp — internal: the task machine behind DownloadHandle, and the
|
||||
// narrow interface it uses to reach engine-owned resources (so this TU doesn't depend on
|
||||
// Engine::Impl).
|
||||
|
||||
#ifndef VDM_TASK_DOWNLOAD_TASK_HPP
|
||||
#define VDM_TASK_DOWNLOAD_TASK_HPP
|
||||
|
||||
#include <cstdint>
|
||||
#include <functional>
|
||||
#include <memory>
|
||||
|
||||
#include "vdm/engine.hpp"
|
||||
#include "vdm/ids.hpp"
|
||||
#include "vdm/net/http_client.hpp"
|
||||
#include "vdm/net/probe.hpp"
|
||||
#include "vdm/rate/token_bucket.hpp"
|
||||
#include "vdm/segment/budget.hpp"
|
||||
#include "vdm/task/download.hpp"
|
||||
|
||||
namespace vdm::task {
|
||||
|
||||
using TimerId = std::uint64_t;
|
||||
|
||||
// Implemented by Engine::Impl. Every method is safe to call from any thread.
|
||||
struct TaskHost {
|
||||
virtual ~TaskHost() = default;
|
||||
|
||||
virtual net::HttpClient &http() = 0;
|
||||
virtual segment::SegmentBudget &budget() = 0;
|
||||
virtual rate::RateLimiter &limiter() = 0;
|
||||
virtual const Engine::Config &config() = 0;
|
||||
|
||||
// One-shot timer. `fn` runs on the engine's timer thread. cancel_timer is a no-op if
|
||||
// it already fired or never existed.
|
||||
virtual TimerId schedule(SteadyTime at, std::function<void()> fn) = 0;
|
||||
virtual void cancel_timer(TimerId id) = 0;
|
||||
|
||||
// Probe pool, outside the segment budget (ADR 0011 §5).
|
||||
virtual void probe(net::ProbeRequest req,
|
||||
std::function<void(Result<net::ProbeResult>)> done) = 0;
|
||||
|
||||
// The task reached a terminal state — drop it from the engine's registry.
|
||||
virtual void task_retired(TaskId id) = 0;
|
||||
};
|
||||
|
||||
// Create a task and begin it (probe or connect). The returned control block is what
|
||||
// DownloadHandle wraps (DownloadHandle{state}); the engine keeps its own copy so the task
|
||||
// outlives a caller that drops its handle.
|
||||
[[nodiscard]] std::shared_ptr<DownloadTaskState> create_task(TaskHost &host, TaskId id,
|
||||
DownloadSpec spec,
|
||||
DownloadCallbacks cbs);
|
||||
|
||||
// Engine shutdown: stop every transfer and fire no further callbacks. Safe on nullptr.
|
||||
void quiesce_task(const std::shared_ptr<DownloadTaskState> &s);
|
||||
|
||||
} // namespace vdm::task
|
||||
|
||||
#endif // VDM_TASK_DOWNLOAD_TASK_HPP
|
||||
@@ -20,3 +20,38 @@ vdm_add_test(veloxcore_event_bus_test util/event_bus_test.cpp)
|
||||
vdm_add_test(veloxcore_thread_pool_test util/thread_pool_test.cpp)
|
||||
vdm_add_test(veloxcore_bytes_test util/bytes_test.cpp)
|
||||
vdm_add_test(veloxcore_log_test util/log_test.cpp)
|
||||
|
||||
# net/ integration tests drive tools/testserver (lane PKG/QA). Skip cleanly if it isn't
|
||||
# in the tree yet (lanes merge independently).
|
||||
vdm_add_test(veloxcore_content_disposition_test net/content_disposition_test.cpp)
|
||||
vdm_add_test(veloxcore_url_test net/url_test.cpp)
|
||||
vdm_add_test(veloxcore_sparse_file_test io/sparse_file_test.cpp)
|
||||
vdm_add_test(veloxcore_write_buffer_test io/write_buffer_test.cpp)
|
||||
vdm_add_test(veloxcore_veloxpart_test meta/veloxpart_test.cpp)
|
||||
vdm_add_test(veloxcore_segmenter_test segment/segmenter_test.cpp)
|
||||
vdm_add_test(veloxcore_budget_test segment/budget_test.cpp)
|
||||
vdm_add_test(veloxcore_engine_api_test task/api_compiles_test.cpp)
|
||||
vdm_add_test(veloxcore_token_bucket_test rate/token_bucket_test.cpp)
|
||||
|
||||
set(_testserver ${CMAKE_SOURCE_DIR}/tools/testserver/testserver.py)
|
||||
vdm_add_test(veloxcore_engine_test task/engine_test.cpp)
|
||||
target_include_directories(veloxcore_engine_test PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/net ${CMAKE_SOURCE_DIR}/core/src)
|
||||
if(EXISTS ${_testserver})
|
||||
target_compile_definitions(veloxcore_engine_test PRIVATE VDM_TESTSERVER_PY="${_testserver}")
|
||||
set_tests_properties(veloxcore_engine_test PROPERTIES TIMEOUT 300)
|
||||
endif()
|
||||
|
||||
foreach(net_it http_client probe)
|
||||
vdm_add_test(veloxcore_${net_it}_test net/${net_it}_test.cpp)
|
||||
target_include_directories(veloxcore_${net_it}_test
|
||||
PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/net)
|
||||
if(EXISTS ${_testserver})
|
||||
target_compile_definitions(veloxcore_${net_it}_test
|
||||
PRIVATE VDM_TESTSERVER_PY="${_testserver}")
|
||||
set_tests_properties(veloxcore_${net_it}_test PROPERTIES TIMEOUT 120)
|
||||
endif()
|
||||
endforeach()
|
||||
if(NOT EXISTS ${_testserver})
|
||||
message(STATUS "veloxcore: tools/testserver not present; net integration tests will "
|
||||
"skip their server-backed cases.")
|
||||
endif()
|
||||
|
||||
@@ -0,0 +1,198 @@
|
||||
#include "vdm/io/sparse_file.hpp"
|
||||
|
||||
#include <fcntl.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#include <array>
|
||||
#include <cstdio>
|
||||
#include <cstdlib>
|
||||
#include <cstring>
|
||||
#include <string>
|
||||
#include <thread>
|
||||
#include <vector>
|
||||
|
||||
#include "vtest.hpp"
|
||||
|
||||
using vdm::Error;
|
||||
using vdm::io::SparseFile;
|
||||
|
||||
namespace {
|
||||
|
||||
struct TempPath {
|
||||
std::string path;
|
||||
TempPath() {
|
||||
const char *dir = std::getenv("TMPDIR");
|
||||
path = (dir ? dir : "/tmp");
|
||||
path += "/vdm_sparse_test_XXXXXX";
|
||||
int fd = ::mkstemp(path.data());
|
||||
if (fd >= 0) {
|
||||
::close(fd);
|
||||
::unlink(path.c_str()); // we only wanted a unique name
|
||||
}
|
||||
}
|
||||
~TempPath() { ::unlink(path.c_str()); }
|
||||
};
|
||||
|
||||
std::vector<std::byte> read_all(const std::string &path) {
|
||||
int fd = ::open(path.c_str(), O_RDONLY);
|
||||
if (fd < 0)
|
||||
return {};
|
||||
std::vector<std::byte> out;
|
||||
std::array<std::byte, 4096> buf{};
|
||||
for (;;) {
|
||||
ssize_t n = ::read(fd, buf.data(), buf.size());
|
||||
if (n <= 0)
|
||||
break;
|
||||
out.insert(out.end(), buf.begin(), buf.begin() + n);
|
||||
}
|
||||
::close(fd);
|
||||
return out;
|
||||
}
|
||||
|
||||
std::uint64_t file_size(const std::string &path) {
|
||||
int fd = ::open(path.c_str(), O_RDONLY);
|
||||
if (fd < 0)
|
||||
return 0;
|
||||
off_t end = ::lseek(fd, 0, SEEK_END);
|
||||
::close(fd);
|
||||
return end < 0 ? 0 : static_cast<std::uint64_t>(end);
|
||||
}
|
||||
|
||||
vdm::ConstByteSpan bytes(const char *s) {
|
||||
return {reinterpret_cast<const std::byte *>(s), std::strlen(s)};
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
VT_TEST(sparse_open_preallocates_full_size) {
|
||||
TempPath tp;
|
||||
SparseFile f;
|
||||
auto r = f.open(tp.path, {.total_size = 1 << 20});
|
||||
VT_REQUIRE(r.has_value());
|
||||
VT_CHECK(f.is_open());
|
||||
VT_CHECK_EQ(file_size(tp.path), 1u << 20);
|
||||
// /tmp is usually a real fs; if it's tmpfs, preallocated() is false and that's fine.
|
||||
VT_CHECK(f.close().has_value());
|
||||
}
|
||||
|
||||
VT_TEST(sparse_write_at_absolute_offsets) {
|
||||
TempPath tp;
|
||||
SparseFile f;
|
||||
VT_REQUIRE(f.open(tp.path, {.total_size = 64}).has_value());
|
||||
|
||||
VT_CHECK(f.write_at(10, bytes("hello")).has_value());
|
||||
VT_CHECK(f.write_at(40, bytes("world")).has_value());
|
||||
VT_CHECK(f.sync().has_value());
|
||||
|
||||
auto data = read_all(tp.path);
|
||||
VT_REQUIRE(data.size() == 64);
|
||||
VT_CHECK_EQ(std::memcmp(data.data() + 10, "hello", 5), 0);
|
||||
VT_CHECK_EQ(std::memcmp(data.data() + 40, "world", 5), 0);
|
||||
f.close().value();
|
||||
}
|
||||
|
||||
VT_TEST(sparse_write_past_end_grows_file) {
|
||||
TempPath tp;
|
||||
SparseFile f;
|
||||
VT_REQUIRE(f.open(tp.path, {.total_size = 16}).has_value());
|
||||
VT_CHECK(f.write_at(1000, bytes("tail")).has_value());
|
||||
VT_CHECK_EQ(file_size(tp.path), 1004u);
|
||||
f.close().value();
|
||||
}
|
||||
|
||||
VT_TEST(sparse_resize_trims) {
|
||||
TempPath tp;
|
||||
SparseFile f;
|
||||
VT_REQUIRE(f.open(tp.path, {.total_size = 4096}).has_value());
|
||||
VT_CHECK(f.resize(100).has_value());
|
||||
VT_CHECK_EQ(file_size(tp.path), 100u);
|
||||
f.close().value();
|
||||
}
|
||||
|
||||
VT_TEST(sparse_open_bad_path_is_path_rejected) {
|
||||
SparseFile f;
|
||||
auto r = f.open("/vdm_no_such_dir_xyz/file.part", {.total_size = 10});
|
||||
VT_REQUIRE(!r.has_value());
|
||||
VT_CHECK_EQ(r.error().code, Error::path_rejected);
|
||||
VT_CHECK(!f.is_open());
|
||||
}
|
||||
|
||||
VT_TEST(sparse_symlinked_target_is_rejected) {
|
||||
// A symlink swapped in as the final path component after DAEMON's canonicalise-and-check
|
||||
// must not be followed: the open is O_NOFOLLOW, so it fails with ELOOP -> path_rejected
|
||||
// rather than redirecting our writes through the link.
|
||||
TempPath link; // the download target the caller hands us
|
||||
TempPath target; // where the symlink points (would-be victim, outside allowed roots)
|
||||
VT_REQUIRE(::symlink(target.path.c_str(), link.path.c_str()) == 0);
|
||||
|
||||
SparseFile f;
|
||||
auto r = f.open(link.path, {.total_size = 4096});
|
||||
VT_REQUIRE(!r.has_value());
|
||||
VT_CHECK_EQ(r.error().code, Error::path_rejected);
|
||||
VT_CHECK(!f.is_open());
|
||||
// the link target was never created/written through
|
||||
VT_CHECK_EQ(::access(target.path.c_str(), F_OK), -1);
|
||||
}
|
||||
|
||||
VT_TEST(sparse_ops_on_closed_file_error) {
|
||||
SparseFile f;
|
||||
VT_CHECK_EQ(f.write_at(0, bytes("x")).error().code, Error::internal);
|
||||
VT_CHECK_EQ(f.sync().error().code, Error::internal);
|
||||
VT_CHECK(f.close().has_value()); // close on a closed file is ok
|
||||
}
|
||||
|
||||
VT_TEST(sparse_advise_dontneed_is_safe) {
|
||||
TempPath tp;
|
||||
SparseFile f;
|
||||
VT_REQUIRE(f.open(tp.path, {.total_size = 8192}).has_value());
|
||||
VT_CHECK(f.write_at(0, bytes("data")).has_value());
|
||||
VT_CHECK(f.sync().has_value());
|
||||
f.advise_dontneed(0, 4096); // must not crash / must be a no-op-safe call
|
||||
f.advise_dontneed(0, 0);
|
||||
f.close().value();
|
||||
}
|
||||
|
||||
VT_TEST(sparse_move_transfers_fd) {
|
||||
TempPath tp;
|
||||
SparseFile a;
|
||||
VT_REQUIRE(a.open(tp.path, {.total_size = 32}).has_value());
|
||||
SparseFile b = std::move(a);
|
||||
VT_CHECK(!a.is_open());
|
||||
VT_CHECK(b.is_open());
|
||||
VT_CHECK(b.write_at(0, bytes("moved")).has_value());
|
||||
b.close().value();
|
||||
}
|
||||
|
||||
VT_TEST(sparse_concurrent_nonoverlapping_writes) {
|
||||
TempPath tp;
|
||||
SparseFile f;
|
||||
constexpr int kSegs = 8;
|
||||
constexpr std::size_t kSeg = 64 * 1024;
|
||||
VT_REQUIRE(f.open(tp.path, {.total_size = kSegs * kSeg}).has_value());
|
||||
|
||||
std::vector<std::jthread> ts;
|
||||
for (int s = 0; s < kSegs; ++s) {
|
||||
ts.emplace_back([&, s] {
|
||||
std::vector<std::byte> chunk(kSeg, static_cast<std::byte>('A' + s));
|
||||
for (std::size_t off = 0; off < kSeg; off += 4096) {
|
||||
auto r = f.write_at(static_cast<std::uint64_t>(s) * kSeg + off,
|
||||
vdm::ConstByteSpan(chunk.data() + off, 4096));
|
||||
if (!r.has_value())
|
||||
VT_FAIL("concurrent write_at failed");
|
||||
}
|
||||
});
|
||||
}
|
||||
ts.clear(); // join
|
||||
VT_CHECK(f.sync().has_value());
|
||||
|
||||
auto data = read_all(tp.path);
|
||||
VT_REQUIRE(data.size() == kSegs * kSeg);
|
||||
for (int s = 0; s < kSegs; ++s) {
|
||||
bool ok = true;
|
||||
for (std::size_t i = 0; i < kSeg; ++i)
|
||||
if (data[s * kSeg + i] != static_cast<std::byte>('A' + s))
|
||||
ok = false;
|
||||
VT_CHECK(ok);
|
||||
}
|
||||
f.close().value();
|
||||
}
|
||||
@@ -0,0 +1,202 @@
|
||||
#include "vdm/io/write_buffer.hpp"
|
||||
|
||||
#include <atomic>
|
||||
#include <cstdlib>
|
||||
#include <cstring>
|
||||
#include <new>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vtest.hpp"
|
||||
|
||||
using vdm::ConstByteSpan;
|
||||
using vdm::Error;
|
||||
using vdm::Result;
|
||||
using vdm::io::WriteBuffer;
|
||||
|
||||
// --- global allocation counter, for the "no alloc in append()" test -----------------
|
||||
namespace {
|
||||
std::atomic<long> g_alloc_calls{0};
|
||||
std::atomic<bool> g_count_allocs{false};
|
||||
} // namespace
|
||||
|
||||
void *operator new(std::size_t n) {
|
||||
if (g_count_allocs.load(std::memory_order_relaxed))
|
||||
g_alloc_calls.fetch_add(1, std::memory_order_relaxed);
|
||||
void *p = std::malloc(n ? n : 1);
|
||||
if (!p)
|
||||
throw std::bad_alloc();
|
||||
return p;
|
||||
}
|
||||
void operator delete(void *p) noexcept {
|
||||
std::free(p);
|
||||
}
|
||||
void operator delete(void *p, std::size_t) noexcept {
|
||||
std::free(p);
|
||||
}
|
||||
void *operator new[](std::size_t n) {
|
||||
return ::operator new(n);
|
||||
}
|
||||
void operator delete[](void *p) noexcept {
|
||||
std::free(p);
|
||||
}
|
||||
void operator delete[](void *p, std::size_t) noexcept {
|
||||
std::free(p);
|
||||
}
|
||||
|
||||
namespace {
|
||||
|
||||
// A flush sink that records (offset, bytes) and never allocates after construction.
|
||||
struct Sink {
|
||||
std::vector<std::byte> data; // pre-reserved
|
||||
std::vector<std::uint64_t> offs; // pre-reserved
|
||||
std::vector<std::size_t> lens;
|
||||
bool fail_next = false;
|
||||
|
||||
WriteBuffer::FlushFn fn() {
|
||||
return [this](std::uint64_t off, ConstByteSpan s) -> Result<void> {
|
||||
if (fail_next) {
|
||||
fail_next = false;
|
||||
return vdm::Err{Error::io_error, "sink forced failure"};
|
||||
}
|
||||
offs.push_back(off);
|
||||
lens.push_back(s.size());
|
||||
data.insert(data.end(), s.begin(), s.end());
|
||||
return vdm::ok();
|
||||
};
|
||||
}
|
||||
};
|
||||
|
||||
ConstByteSpan sv(const char *s) {
|
||||
return {reinterpret_cast<const std::byte *>(s), std::strlen(s)};
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
VT_TEST(wb_accumulates_then_flushes_on_fill) {
|
||||
Sink sink;
|
||||
sink.data.reserve(1 << 16);
|
||||
sink.offs.reserve(64);
|
||||
sink.lens.reserve(64);
|
||||
|
||||
WriteBuffer wb(0, 8, sink.fn());
|
||||
VT_CHECK(wb.append(sv("abc")).has_value()); // 3 buffered
|
||||
VT_CHECK_EQ(wb.pending(), 3u);
|
||||
VT_CHECK(sink.offs.empty()); // no flush yet
|
||||
|
||||
VT_CHECK(wb.append(sv("defgh")).has_value()); // fills to 8 -> flush
|
||||
VT_REQUIRE(sink.offs.size() == 1);
|
||||
VT_CHECK_EQ(sink.offs[0], 0u);
|
||||
VT_CHECK_EQ(sink.lens[0], 8u);
|
||||
VT_CHECK_EQ(wb.pending(), 0u);
|
||||
VT_CHECK_EQ(wb.next_offset(), 8u);
|
||||
|
||||
VT_CHECK(wb.append(sv("ij")).has_value());
|
||||
VT_CHECK(wb.flush().has_value()); // explicit tail flush
|
||||
VT_REQUIRE(sink.offs.size() == 2);
|
||||
VT_CHECK_EQ(sink.offs[1], 8u);
|
||||
VT_CHECK_EQ(sink.lens[1], 2u);
|
||||
|
||||
VT_CHECK_EQ(std::string(reinterpret_cast<const char *>(sink.data.data()), sink.data.size()),
|
||||
std::string("abcdefghij"));
|
||||
VT_CHECK_EQ(wb.total_appended(), 10u);
|
||||
}
|
||||
|
||||
VT_TEST(wb_flush_is_noop_when_empty) {
|
||||
Sink sink;
|
||||
sink.offs.reserve(4);
|
||||
WriteBuffer wb(100, 16, sink.fn());
|
||||
VT_CHECK(wb.flush().has_value());
|
||||
VT_CHECK(sink.offs.empty());
|
||||
}
|
||||
|
||||
VT_TEST(wb_oversized_chunk_writes_through) {
|
||||
Sink sink;
|
||||
sink.data.reserve(1 << 16);
|
||||
sink.offs.reserve(16);
|
||||
sink.lens.reserve(16);
|
||||
|
||||
WriteBuffer wb(0, 8, sink.fn());
|
||||
VT_CHECK(wb.append(sv("ab")).has_value()); // 2 buffered
|
||||
// 20 bytes arriving: buffer isn't empty, so first 6 top it off + flush(8), then the
|
||||
// remaining 14 (>= capacity, buffer now empty) write straight through.
|
||||
std::string big(20, 'x');
|
||||
VT_CHECK(wb.append(sv(big.c_str())).has_value());
|
||||
VT_CHECK(wb.flush().has_value());
|
||||
|
||||
// reconstruct
|
||||
std::string got(reinterpret_cast<const char *>(sink.data.data()), sink.data.size());
|
||||
VT_CHECK_EQ(got, std::string("ab") + big);
|
||||
VT_CHECK_EQ(wb.total_appended(), 22u);
|
||||
// one full-buffer flush + one passthrough; order preserved
|
||||
VT_CHECK(sink.offs.size() >= 2);
|
||||
VT_CHECK_EQ(sink.offs.front(), 0u);
|
||||
}
|
||||
|
||||
VT_TEST(wb_exact_capacity_chunk_from_empty_writes_through) {
|
||||
Sink sink;
|
||||
sink.data.reserve(64);
|
||||
sink.offs.reserve(4);
|
||||
sink.lens.reserve(4);
|
||||
WriteBuffer wb(0, 4, sink.fn());
|
||||
VT_CHECK(wb.append(sv("wxyz")).has_value()); // == capacity, empty -> passthrough
|
||||
VT_REQUIRE(sink.offs.size() == 1);
|
||||
VT_CHECK_EQ(sink.lens[0], 4u);
|
||||
VT_CHECK_EQ(wb.pending(), 0u);
|
||||
}
|
||||
|
||||
VT_TEST(wb_flush_error_propagates_without_advancing_durable_offset) {
|
||||
Sink sink;
|
||||
sink.data.reserve(64);
|
||||
sink.offs.reserve(4);
|
||||
sink.lens.reserve(4);
|
||||
WriteBuffer wb(0, 8, sink.fn());
|
||||
|
||||
VT_CHECK(wb.append(sv("abc")).has_value()); // 3 buffered, nothing durable yet
|
||||
VT_CHECK_EQ(wb.next_offset(), 0u);
|
||||
|
||||
sink.fail_next = true;
|
||||
auto r = wb.flush(); // forced failure
|
||||
VT_REQUIRE(!r.has_value());
|
||||
VT_CHECK_EQ(r.error().code, Error::io_error);
|
||||
VT_CHECK_EQ(wb.next_offset(), 0u); // durable offset did NOT move
|
||||
VT_CHECK(sink.offs.empty());
|
||||
|
||||
// a retry flush succeeds and advances
|
||||
VT_CHECK(wb.flush().has_value());
|
||||
VT_CHECK_EQ(wb.next_offset(), 3u);
|
||||
VT_REQUIRE(sink.lens.size() == 1);
|
||||
VT_CHECK_EQ(sink.lens[0], 3u);
|
||||
}
|
||||
|
||||
VT_TEST(wb_append_does_not_allocate) {
|
||||
// flush sink that never allocates: just sum sizes.
|
||||
std::atomic<std::uint64_t> total{0};
|
||||
auto flush = [&total](std::uint64_t, ConstByteSpan s) -> Result<void> {
|
||||
total.fetch_add(s.size());
|
||||
return vdm::ok();
|
||||
};
|
||||
WriteBuffer wb(0, 4096, flush);
|
||||
|
||||
// Warm up (any first-call lazy init happens now, outside the measured window).
|
||||
std::string warm(100, 'w');
|
||||
(void)wb.append(sv(warm.c_str()));
|
||||
(void)wb.flush();
|
||||
|
||||
std::vector<std::byte> chunk(512, std::byte{7});
|
||||
g_alloc_calls.store(0);
|
||||
g_count_allocs.store(true);
|
||||
for (int i = 0; i < 5000; ++i) {
|
||||
auto r = wb.append(ConstByteSpan(chunk.data(), 137 + (i % 200)));
|
||||
if (!r.has_value()) {
|
||||
g_count_allocs.store(false);
|
||||
VT_FAIL("append failed");
|
||||
return;
|
||||
}
|
||||
}
|
||||
(void)wb.flush();
|
||||
g_count_allocs.store(false);
|
||||
|
||||
VT_CHECK_EQ(g_alloc_calls.load(), 0L);
|
||||
VT_CHECK(total.load() > 0);
|
||||
}
|
||||
@@ -0,0 +1,246 @@
|
||||
#include "vdm/meta/veloxpart.hpp"
|
||||
|
||||
#include <unistd.h>
|
||||
|
||||
#include <cstdint>
|
||||
#include <cstdlib>
|
||||
#include <cstring>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vdm/util/crc32.hpp"
|
||||
#include "vtest.hpp"
|
||||
|
||||
using namespace vdm;
|
||||
using namespace vdm::meta;
|
||||
|
||||
namespace {
|
||||
|
||||
VeloxPart sample_full() {
|
||||
VeloxPart vp;
|
||||
vp.version = kVersion;
|
||||
vp.total_size = 5'000'000'000ull;
|
||||
vp.downloaded = 1'234'567;
|
||||
vp.urls = {"https://origin.example/big.iso", "https://cdn.example/big.iso?sig=abc",
|
||||
"https://mirror1.example/big.iso"};
|
||||
vp.etag = "\"deadbeef-1234\"";
|
||||
vp.last_modified = "Wed, 01 Jan 2025 00:00:00 GMT";
|
||||
vp.content_type = "application/octet-stream";
|
||||
for (int i = 0; i < 8; ++i) {
|
||||
SegmentRecord s;
|
||||
s.start = static_cast<std::uint64_t>(i) * 625'000'000ull;
|
||||
s.end = s.start + 625'000'000ull - 1; // inclusive
|
||||
s.completed = (i < 2) ? s.end - s.start + 1 : 100'000ull * (i + 1);
|
||||
vp.segments.push_back(s);
|
||||
}
|
||||
vp.flags = kFlagHasShaState;
|
||||
vp.sha256_state = {std::byte{1}, std::byte{2}, std::byte{3}, std::byte{0xAA}, std::byte{0xFF}};
|
||||
return vp;
|
||||
}
|
||||
|
||||
VeloxPart sample_minimal() {
|
||||
VeloxPart vp;
|
||||
vp.total_size = 0; // chunked / unknown
|
||||
vp.urls = {"http://x/y"};
|
||||
SegmentRecord s;
|
||||
s.start = 0;
|
||||
s.end = 0; // 1-byte resource
|
||||
s.completed = 0;
|
||||
vp.segments.push_back(s);
|
||||
return vp;
|
||||
}
|
||||
|
||||
// Re-CRC a mutated body (everything except the final u32).
|
||||
std::vector<std::byte> refresh_crc(std::vector<std::byte> image) {
|
||||
std::uint32_t c = crc32(ConstByteSpan(image.data(), image.size() - 4));
|
||||
for (int i = 0; i < 4; ++i)
|
||||
image[image.size() - 4 + i] = static_cast<std::byte>((c >> (8 * i)) & 0xFF);
|
||||
return image;
|
||||
}
|
||||
|
||||
struct TempPath {
|
||||
std::string path;
|
||||
TempPath() {
|
||||
const char *d = std::getenv("TMPDIR");
|
||||
path = (d ? d : "/tmp");
|
||||
path += "/vdm_vp_test_XXXXXX";
|
||||
int fd = ::mkstemp(path.data());
|
||||
if (fd >= 0) {
|
||||
::close(fd);
|
||||
::unlink(path.c_str());
|
||||
}
|
||||
}
|
||||
~TempPath() {
|
||||
::unlink(path.c_str());
|
||||
::unlink((path + ".tmp").c_str());
|
||||
}
|
||||
};
|
||||
|
||||
} // namespace
|
||||
|
||||
VT_TEST(crc32_known_vector) {
|
||||
const char *s = "123456789";
|
||||
VT_CHECK_EQ(crc32(ConstByteSpan(reinterpret_cast<const std::byte *>(s), 9)), 0xCBF43926u);
|
||||
VT_CHECK_EQ(crc32(ConstByteSpan{}), 0u);
|
||||
}
|
||||
|
||||
VT_TEST(vp_roundtrip_full) {
|
||||
VeloxPart in = sample_full();
|
||||
auto image = serialize_veloxpart(in);
|
||||
auto out = parse_veloxpart(ConstByteSpan(image.data(), image.size()));
|
||||
VT_REQUIRE(out.has_value());
|
||||
VT_CHECK(out.value() == in);
|
||||
VT_CHECK_EQ(out.value().effective_url(),
|
||||
std::string_view("https://cdn.example/big.iso?sig=abc"));
|
||||
VT_CHECK_EQ(out.value().segments.size(), 8u);
|
||||
}
|
||||
|
||||
VT_TEST(vp_roundtrip_minimal) {
|
||||
VeloxPart in = sample_minimal();
|
||||
auto image = serialize_veloxpart(in);
|
||||
auto out = parse_veloxpart(ConstByteSpan(image.data(), image.size()));
|
||||
VT_REQUIRE(out.has_value());
|
||||
VT_CHECK(out.value() == in);
|
||||
VT_CHECK(out.value().sha256_state.empty());
|
||||
}
|
||||
|
||||
VT_TEST(vp_serialize_is_deterministic) {
|
||||
VeloxPart in = sample_full();
|
||||
auto a = serialize_veloxpart(in);
|
||||
auto b = serialize_veloxpart(in);
|
||||
VT_CHECK(a == b);
|
||||
}
|
||||
|
||||
VT_TEST(vp_file_roundtrip) {
|
||||
TempPath tp;
|
||||
VeloxPart in = sample_full();
|
||||
VT_REQUIRE(write_veloxpart_file(tp.path, in, /*fsync=*/true).has_value());
|
||||
auto out = read_veloxpart_file(tp.path);
|
||||
VT_REQUIRE(out.has_value());
|
||||
VT_CHECK(out.value() == in);
|
||||
// the temp file must be gone after the atomic rename
|
||||
VT_CHECK_EQ(::access((tp.path + ".tmp").c_str(), F_OK), -1);
|
||||
}
|
||||
|
||||
VT_TEST(vp_read_missing_file_errors) {
|
||||
auto out = read_veloxpart_file("/vdm_no_such_dir_zz/x.veloxpart.meta");
|
||||
VT_REQUIRE(!out.has_value());
|
||||
VT_CHECK_EQ(out.error().code, Error::path_rejected);
|
||||
}
|
||||
|
||||
// --- truncation / corruption table -------------------------------------------------
|
||||
|
||||
VT_TEST(vp_reject_empty_and_tiny) {
|
||||
VT_CHECK_EQ(parse_veloxpart(ConstByteSpan{}).error().code, Error::meta_corrupt);
|
||||
std::array<std::byte, 3> three{};
|
||||
VT_CHECK_EQ(parse_veloxpart(ConstByteSpan(three.data(), 3)).error().code, Error::meta_corrupt);
|
||||
std::array<std::byte, 51> almost{};
|
||||
VT_CHECK_EQ(parse_veloxpart(ConstByteSpan(almost.data(), almost.size())).error().code,
|
||||
Error::meta_corrupt);
|
||||
}
|
||||
|
||||
VT_TEST(vp_reject_bad_magic) {
|
||||
auto image = serialize_veloxpart(sample_minimal());
|
||||
image[1] = std::byte{'X'}; // "VDMP" -> "VXMP"
|
||||
image = refresh_crc(std::move(image)); // fix CRC so we're testing the magic check
|
||||
auto r = parse_veloxpart(ConstByteSpan(image.data(), image.size()));
|
||||
VT_REQUIRE(!r.has_value());
|
||||
VT_CHECK_EQ(r.error().code, Error::meta_corrupt);
|
||||
}
|
||||
|
||||
VT_TEST(vp_reject_crc_mismatch) {
|
||||
auto image = serialize_veloxpart(sample_minimal());
|
||||
image[20] ^= std::byte{0x40}; // flip a payload bit, do NOT refresh the CRC
|
||||
auto r = parse_veloxpart(ConstByteSpan(image.data(), image.size()));
|
||||
VT_REQUIRE(!r.has_value());
|
||||
VT_CHECK_EQ(r.error().code, Error::meta_corrupt);
|
||||
|
||||
// flipping the CRC field itself is also a mismatch
|
||||
auto image2 = serialize_veloxpart(sample_minimal());
|
||||
image2.back() ^= std::byte{0xFF};
|
||||
VT_CHECK_EQ(parse_veloxpart(ConstByteSpan(image2.data(), image2.size())).error().code,
|
||||
Error::meta_corrupt);
|
||||
}
|
||||
|
||||
VT_TEST(vp_future_version_is_unsupported) {
|
||||
auto image = serialize_veloxpart(sample_minimal());
|
||||
image[4] = std::byte{99}; // version u16 low byte
|
||||
image[5] = std::byte{0};
|
||||
image = refresh_crc(std::move(image));
|
||||
auto r = parse_veloxpart(ConstByteSpan(image.data(), image.size()));
|
||||
VT_REQUIRE(!r.has_value());
|
||||
VT_CHECK_EQ(r.error().code, Error::meta_version_unsupported);
|
||||
}
|
||||
|
||||
VT_TEST(vp_reject_truncation_at_every_stage) {
|
||||
auto full = serialize_veloxpart(sample_full());
|
||||
// Chop the image at many lengths; each must be meta_corrupt, never a crash.
|
||||
for (std::size_t len = full.size() - 1; len >= 1; len = (len > 8 ? len - 7 : len - 1)) {
|
||||
auto r = parse_veloxpart(ConstByteSpan(full.data(), len));
|
||||
VT_CHECK(!r.has_value());
|
||||
if (r.has_value())
|
||||
break;
|
||||
VT_CHECK_EQ(r.error().code, Error::meta_corrupt);
|
||||
if (len == 1)
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
VT_TEST(vp_reject_hostile_url_count) {
|
||||
auto image = serialize_veloxpart(sample_minimal());
|
||||
// url_count u32 sits right after magic(4)+ver(2)+flags(2)+total(8)+downloaded(8) = 24
|
||||
for (int i = 0; i < 4; ++i)
|
||||
image[24 + i] = std::byte{0xFF};
|
||||
image = refresh_crc(std::move(image));
|
||||
auto r = parse_veloxpart(ConstByteSpan(image.data(), image.size()));
|
||||
VT_REQUIRE(!r.has_value());
|
||||
VT_CHECK_EQ(r.error().code, Error::meta_corrupt);
|
||||
}
|
||||
|
||||
VT_TEST(vp_reject_hostile_lp_string_length) {
|
||||
// The case AGENT-CORE singles out: a length prefix that reaches past the buffer.
|
||||
// Build a minimal image, then overwrite url[0]'s length prefix with a huge value.
|
||||
auto image = serialize_veloxpart(sample_minimal());
|
||||
// layout up to url[0] length: magic4 ver2 flags2 total8 downloaded8 url_count4 = 28
|
||||
for (int i = 0; i < 4; ++i)
|
||||
image[28 + i] = std::byte{0xFF}; // url[0] len = 4 GiB - 1
|
||||
image = refresh_crc(std::move(image));
|
||||
auto r = parse_veloxpart(ConstByteSpan(image.data(), image.size()));
|
||||
VT_REQUIRE(!r.has_value());
|
||||
VT_CHECK_EQ(r.error().code, Error::meta_corrupt);
|
||||
}
|
||||
|
||||
VT_TEST(vp_reject_hostile_segment_count) {
|
||||
auto image = serialize_veloxpart(sample_minimal());
|
||||
// find the segment_count: it's u32 right before the (24-byte) segment records and the
|
||||
// trailing crc. minimal sample has 1 segment and no sha state, so:
|
||||
// segment_count is at size - 4(crc) - 24(one segment) - 4 = size - 32
|
||||
std::size_t sc = image.size() - 32;
|
||||
for (int i = 0; i < 4; ++i)
|
||||
image[sc + i] = std::byte{0xFF};
|
||||
image = refresh_crc(std::move(image));
|
||||
auto r = parse_veloxpart(ConstByteSpan(image.data(), image.size()));
|
||||
VT_REQUIRE(!r.has_value());
|
||||
VT_CHECK_EQ(r.error().code, Error::meta_corrupt);
|
||||
}
|
||||
|
||||
VT_TEST(vp_reject_trailing_bytes) {
|
||||
auto image = serialize_veloxpart(sample_minimal());
|
||||
image.push_back(std::byte{0});
|
||||
image.push_back(std::byte{0});
|
||||
image = refresh_crc(std::move(image)); // CRC now covers the padding too
|
||||
auto r = parse_veloxpart(ConstByteSpan(image.data(), image.size()));
|
||||
VT_REQUIRE(!r.has_value());
|
||||
VT_CHECK_EQ(r.error().code, Error::meta_corrupt);
|
||||
}
|
||||
|
||||
VT_TEST(vp_reject_segment_completed_over_length) {
|
||||
VeloxPart in = sample_minimal();
|
||||
in.segments[0].start = 0;
|
||||
in.segments[0].end = 99; // length 100
|
||||
in.segments[0].completed = 500; // impossible
|
||||
auto image = serialize_veloxpart(in);
|
||||
auto r = parse_veloxpart(ConstByteSpan(image.data(), image.size()));
|
||||
VT_REQUIRE(!r.has_value());
|
||||
VT_CHECK_EQ(r.error().code, Error::meta_corrupt);
|
||||
}
|
||||
@@ -0,0 +1,165 @@
|
||||
#include "vdm/net/content_disposition.hpp"
|
||||
|
||||
#include <string>
|
||||
#include <string_view>
|
||||
|
||||
#include "vtest.hpp"
|
||||
|
||||
using vdm::net::ContentDisposition;
|
||||
using vdm::net::parse_content_disposition;
|
||||
using Type = vdm::net::ContentDisposition::Type;
|
||||
|
||||
namespace {
|
||||
// "€" is U+20AC -> UTF-8 E2 82 AC. "£" is U+00A3 -> UTF-8 C2 A3.
|
||||
const std::string kEuro = "\xE2\x82\xAC";
|
||||
const std::string kPound = "\xC2\xA3";
|
||||
const std::string kEAcute = "\xC3\xA9"; // é U+00E9
|
||||
} // namespace
|
||||
|
||||
VT_TEST(cd_plain_quoted) {
|
||||
auto cd = parse_content_disposition(R"(attachment; filename="report.pdf")");
|
||||
VT_CHECK(cd.type == Type::attachment);
|
||||
VT_CHECK_EQ(cd.filename, std::string("report.pdf"));
|
||||
VT_CHECK(!cd.filename_from_ext);
|
||||
}
|
||||
|
||||
VT_TEST(cd_plain_token_unquoted) {
|
||||
auto cd = parse_content_disposition("attachment; filename=report.pdf");
|
||||
VT_CHECK_EQ(cd.filename, std::string("report.pdf"));
|
||||
}
|
||||
|
||||
VT_TEST(cd_inline_no_filename) {
|
||||
auto cd = parse_content_disposition("inline");
|
||||
VT_CHECK(cd.type == Type::inline_);
|
||||
VT_CHECK(!cd.has_filename());
|
||||
}
|
||||
|
||||
VT_TEST(cd_rfc5987_utf8_ext_value) {
|
||||
auto cd = parse_content_disposition("attachment; filename*=UTF-8''%e2%82%ac%20rates.pdf");
|
||||
VT_CHECK_EQ(cd.filename, kEuro + " rates.pdf");
|
||||
VT_CHECK(cd.filename_from_ext);
|
||||
}
|
||||
|
||||
VT_TEST(cd_prefers_ext_over_plain) {
|
||||
auto cd = parse_content_disposition(
|
||||
R"(attachment; filename="EURO rates.pdf"; filename*=UTF-8''%e2%82%ac%20rates.pdf)");
|
||||
VT_CHECK_EQ(cd.filename, kEuro + " rates.pdf");
|
||||
VT_CHECK(cd.filename_from_ext);
|
||||
}
|
||||
|
||||
VT_TEST(cd_rfc5987_latin1_ext_value) {
|
||||
// %A3 = £ in ISO-8859-1
|
||||
auto cd = parse_content_disposition("attachment; filename*=ISO-8859-1''%A3rates.pdf");
|
||||
VT_CHECK_EQ(cd.filename, kPound + "rates.pdf");
|
||||
}
|
||||
|
||||
VT_TEST(cd_legacy_rfc2047_base64) {
|
||||
// base64("<euro> rates.pdf") with euro as UTF-8
|
||||
// "€ rates.pdf" -> bytes E2 82 AC 20 72 61 74 65 73 2E 70 64 66 -> base64:
|
||||
auto cd =
|
||||
parse_content_disposition(R"(attachment; filename="=?UTF-8?B?4oKsIHJhdGVzLnBkZg==?=")");
|
||||
VT_CHECK_EQ(cd.filename, kEuro + " rates.pdf");
|
||||
}
|
||||
|
||||
VT_TEST(cd_legacy_rfc2047_qencoded_latin1) {
|
||||
// =?ISO-8859-1?Q?=A3rates.pdf?= -> £rates.pdf
|
||||
auto cd = parse_content_disposition(R"(attachment; filename="=?ISO-8859-1?Q?=A3rates.pdf?=")");
|
||||
VT_CHECK_EQ(cd.filename, kPound + "rates.pdf");
|
||||
}
|
||||
|
||||
VT_TEST(cd_raw_utf8_bytes_in_quotes) {
|
||||
std::string h = "attachment; filename=\"caf" + kEAcute + ".txt\"";
|
||||
auto cd = parse_content_disposition(h);
|
||||
VT_CHECK_EQ(cd.filename, "caf" + kEAcute + ".txt");
|
||||
}
|
||||
|
||||
VT_TEST(cd_raw_latin1_byte_in_quotes) {
|
||||
// 0xE9 is 'é' in Latin-1; not valid UTF-8 alone -> transcoded
|
||||
std::string h = "attachment; filename=\"caf\xE9.txt\"";
|
||||
auto cd = parse_content_disposition(h);
|
||||
VT_CHECK_EQ(cd.filename, "caf" + kEAcute + ".txt");
|
||||
}
|
||||
|
||||
VT_TEST(cd_strips_path_components) {
|
||||
VT_CHECK_EQ(parse_content_disposition(R"(attachment; filename="../../etc/passwd")").filename,
|
||||
std::string("passwd"));
|
||||
// real Windows paths in the wild use unescaped backslashes as separators
|
||||
VT_CHECK_EQ(parse_content_disposition(R"(attachment; filename="C:\Windows\evil.exe")").filename,
|
||||
std::string("evil.exe"));
|
||||
// a base64 payload can contain '/', so decode must happen before path stripping
|
||||
VT_CHECK_EQ(parse_content_disposition(R"(attachment; filename="=?UTF-8?B?Li4vLi4vc2VjcmV0?=")")
|
||||
.filename,
|
||||
std::string("secret")); // decodes to "../../secret", then stripped
|
||||
}
|
||||
|
||||
VT_TEST(cd_quoted_dquote_escape) {
|
||||
// \" is the one escape we resolve, so a quote can appear mid-name
|
||||
auto cd = parse_content_disposition(R"(attachment; filename="quote\"here.txt")");
|
||||
VT_CHECK_EQ(cd.filename, std::string("quote\"here.txt"));
|
||||
}
|
||||
|
||||
VT_TEST(cd_semicolon_inside_quotes_is_not_a_separator) {
|
||||
auto cd = parse_content_disposition(R"(attachment; filename="a;b;c.txt")");
|
||||
VT_CHECK_EQ(cd.filename, std::string("a;b;c.txt"));
|
||||
}
|
||||
|
||||
VT_TEST(cd_form_data) {
|
||||
auto cd = parse_content_disposition(R"(form-data; name="file"; filename="upload.bin")");
|
||||
VT_CHECK(cd.type == Type::form_data);
|
||||
VT_CHECK_EQ(cd.filename, std::string("upload.bin"));
|
||||
}
|
||||
|
||||
VT_TEST(cd_rfc2231_continuations) {
|
||||
auto cd = parse_content_disposition(
|
||||
"attachment; filename*0*=UTF-8''%e2%82%ac; filename*1*=%20rates; filename*2=.pdf");
|
||||
VT_CHECK_EQ(cd.filename, kEuro + " rates.pdf");
|
||||
}
|
||||
|
||||
VT_TEST(cd_empty_and_garbage_do_not_crash) {
|
||||
VT_CHECK(!parse_content_disposition("").has_filename());
|
||||
VT_CHECK(!parse_content_disposition(";;;;").has_filename());
|
||||
VT_CHECK(!parse_content_disposition("attachment;").has_filename());
|
||||
VT_CHECK(!parse_content_disposition(R"(attachment; filename=)").has_filename());
|
||||
VT_CHECK(!parse_content_disposition(R"(attachment; filename="")").has_filename());
|
||||
// truncated ext-value
|
||||
auto cd = parse_content_disposition("attachment; filename*=UTF-8''%e2%82");
|
||||
VT_CHECK(cd.type == Type::attachment); // no crash; filename is whatever fell out
|
||||
// truncated encoded-word
|
||||
auto trunc = parse_content_disposition(R"(attachment; filename="=?UTF-8?B?4oKs")");
|
||||
(void)trunc;
|
||||
}
|
||||
|
||||
VT_TEST(cd_bad_percent_escapes_in_ext_value) {
|
||||
// stray % and non-hex digits are emitted literally, no crash
|
||||
auto cd = parse_content_disposition("attachment; filename*=UTF-8''%ZZ%%file%2");
|
||||
VT_CHECK(cd.type == Type::attachment);
|
||||
}
|
||||
|
||||
VT_TEST(cd_strips_control_bytes_and_nul) {
|
||||
// A mangled ext-value that decodes to bytes with embedded NULs (fuzz-found).
|
||||
std::string h1("attachment; filename*=x''%e2%82%a");
|
||||
h1.push_back('\0');
|
||||
h1.push_back('\0');
|
||||
h1 += "ff.pdf";
|
||||
auto cd = parse_content_disposition(h1);
|
||||
for (unsigned char c : cd.filename)
|
||||
VT_CHECK(c >= 0x20 && c != 0x7F);
|
||||
// a plain filename with a tab / newline / SOH loses them
|
||||
std::string h2("attachment; filename=\"a\tb\nc");
|
||||
h2.push_back('\x01');
|
||||
h2 += ".txt\"";
|
||||
auto cd2 = parse_content_disposition(h2);
|
||||
VT_CHECK_EQ(cd2.filename, std::string("abc.txt"));
|
||||
}
|
||||
|
||||
VT_TEST(cd_case_insensitive_keys_and_type) {
|
||||
auto cd = parse_content_disposition(R"(ATTACHMENT; FileName="x.txt")");
|
||||
VT_CHECK(cd.type == Type::attachment);
|
||||
VT_CHECK_EQ(cd.filename, std::string("x.txt"));
|
||||
}
|
||||
|
||||
VT_TEST(cd_unknown_type_is_other) {
|
||||
auto cd = parse_content_disposition(R"(signal; filename="x.txt")");
|
||||
VT_CHECK(cd.type == Type::other);
|
||||
VT_CHECK_EQ(cd.filename, std::string("x.txt"));
|
||||
}
|
||||
@@ -0,0 +1,270 @@
|
||||
#include "vdm/net/http_client.hpp"
|
||||
|
||||
#include <atomic>
|
||||
#include <chrono>
|
||||
#include <future>
|
||||
#include <mutex>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "testserver_fixture.hpp"
|
||||
#include "vtest.hpp"
|
||||
|
||||
using namespace vdm;
|
||||
using namespace vdm::net;
|
||||
using vdm::testing::TestServer;
|
||||
|
||||
namespace {
|
||||
|
||||
// Collects callback output from a transfer and lets the test thread wait for the end.
|
||||
struct Recorder {
|
||||
std::mutex mu;
|
||||
ResponseHead head;
|
||||
bool head_seen = false;
|
||||
std::uint64_t bytes = 0;
|
||||
std::promise<Result<TransferStats>> done;
|
||||
std::future<Result<TransferStats>> done_fut = done.get_future();
|
||||
|
||||
DataAction want_on_head = DataAction::proceed; // set before start()
|
||||
std::atomic<DataAction> want_on_data{DataAction::proceed};
|
||||
std::atomic<int> data_calls{0};
|
||||
|
||||
TransferCallbacks callbacks() {
|
||||
return TransferCallbacks{
|
||||
.on_head =
|
||||
[this](const ResponseHead &h) {
|
||||
std::lock_guard lk(mu);
|
||||
head = h;
|
||||
head_seen = true;
|
||||
return want_on_head;
|
||||
},
|
||||
.on_data =
|
||||
[this](ConstByteSpan s) {
|
||||
data_calls.fetch_add(1);
|
||||
std::lock_guard lk(mu);
|
||||
bytes += s.size();
|
||||
return want_on_data.load();
|
||||
},
|
||||
.on_finished = [this](Result<TransferStats> r) { done.set_value(std::move(r)); },
|
||||
};
|
||||
}
|
||||
|
||||
Result<TransferStats> wait(std::chrono::seconds to = std::chrono::seconds(20)) {
|
||||
if (done_fut.wait_for(to) != std::future_status::ready)
|
||||
return Err{Error::timeout, "test wait timed out"};
|
||||
return done_fut.get();
|
||||
}
|
||||
};
|
||||
|
||||
// on_data needs an atomic for the cancel/pause tests to flip it from the test thread.
|
||||
struct AtomicAction {
|
||||
std::atomic<DataAction> a{DataAction::proceed};
|
||||
void store(DataAction v) { a.store(v); }
|
||||
operator DataAction() const { return a.load(); }
|
||||
DataAction load() const { return a.load(); }
|
||||
};
|
||||
|
||||
} // namespace
|
||||
|
||||
VT_TEST(http_plain_full_get) {
|
||||
TestServer srv;
|
||||
VT_REQUIRE(srv.available());
|
||||
HttpClient client({.workers = 1});
|
||||
|
||||
Recorder rec;
|
||||
Request req;
|
||||
req.url = srv.url("/plain/file/64K");
|
||||
auto t = client.start(std::move(req), rec.callbacks());
|
||||
|
||||
auto r = rec.wait();
|
||||
VT_REQUIRE(r.has_value());
|
||||
VT_CHECK_EQ(r.value().http_status, 200);
|
||||
VT_CHECK_EQ(r.value().bytes_received, 65536u);
|
||||
VT_CHECK(rec.head_seen);
|
||||
VT_CHECK_EQ(rec.head.status, 200);
|
||||
VT_CHECK(rec.head.headers.has("Accept-Ranges"));
|
||||
VT_CHECK_EQ(rec.bytes, 65536u);
|
||||
VT_CHECK(t.id() != 0);
|
||||
}
|
||||
|
||||
VT_TEST(http_ranged_get_is_206) {
|
||||
TestServer srv;
|
||||
VT_REQUIRE(srv.available());
|
||||
HttpClient client({.workers = 1});
|
||||
|
||||
Recorder rec;
|
||||
Request req;
|
||||
req.url = srv.url("/plain/file/64K");
|
||||
req.range = ByteRange{1000, 1999};
|
||||
auto t = client.start(std::move(req), rec.callbacks());
|
||||
|
||||
auto r = rec.wait();
|
||||
VT_REQUIRE(r.has_value());
|
||||
VT_CHECK_EQ(rec.head.status, 206);
|
||||
auto cr = rec.head.headers.get("Content-Range");
|
||||
VT_REQUIRE(cr.has_value());
|
||||
VT_CHECK_EQ(std::string(*cr), std::string("bytes 1000-1999/65536"));
|
||||
VT_CHECK_EQ(rec.bytes, 1000u);
|
||||
VT_CHECK_EQ(r.value().bytes_received, 1000u);
|
||||
}
|
||||
|
||||
VT_TEST(http_follows_redirect_chain) {
|
||||
TestServer srv;
|
||||
VT_REQUIRE(srv.available());
|
||||
HttpClient client({.workers = 1});
|
||||
|
||||
Recorder rec;
|
||||
Request req;
|
||||
req.url = srv.url("/redirect-chain/file/16K");
|
||||
auto t = client.start(std::move(req), rec.callbacks());
|
||||
|
||||
auto r = rec.wait();
|
||||
VT_REQUIRE(r.has_value());
|
||||
VT_CHECK_EQ(rec.head.status, 200); // on_head fires once, for the final response
|
||||
VT_CHECK_EQ(rec.bytes, 16u * 1024u);
|
||||
VT_CHECK(r.value().effective_url != srv.url("/redirect-chain/file/16K"));
|
||||
VT_CHECK(r.value().effective_url.find("_r=done") != std::string::npos);
|
||||
}
|
||||
|
||||
VT_TEST(http_404_is_not_found_error) {
|
||||
TestServer srv;
|
||||
VT_REQUIRE(srv.available());
|
||||
HttpClient client({.workers = 1});
|
||||
|
||||
Recorder rec;
|
||||
Request req;
|
||||
req.url = srv.url("/plain/nope");
|
||||
auto t = client.start(std::move(req), rec.callbacks());
|
||||
|
||||
auto r = rec.wait();
|
||||
VT_REQUIRE(!r.has_value());
|
||||
VT_CHECK_EQ(r.error().code, Error::not_found);
|
||||
VT_CHECK_EQ(r.error().http_status, 404);
|
||||
}
|
||||
|
||||
VT_TEST(http_connection_refused_is_connect_failed) {
|
||||
HttpClient client({.workers = 1});
|
||||
Recorder rec;
|
||||
Request req;
|
||||
req.url = "http://127.0.0.1:1/nothing"; // nothing listens on :1
|
||||
req.connect_timeout_ms = 2000;
|
||||
auto t = client.start(std::move(req), rec.callbacks());
|
||||
|
||||
auto r = rec.wait();
|
||||
VT_REQUIRE(!r.has_value());
|
||||
VT_CHECK_EQ(r.error().code, Error::connect_failed);
|
||||
}
|
||||
|
||||
VT_TEST(http_head_probe_stops_after_headers) {
|
||||
TestServer srv;
|
||||
VT_REQUIRE(srv.available());
|
||||
HttpClient client({.workers = 1});
|
||||
|
||||
Recorder rec;
|
||||
rec.want_on_head = DataAction::abort; // probe: headers only
|
||||
Request req;
|
||||
req.url = srv.url("/plain/file/1M");
|
||||
req.method = Method::head;
|
||||
auto t = client.start(std::move(req), rec.callbacks());
|
||||
|
||||
auto r = rec.wait();
|
||||
VT_REQUIRE(r.has_value()); // head_complete, NOT canceled
|
||||
VT_CHECK(rec.head_seen);
|
||||
VT_REQUIRE(rec.head.content_length.has_value());
|
||||
VT_CHECK_EQ(*rec.head.content_length, 1024u * 1024u);
|
||||
VT_CHECK_EQ(rec.data_calls.load(), 0);
|
||||
}
|
||||
|
||||
VT_TEST(http_ranged_get_probe_stops_without_downloading_file) {
|
||||
TestServer srv;
|
||||
VT_REQUIRE(srv.available());
|
||||
HttpClient client({.workers = 1});
|
||||
|
||||
Recorder rec;
|
||||
rec.want_on_head = DataAction::abort;
|
||||
Request req;
|
||||
req.url = srv.url("/plain/file/8M");
|
||||
req.range = ByteRange{0, 0}; // classic HEAD-refused fallback
|
||||
auto t = client.start(std::move(req), rec.callbacks());
|
||||
|
||||
auto r = rec.wait();
|
||||
VT_REQUIRE(r.has_value());
|
||||
VT_CHECK(rec.bytes <= 1u); // at most the one probe byte, usually 0
|
||||
}
|
||||
|
||||
VT_TEST(http_cancel_mid_transfer) {
|
||||
TestServer srv;
|
||||
VT_REQUIRE(srv.available());
|
||||
HttpClient client({.workers = 1});
|
||||
|
||||
AtomicAction data_action;
|
||||
std::promise<Result<TransferStats>> done;
|
||||
auto fut = done.get_future();
|
||||
std::atomic<int> calls{0};
|
||||
|
||||
TransferCallbacks cbs{
|
||||
.on_head = [](const ResponseHead &) { return DataAction::proceed; },
|
||||
.on_data =
|
||||
[&](ConstByteSpan) {
|
||||
calls.fetch_add(1);
|
||||
return data_action.load();
|
||||
},
|
||||
.on_finished = [&](Result<TransferStats> r) { done.set_value(std::move(r)); },
|
||||
};
|
||||
|
||||
Request req;
|
||||
req.url = srv.url("/throttled/file/4M"); // ~128 KiB/s => lots of chunks
|
||||
auto t = client.start(std::move(req), std::move(cbs));
|
||||
|
||||
// wait for the transfer to actually start, then cancel
|
||||
for (int i = 0; i < 200 && calls.load() == 0; ++i)
|
||||
std::this_thread::sleep_for(std::chrono::milliseconds(10));
|
||||
VT_REQUIRE(calls.load() > 0);
|
||||
t.cancel();
|
||||
|
||||
VT_REQUIRE(fut.wait_for(std::chrono::seconds(10)) == std::future_status::ready);
|
||||
auto r = fut.get();
|
||||
VT_REQUIRE(!r.has_value());
|
||||
VT_CHECK_EQ(r.error().code, Error::canceled);
|
||||
}
|
||||
|
||||
VT_TEST(http_pause_then_resume_completes) {
|
||||
TestServer srv;
|
||||
VT_REQUIRE(srv.available());
|
||||
HttpClient client({.workers = 1});
|
||||
|
||||
std::atomic<int> calls{0};
|
||||
std::atomic<std::uint64_t> total{0};
|
||||
std::promise<Result<TransferStats>> done;
|
||||
auto fut = done.get_future();
|
||||
|
||||
TransferCallbacks cbs{
|
||||
.on_head = [](const ResponseHead &) { return DataAction::proceed; },
|
||||
.on_data =
|
||||
[&](ConstByteSpan s) {
|
||||
calls.fetch_add(1);
|
||||
total.fetch_add(s.size());
|
||||
return DataAction::proceed;
|
||||
},
|
||||
.on_finished = [&](Result<TransferStats> r) { done.set_value(std::move(r)); },
|
||||
};
|
||||
|
||||
Request req;
|
||||
req.url = srv.url("/throttled/file/1M");
|
||||
auto t = client.start(std::move(req), std::move(cbs));
|
||||
|
||||
for (int i = 0; i < 200 && calls.load() == 0; ++i)
|
||||
std::this_thread::sleep_for(std::chrono::milliseconds(10));
|
||||
VT_REQUIRE(calls.load() > 0);
|
||||
|
||||
t.pause();
|
||||
int calls_at_pause = calls.load();
|
||||
std::this_thread::sleep_for(std::chrono::milliseconds(400));
|
||||
VT_CHECK(calls.load() - calls_at_pause <= 1); // at most one in-flight chunk slips through
|
||||
t.resume();
|
||||
|
||||
VT_REQUIRE(fut.wait_for(std::chrono::seconds(20)) == std::future_status::ready);
|
||||
auto r = fut.get();
|
||||
VT_REQUIRE(r.has_value());
|
||||
VT_CHECK_EQ(total.load(), 1024u * 1024u);
|
||||
}
|
||||
@@ -0,0 +1,163 @@
|
||||
#include "vdm/net/probe.hpp"
|
||||
|
||||
#include <chrono>
|
||||
#include <future>
|
||||
#include <string>
|
||||
|
||||
#include "testserver_fixture.hpp"
|
||||
#include "vtest.hpp"
|
||||
|
||||
using namespace vdm;
|
||||
using namespace vdm::net;
|
||||
using vdm::testing::TestServer;
|
||||
|
||||
namespace {
|
||||
Result<ProbeResult> run_probe(Prober &p, const std::string &url) {
|
||||
std::promise<Result<ProbeResult>> prom;
|
||||
auto fut = prom.get_future();
|
||||
ProbeRequest req;
|
||||
req.url = url;
|
||||
req.overall_timeout_ms = 15000;
|
||||
p.probe(std::move(req), [&](Result<ProbeResult> r) { prom.set_value(std::move(r)); });
|
||||
if (fut.wait_for(std::chrono::seconds(25)) != std::future_status::ready)
|
||||
return Err{Error::timeout, "probe test wait"};
|
||||
return fut.get();
|
||||
}
|
||||
const std::string kEuro = "\xE2\x82\xAC";
|
||||
} // namespace
|
||||
|
||||
VT_TEST(probe_plain_resumable_file) {
|
||||
TestServer srv;
|
||||
VT_REQUIRE(srv.available());
|
||||
Prober p(4);
|
||||
|
||||
auto r = run_probe(p, srv.url("/plain/file/2M"));
|
||||
VT_REQUIRE(r.has_value());
|
||||
const auto &pr = r.value();
|
||||
VT_CHECK_EQ(pr.http_status, 206); // proven via the ranged GET
|
||||
VT_CHECK(pr.accept_ranges);
|
||||
VT_CHECK(pr.resumable); // 206 + ETag validator
|
||||
VT_REQUIRE(pr.total_size.has_value());
|
||||
VT_CHECK_EQ(*pr.total_size, 2u * 1024 * 1024);
|
||||
VT_CHECK(!pr.etag.empty());
|
||||
VT_CHECK_EQ(pr.filename_from_url, std::string("2M"));
|
||||
VT_CHECK(!pr.requires_auth);
|
||||
}
|
||||
|
||||
VT_TEST(probe_no_range_server_is_not_resumable) {
|
||||
TestServer srv;
|
||||
VT_REQUIRE(srv.available());
|
||||
Prober p(4);
|
||||
|
||||
auto r = run_probe(p, srv.url("/no-range/file/1M"));
|
||||
VT_REQUIRE(r.has_value());
|
||||
const auto &pr = r.value();
|
||||
VT_CHECK(!pr.resumable); // no 206 ever
|
||||
VT_CHECK(!pr.accept_ranges);
|
||||
VT_REQUIRE(pr.total_size.has_value());
|
||||
VT_CHECK_EQ(*pr.total_size, 1u * 1024 * 1024);
|
||||
}
|
||||
|
||||
VT_TEST(probe_lying_accept_ranges_still_not_resumable) {
|
||||
TestServer srv;
|
||||
VT_REQUIRE(srv.available());
|
||||
Prober p(4);
|
||||
|
||||
// advertises Accept-Ranges: bytes but never returns 206
|
||||
auto r = run_probe(p, srv.url("/lies-about-accept-ranges/file/1M"));
|
||||
VT_REQUIRE(r.has_value());
|
||||
const auto &pr = r.value();
|
||||
VT_CHECK(pr.accept_ranges); // it advertised
|
||||
VT_CHECK(!pr.resumable); // ...but never proved it -> resume is off
|
||||
}
|
||||
|
||||
VT_TEST(probe_reads_utf8_content_disposition) {
|
||||
TestServer srv;
|
||||
VT_REQUIRE(srv.available());
|
||||
Prober p(4);
|
||||
|
||||
auto r = run_probe(p, srv.url("/utf8-content-disposition/file/8K"));
|
||||
VT_REQUIRE(r.has_value());
|
||||
VT_CHECK_EQ(r.value().filename_from_disposition, kEuro + " rates.pdf");
|
||||
VT_CHECK_EQ(suggest_filename(r.value()), kEuro + " rates.pdf");
|
||||
}
|
||||
|
||||
VT_TEST(probe_reads_legacy_content_disposition) {
|
||||
TestServer srv;
|
||||
VT_REQUIRE(srv.available());
|
||||
Prober p(4);
|
||||
|
||||
auto r = run_probe(p, srv.url("/legacy-content-disposition/file/8K"));
|
||||
VT_REQUIRE(r.has_value());
|
||||
VT_CHECK(!r.value().filename_from_disposition.empty()); // decoded, not the raw =?...?=
|
||||
VT_CHECK(r.value().filename_from_disposition.find("=?") == std::string::npos);
|
||||
}
|
||||
|
||||
VT_TEST(probe_follows_redirect_and_reports_effective_url) {
|
||||
TestServer srv;
|
||||
VT_REQUIRE(srv.available());
|
||||
Prober p(4);
|
||||
|
||||
auto r = run_probe(p, srv.url("/redirect-chain/file/64K"));
|
||||
VT_REQUIRE(r.has_value());
|
||||
const auto &pr = r.value();
|
||||
VT_CHECK(pr.effective_url != srv.url("/redirect-chain/file/64K"));
|
||||
VT_REQUIRE(pr.redirect_chain.size() >= 2);
|
||||
VT_CHECK_EQ(pr.redirect_chain.front(), srv.url("/redirect-chain/file/64K"));
|
||||
VT_CHECK_EQ(pr.redirect_chain.back(), pr.effective_url);
|
||||
}
|
||||
|
||||
VT_TEST(probe_401_is_requires_auth_not_error) {
|
||||
TestServer srv;
|
||||
VT_REQUIRE(srv.available());
|
||||
Prober p(4);
|
||||
|
||||
auto r = run_probe(p, srv.url("/401-basic/file/64K"));
|
||||
VT_REQUIRE(r.has_value()); // success result...
|
||||
VT_CHECK(r.value().requires_auth); // ...flagged for the credential dialog
|
||||
}
|
||||
|
||||
VT_TEST(probe_404_is_an_error) {
|
||||
TestServer srv;
|
||||
VT_REQUIRE(srv.available());
|
||||
Prober p(4);
|
||||
|
||||
auto r = run_probe(p, srv.url("/plain/nope"));
|
||||
VT_REQUIRE(!r.has_value());
|
||||
VT_CHECK_EQ(r.error().code, Error::not_found);
|
||||
}
|
||||
|
||||
VT_TEST(probe_dead_host_is_connect_failed) {
|
||||
Prober p(4);
|
||||
std::promise<Result<ProbeResult>> prom;
|
||||
auto fut = prom.get_future();
|
||||
ProbeRequest req;
|
||||
req.url = "http://127.0.0.1:1/x";
|
||||
req.connect_timeout_ms = 2000;
|
||||
p.probe(std::move(req), [&](Result<ProbeResult> r) { prom.set_value(std::move(r)); });
|
||||
VT_REQUIRE(fut.wait_for(std::chrono::seconds(10)) == std::future_status::ready);
|
||||
auto r = fut.get();
|
||||
VT_REQUIRE(!r.has_value());
|
||||
VT_CHECK_EQ(r.error().code, Error::connect_failed);
|
||||
}
|
||||
|
||||
VT_TEST(probe_pool_serialises_excess_requests) {
|
||||
TestServer srv;
|
||||
VT_REQUIRE(srv.available());
|
||||
Prober p(2); // only 2 at a time
|
||||
|
||||
constexpr int kN = 8;
|
||||
std::vector<std::future<Result<ProbeResult>>> futs;
|
||||
std::vector<std::promise<Result<ProbeResult>>> proms(kN);
|
||||
for (int i = 0; i < kN; ++i) {
|
||||
futs.push_back(proms[i].get_future());
|
||||
ProbeRequest req;
|
||||
req.url = srv.url("/plain/file/4K");
|
||||
p.probe(std::move(req),
|
||||
[pr = &proms[i]](Result<ProbeResult> r) { pr->set_value(std::move(r)); });
|
||||
}
|
||||
for (auto &f : futs) {
|
||||
VT_REQUIRE(f.wait_for(std::chrono::seconds(30)) == std::future_status::ready);
|
||||
VT_CHECK(f.get().has_value());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
// testserver_fixture.hpp — spawn tools/testserver for a test, tear it down after.
|
||||
//
|
||||
// Linux-only (fork/exec/pipe/kill). The path to testserver.py is injected by CMake as
|
||||
// VDM_TESTSERVER_PY; if it's empty or missing the fixture reports unavailable() and the
|
||||
// test should skip.
|
||||
|
||||
#ifndef VDM_TESTS_NET_TESTSERVER_FIXTURE_HPP
|
||||
#define VDM_TESTS_NET_TESTSERVER_FIXTURE_HPP
|
||||
|
||||
#include <fcntl.h>
|
||||
#include <signal.h>
|
||||
#include <sys/wait.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#include <cerrno>
|
||||
#include <chrono>
|
||||
#include <cstdio>
|
||||
#include <cstdlib>
|
||||
#include <cstring>
|
||||
#include <string>
|
||||
#include <thread>
|
||||
|
||||
#ifndef VDM_TESTSERVER_PY
|
||||
#define VDM_TESTSERVER_PY ""
|
||||
#endif
|
||||
|
||||
namespace vdm::testing {
|
||||
|
||||
class TestServer {
|
||||
public:
|
||||
TestServer() {
|
||||
const char *script = VDM_TESTSERVER_PY;
|
||||
if (!script || !*script || ::access(script, R_OK) != 0)
|
||||
return;
|
||||
|
||||
int pipefd[2];
|
||||
if (::pipe(pipefd) != 0)
|
||||
return;
|
||||
|
||||
pid_ = ::fork();
|
||||
if (pid_ < 0) {
|
||||
::close(pipefd[0]);
|
||||
::close(pipefd[1]);
|
||||
return;
|
||||
}
|
||||
if (pid_ == 0) {
|
||||
::dup2(pipefd[1], STDOUT_FILENO);
|
||||
::close(pipefd[0]);
|
||||
::close(pipefd[1]);
|
||||
int devnull = ::open("/dev/null", O_WRONLY);
|
||||
if (devnull >= 0)
|
||||
::dup2(devnull, STDERR_FILENO);
|
||||
::execlp("python3", "python3", script, "--port", "0", "--seed", "9", "--loris-seconds",
|
||||
"1", "--throttle-bps", "131072", static_cast<char *>(nullptr));
|
||||
::_exit(127);
|
||||
}
|
||||
::close(pipefd[1]);
|
||||
|
||||
// Read the port line the server prints to stdout.
|
||||
std::string line;
|
||||
char c = 0;
|
||||
auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(10);
|
||||
while (std::chrono::steady_clock::now() < deadline) {
|
||||
ssize_t r = ::read(pipefd[0], &c, 1);
|
||||
if (r == 1) {
|
||||
if (c == '\n')
|
||||
break;
|
||||
line += c;
|
||||
} else if (r == 0) {
|
||||
break;
|
||||
} else if (errno != EINTR) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
::close(pipefd[0]);
|
||||
if (!line.empty())
|
||||
port_ = std::atoi(line.c_str());
|
||||
|
||||
// Give the listener a moment to accept.
|
||||
std::this_thread::sleep_for(std::chrono::milliseconds(150));
|
||||
}
|
||||
|
||||
~TestServer() {
|
||||
if (pid_ > 0) {
|
||||
::kill(pid_, SIGTERM);
|
||||
int status = 0;
|
||||
for (int i = 0; i < 50; ++i) {
|
||||
if (::waitpid(pid_, &status, WNOHANG) == pid_)
|
||||
return;
|
||||
std::this_thread::sleep_for(std::chrono::milliseconds(20));
|
||||
}
|
||||
::kill(pid_, SIGKILL);
|
||||
::waitpid(pid_, &status, 0);
|
||||
}
|
||||
}
|
||||
|
||||
TestServer(const TestServer &) = delete;
|
||||
TestServer &operator=(const TestServer &) = delete;
|
||||
|
||||
[[nodiscard]] bool available() const { return port_ > 0; }
|
||||
[[nodiscard]] int port() const { return port_; }
|
||||
[[nodiscard]] std::string url(const std::string &path) const {
|
||||
return "http://127.0.0.1:" + std::to_string(port_) + path;
|
||||
}
|
||||
|
||||
private:
|
||||
pid_t pid_ = -1;
|
||||
int port_ = 0;
|
||||
};
|
||||
|
||||
} // namespace vdm::testing
|
||||
|
||||
#endif // VDM_TESTS_NET_TESTSERVER_FIXTURE_HPP
|
||||
@@ -0,0 +1,71 @@
|
||||
#include "vdm/net/url.hpp"
|
||||
|
||||
#include <string>
|
||||
|
||||
#include "vtest.hpp"
|
||||
|
||||
using vdm::net::split_url;
|
||||
using vdm::net::SplitUrl;
|
||||
using vdm::net::url_filename;
|
||||
|
||||
VT_TEST(url_basic_https) {
|
||||
auto u = split_url("https://example.com/path/to/file.iso?x=1#frag");
|
||||
VT_CHECK(u.valid);
|
||||
VT_CHECK_EQ(u.scheme, std::string("https"));
|
||||
VT_CHECK_EQ(u.host, std::string("example.com"));
|
||||
VT_CHECK(!u.port.has_value());
|
||||
VT_CHECK_EQ(u.path, std::string("/path/to/file.iso"));
|
||||
VT_CHECK_EQ(u.query, std::string("x=1"));
|
||||
VT_CHECK_EQ(u.fragment, std::string("frag"));
|
||||
}
|
||||
|
||||
VT_TEST(url_port_userinfo_lowercasing) {
|
||||
auto u = split_url("HTTP://User:[email protected]:8080/a");
|
||||
VT_CHECK_EQ(u.scheme, std::string("http"));
|
||||
VT_CHECK_EQ(u.userinfo, std::string("User:pw"));
|
||||
VT_CHECK_EQ(u.host, std::string("host.example.com"));
|
||||
VT_REQUIRE(u.port.has_value());
|
||||
VT_CHECK_EQ(*u.port, 8080);
|
||||
}
|
||||
|
||||
VT_TEST(url_ipv6_host) {
|
||||
auto u = split_url("http://[2001:db8::1]:9000/file");
|
||||
VT_CHECK_EQ(u.host, std::string("2001:db8::1"));
|
||||
VT_REQUIRE(u.port.has_value());
|
||||
VT_CHECK_EQ(*u.port, 9000);
|
||||
}
|
||||
|
||||
VT_TEST(url_no_path) {
|
||||
auto u = split_url("https://example.com");
|
||||
VT_CHECK(u.valid);
|
||||
VT_CHECK_EQ(u.path, std::string(""));
|
||||
}
|
||||
|
||||
VT_TEST(url_non_http_is_invalid_but_parsed) {
|
||||
auto u = split_url("ftp://host/file");
|
||||
VT_CHECK(!u.valid); // not http/https
|
||||
VT_CHECK_EQ(u.scheme, std::string("ftp"));
|
||||
VT_CHECK(!u.is_http());
|
||||
}
|
||||
|
||||
VT_TEST(url_garbage_does_not_crash) {
|
||||
VT_CHECK(!split_url("").valid);
|
||||
VT_CHECK(!split_url("not a url").valid);
|
||||
VT_CHECK(!split_url("://noscheme/x").valid);
|
||||
VT_CHECK(!split_url("http://").valid);
|
||||
auto a = split_url("http://////");
|
||||
auto b = split_url("https://h/%%%/%");
|
||||
(void)a;
|
||||
(void)b;
|
||||
}
|
||||
|
||||
VT_TEST(url_filename_extraction) {
|
||||
VT_CHECK_EQ(url_filename("https://x.com/a/b/report%20final.pdf"),
|
||||
std::string("report final.pdf"));
|
||||
VT_CHECK_EQ(url_filename("https://x.com/a/b/file.iso?sig=abc"), std::string("file.iso"));
|
||||
VT_CHECK_EQ(url_filename("https://x.com/dir/"), std::string(""));
|
||||
VT_CHECK_EQ(url_filename("https://x.com"), std::string(""));
|
||||
VT_CHECK_EQ(url_filename("https://x.com/%2e%2e"), std::string("")); // ".." rejected
|
||||
VT_CHECK_EQ(url_filename("https://x.com/a%2Fb"),
|
||||
std::string("a_b")); // decoded '/' neutralised
|
||||
}
|
||||
@@ -0,0 +1,145 @@
|
||||
#include "vdm/rate/token_bucket.hpp"
|
||||
|
||||
#include <atomic>
|
||||
#include <chrono>
|
||||
#include <thread>
|
||||
#include <vector>
|
||||
|
||||
#include "vtest.hpp"
|
||||
|
||||
using namespace vdm;
|
||||
using namespace vdm::rate;
|
||||
using namespace std::chrono_literals;
|
||||
|
||||
namespace {
|
||||
TaskId tid(std::uint64_t v) {
|
||||
return TaskId{v};
|
||||
}
|
||||
QueueId qid(std::uint64_t v) {
|
||||
return QueueId{v};
|
||||
}
|
||||
} // namespace
|
||||
|
||||
VT_TEST(tb_unlimited_never_waits) {
|
||||
TokenBucket b(0);
|
||||
for (int i = 0; i < 1000; ++i)
|
||||
VT_CHECK_EQ(b.consume(1'000'000).count(), 0);
|
||||
}
|
||||
|
||||
VT_TEST(tb_burst_then_throttle) {
|
||||
// 1000 B/s, default burst = 1 s = 1000 tokens.
|
||||
TokenBucket b(1000);
|
||||
VT_CHECK_EQ(b.consume(1000).count(), 0); // drains the burst
|
||||
auto w = b.consume(1000); // empty now: must wait ~1 s
|
||||
VT_CHECK(w >= 900ms && w <= 1100ms);
|
||||
}
|
||||
|
||||
VT_TEST(tb_refills_over_time) {
|
||||
TokenBucket b(10'000, /*burst=*/10'000);
|
||||
VT_CHECK_EQ(b.consume(10'000).count(), 0);
|
||||
std::this_thread::sleep_for(120ms); // ~1200 tokens back
|
||||
auto w = b.consume(1000);
|
||||
VT_CHECK_EQ(w.count(), 0); // affordable from the refill
|
||||
auto w2 = b.consume(5000);
|
||||
VT_CHECK(w2.count() > 0); // not that much yet
|
||||
}
|
||||
|
||||
VT_TEST(tb_burst_caps_accumulation) {
|
||||
TokenBucket b(1000, /*burst=*/2000);
|
||||
std::this_thread::sleep_for(100ms); // idle far longer than burst/rate would fill
|
||||
std::this_thread::sleep_for(100ms);
|
||||
VT_CHECK_EQ(b.consume(2000).count(), 0); // at most the 2000 cap accumulated
|
||||
VT_CHECK(b.consume(1).count() > 0); // and no more
|
||||
}
|
||||
|
||||
VT_TEST(tb_set_rate_zero_makes_unlimited) {
|
||||
TokenBucket b(1000);
|
||||
VT_CHECK_EQ(b.consume(1000).count(), 0);
|
||||
VT_CHECK(b.consume(1000).count() > 0);
|
||||
b.set_rate(0);
|
||||
VT_CHECK_EQ(b.consume(1'000'000).count(), 0);
|
||||
}
|
||||
|
||||
// --- the hierarchy --------------------------------------------------------------------
|
||||
|
||||
VT_TEST(rl_all_unlimited_by_default) {
|
||||
RateLimiter rl;
|
||||
rl.attach_task(tid(1), std::nullopt);
|
||||
for (int i = 0; i < 100; ++i)
|
||||
VT_CHECK_EQ(rl.acquire(tid(1), 1'000'000).count(), 0);
|
||||
}
|
||||
|
||||
VT_TEST(rl_tightest_level_binds) {
|
||||
RateLimiter rl;
|
||||
rl.set_global_limit(100'000);
|
||||
rl.set_queue_limit(qid(9), 20'000);
|
||||
rl.set_task_limit(tid(1), 50'000);
|
||||
rl.attach_task(tid(1), qid(9));
|
||||
|
||||
// burst: task 50k, queue 20k, global 100k -> the queue's 20k is the ceiling
|
||||
VT_CHECK_EQ(rl.acquire(tid(1), 20'000).count(), 0);
|
||||
auto w = rl.acquire(tid(1), 5'000);
|
||||
VT_CHECK(w.count() > 0); // queue bucket is dry even though task & global aren't
|
||||
}
|
||||
|
||||
VT_TEST(rl_no_partial_consumption_on_miss) {
|
||||
RateLimiter rl;
|
||||
rl.set_global_limit(1'000'000); // plenty
|
||||
rl.set_task_limit(tid(1), 1000); // 1 s burst
|
||||
rl.attach_task(tid(1), std::nullopt);
|
||||
|
||||
VT_CHECK_EQ(rl.acquire(tid(1), 1000).count(), 0); // drain the task bucket
|
||||
for (int i = 0; i < 5; ++i)
|
||||
VT_CHECK(rl.acquire(tid(1), 1000).count() > 0); // task bucket blocks, repeatedly
|
||||
|
||||
// global must NOT have been charged for any of those blocked attempts: a fresh task
|
||||
// limited only by the global bucket can still spend nearly its whole burst (only the
|
||||
// one *successful* 1000-byte acquire above was charged).
|
||||
rl.attach_task(tid(2), std::nullopt);
|
||||
VT_CHECK_EQ(rl.acquire(tid(2), 990'000).count(), 0);
|
||||
}
|
||||
|
||||
VT_TEST(rl_detach_then_acquire_is_safe_and_unlimited) {
|
||||
RateLimiter rl;
|
||||
rl.set_task_limit(tid(1), 1000);
|
||||
rl.attach_task(tid(1), std::nullopt);
|
||||
VT_CHECK_EQ(rl.acquire(tid(1), 1000).count(), 0);
|
||||
rl.detach_task(tid(1));
|
||||
// unknown task -> no task/queue bucket, only global (unlimited here)
|
||||
VT_CHECK_EQ(rl.acquire(tid(1), 1'000'000).count(), 0);
|
||||
}
|
||||
|
||||
VT_TEST(rl_enforces_aggregate_rate_under_load) {
|
||||
RateLimiter rl;
|
||||
const std::uint64_t rate = 4'000'000; // 4 MB/s global
|
||||
rl.set_global_limit(rate);
|
||||
for (std::uint64_t i = 1; i <= 8; ++i)
|
||||
rl.attach_task(tid(i), std::nullopt);
|
||||
|
||||
std::atomic<std::uint64_t> moved{0};
|
||||
auto t0 = std::chrono::steady_clock::now();
|
||||
std::vector<std::jthread> ws;
|
||||
for (std::uint64_t i = 1; i <= 8; ++i) {
|
||||
ws.emplace_back([&, id = tid(i)] {
|
||||
for (int k = 0; k < 400; ++k) {
|
||||
std::uint64_t chunk = 16 * 1024;
|
||||
for (;;) {
|
||||
auto w = rl.acquire(id, chunk);
|
||||
if (w.count() == 0)
|
||||
break;
|
||||
std::this_thread::sleep_for(
|
||||
std::min<std::chrono::nanoseconds>(w, std::chrono::milliseconds(20)));
|
||||
}
|
||||
moved.fetch_add(chunk);
|
||||
}
|
||||
});
|
||||
}
|
||||
ws.clear(); // join
|
||||
auto secs = std::chrono::duration<double>(std::chrono::steady_clock::now() - t0).count();
|
||||
|
||||
double effective = moved.load() / secs;
|
||||
// Allow one burst's worth of slop plus scheduling noise: effective rate should sit
|
||||
// within ~2x of the configured limit, never wildly above.
|
||||
VT_CHECK(effective <= rate * 2.5);
|
||||
VT_CHECK(moved.load() == 8u * 400u * 16u * 1024u);
|
||||
}
|
||||
@@ -0,0 +1,264 @@
|
||||
#include "vdm/segment/budget.hpp"
|
||||
|
||||
#include <atomic>
|
||||
#include <chrono>
|
||||
#include <mutex>
|
||||
#include <thread>
|
||||
#include <vector>
|
||||
|
||||
#include "vtest.hpp"
|
||||
|
||||
using namespace vdm;
|
||||
using namespace vdm::segment;
|
||||
using EB = SegmentBudget::EngineBudget;
|
||||
|
||||
namespace {
|
||||
|
||||
TaskId tid(std::uint64_t v) {
|
||||
return TaskId{v};
|
||||
}
|
||||
|
||||
// A test task that reacts to slot targets the way stage 8's download_task will: start
|
||||
// workers up to the target, release them when the target drops. Purely bookkeeping.
|
||||
struct FakeTask {
|
||||
SegmentBudget *budget = nullptr;
|
||||
TaskId id{};
|
||||
std::mutex mu;
|
||||
std::uint32_t workers = 0;
|
||||
std::uint32_t target = 0;
|
||||
|
||||
FakeTask() = default;
|
||||
FakeTask(SegmentBudget *b, TaskId i) : budget(b), id(i) {}
|
||||
|
||||
void on_target(std::uint32_t t) {
|
||||
std::lock_guard lk(mu);
|
||||
target = t;
|
||||
while (workers < target) {
|
||||
if (!budget->confirm_slot(id))
|
||||
break;
|
||||
++workers;
|
||||
}
|
||||
// over target -> yield the excess immediately (a real task waits for a boundary)
|
||||
while (workers > target) {
|
||||
budget->release_slot(id);
|
||||
--workers;
|
||||
}
|
||||
}
|
||||
std::uint32_t held() {
|
||||
std::lock_guard lk(mu);
|
||||
return workers;
|
||||
}
|
||||
};
|
||||
|
||||
} // namespace
|
||||
|
||||
VT_TEST(budget_single_task_grows_to_cap) {
|
||||
SegmentBudget b({.max_active_segments = 32});
|
||||
FakeTask t{&b, tid(1)};
|
||||
b.register_task(tid(1), {.host = "h", .per_task_cap = 8, .resumable = true},
|
||||
[&](std::uint32_t n) { t.on_target(n); });
|
||||
b.set_want(tid(1), 8);
|
||||
VT_CHECK_EQ(t.held(), 8u);
|
||||
VT_CHECK_EQ(b.segments_active(tid(1)), 8u);
|
||||
VT_CHECK_EQ(b.budget().active, 8u);
|
||||
VT_CHECK_EQ(b.budget().tasks_starved, 0u);
|
||||
}
|
||||
|
||||
VT_TEST(budget_min_one_before_seconds) {
|
||||
// Budget of 3, two tasks each wanting 8. min-1 first: each gets 1, then the higher-
|
||||
// priority one grows to 2.
|
||||
SegmentBudget b({.max_active_segments = 3});
|
||||
FakeTask a{&b, tid(1)}, c{&b, tid(2)};
|
||||
b.register_task(tid(1), {.host = "h1", .per_task_cap = 8, .resumable = true},
|
||||
[&](std::uint32_t n) { a.on_target(n); });
|
||||
b.register_task(tid(2), {.host = "h2", .per_task_cap = 8, .resumable = true},
|
||||
[&](std::uint32_t n) { c.on_target(n); });
|
||||
std::vector<TaskId> order = {tid(1), tid(2)};
|
||||
b.set_task_order(order);
|
||||
b.set_want(tid(1), 8);
|
||||
b.set_want(tid(2), 8);
|
||||
|
||||
VT_CHECK(a.held() >= 1); // guarantee
|
||||
VT_CHECK(c.held() >= 1); // guarantee — the load-bearing property
|
||||
VT_CHECK_EQ(a.held() + c.held(), 3u);
|
||||
VT_CHECK_EQ(a.held(), 2u); // higher priority took the growth slot
|
||||
}
|
||||
|
||||
VT_TEST(budget_new_high_priority_task_gets_min_one_via_yield) {
|
||||
SegmentBudget b({.max_active_segments = 4});
|
||||
FakeTask a{&b, tid(1)};
|
||||
b.register_task(tid(1), {.host = "h", .per_task_cap = 8, .resumable = true},
|
||||
[&](std::uint32_t n) { a.on_target(n); });
|
||||
b.set_task_order(std::vector<TaskId>{tid(1)});
|
||||
b.set_want(tid(1), 8);
|
||||
VT_CHECK_EQ(a.held(), 4u); // hogging the whole budget
|
||||
|
||||
// a second, higher-priority task arrives
|
||||
FakeTask c{&b, tid(2)};
|
||||
b.register_task(tid(2), {.host = "h2", .per_task_cap = 8, .resumable = true},
|
||||
[&](std::uint32_t n) { c.on_target(n); });
|
||||
b.set_task_order(std::vector<TaskId>{tid(2), tid(1)});
|
||||
b.set_want(tid(2), 8);
|
||||
|
||||
// a yields so c gets at least its guaranteed slot; the surplus is shared round-robin.
|
||||
VT_CHECK(c.held() >= 1); // min-1 — the load-bearing guarantee
|
||||
VT_CHECK(a.held() >= 1); // a keeps its own min-1
|
||||
VT_CHECK(a.held() < 4); // a really did yield at least one
|
||||
VT_CHECK_EQ(a.held() + c.held(), 4u);
|
||||
VT_CHECK_EQ(b.budget().active, 4u);
|
||||
VT_CHECK_EQ(b.budget().tasks_starved, 0u);
|
||||
}
|
||||
|
||||
VT_TEST(budget_host_cap_clamps_effective_target) {
|
||||
SegmentBudget b({.max_active_segments = 32});
|
||||
FakeTask t{&b, tid(1)};
|
||||
b.set_host_segment_cap("slowcdn", 4);
|
||||
b.register_task(tid(1), {.host = "slowcdn", .per_task_cap = 16, .resumable = true},
|
||||
[&](std::uint32_t n) { t.on_target(n); });
|
||||
b.set_want(tid(1), 16);
|
||||
VT_CHECK_EQ(t.held(), 4u); // clamped by the host cap, not per_task_cap
|
||||
|
||||
b.set_host_segment_cap("slowcdn", 0); // clear
|
||||
VT_CHECK_EQ(t.held(), 16u);
|
||||
}
|
||||
|
||||
VT_TEST(budget_non_resumable_task_capped_at_one) {
|
||||
SegmentBudget b({.max_active_segments = 32});
|
||||
FakeTask t{&b, tid(1)};
|
||||
b.register_task(tid(1), {.host = "h", .per_task_cap = 8, .resumable = false},
|
||||
[&](std::uint32_t n) { t.on_target(n); });
|
||||
b.set_want(tid(1), 8);
|
||||
VT_CHECK_EQ(t.held(), 1u);
|
||||
}
|
||||
|
||||
VT_TEST(budget_live_lower_sheds_via_yield_lowest_priority_first) {
|
||||
SegmentBudget b({.max_active_segments = 24});
|
||||
FakeTask a{&b, tid(1)}, c{&b, tid(2)}, d{&b, tid(3)};
|
||||
for (auto *ft : {&a, &c, &d})
|
||||
b.register_task(ft->id, {.host = "h", .per_task_cap = 8, .resumable = true},
|
||||
[ft](std::uint32_t n) { ft->on_target(n); });
|
||||
b.set_task_order(std::vector<TaskId>{tid(1), tid(2), tid(3)});
|
||||
for (auto id : {tid(1), tid(2), tid(3)})
|
||||
b.set_want(id, 8);
|
||||
VT_CHECK_EQ(a.held() + c.held() + d.held(), 24u); // 8 + 8 + 8
|
||||
|
||||
b.set_max_active_segments(10); // live cut
|
||||
VT_CHECK_EQ(a.held() + c.held() + d.held(), 10u);
|
||||
VT_CHECK(a.held() >= c.held() && c.held() >= d.held()); // priority order preserved
|
||||
VT_CHECK(a.held() >= 1 && c.held() >= 1 && d.held() >= 1); // min-1 still honoured
|
||||
}
|
||||
|
||||
VT_TEST(budget_live_lower_below_task_count_starves_the_tail) {
|
||||
SegmentBudget b({.max_active_segments = 6});
|
||||
std::vector<FakeTask> ts(4);
|
||||
for (std::uint32_t i = 0; i < 4; ++i) {
|
||||
ts[i].budget = &b;
|
||||
ts[i].id = tid(i + 1);
|
||||
}
|
||||
for (auto &ft : ts)
|
||||
b.register_task(ft.id, {.host = "h", .per_task_cap = 4, .resumable = true},
|
||||
[&ft](std::uint32_t n) { ft.on_target(n); });
|
||||
b.set_task_order(std::vector<TaskId>{tid(1), tid(2), tid(3), tid(4)});
|
||||
for (auto &ft : ts)
|
||||
b.set_want(ft.id, 4);
|
||||
VT_CHECK_EQ(b.budget().tasks_starved, 0u);
|
||||
|
||||
b.set_max_active_segments(3); // below the running-task count
|
||||
VT_CHECK_EQ(ts[0].held(), 1u);
|
||||
VT_CHECK_EQ(ts[3].held(), 0u); // lowest priority shed to zero
|
||||
VT_CHECK_EQ(b.budget().tasks_starved, 1u);
|
||||
VT_REQUIRE(b.starved_tasks().size() == 1);
|
||||
VT_CHECK_EQ(b.starved_tasks()[0], tid(4));
|
||||
VT_CHECK(b.starved_since(tid(4)).has_value());
|
||||
VT_CHECK(!b.starved_since(tid(1)).has_value());
|
||||
}
|
||||
|
||||
VT_TEST(budget_deregister_frees_slots_to_starved) {
|
||||
SegmentBudget b({.max_active_segments = 4});
|
||||
FakeTask a{&b, tid(1)}, c{&b, tid(2)};
|
||||
b.register_task(tid(1), {.host = "h", .per_task_cap = 8, .resumable = true},
|
||||
[&](std::uint32_t n) { a.on_target(n); });
|
||||
b.set_task_order(std::vector<TaskId>{tid(1)});
|
||||
b.set_want(tid(1), 8);
|
||||
VT_CHECK_EQ(a.held(), 4u);
|
||||
|
||||
b.register_task(tid(2), {.host = "h", .per_task_cap = 8, .resumable = true},
|
||||
[&](std::uint32_t n) { c.on_target(n); });
|
||||
b.set_task_order(std::vector<TaskId>{tid(1), tid(2)});
|
||||
b.set_want(tid(2), 8);
|
||||
VT_CHECK(c.held() >= 1); // min-1 from a's yield
|
||||
|
||||
b.deregister_task(tid(1));
|
||||
VT_CHECK_EQ(c.held(), 4u); // c grows into the whole freed budget
|
||||
VT_CHECK_EQ(b.budget().active, 4u);
|
||||
}
|
||||
|
||||
VT_TEST(budget_on_changed_fires_on_starved_edge) {
|
||||
SegmentBudget b({.max_active_segments = 1, .notify_period = std::chrono::milliseconds{40}});
|
||||
std::mutex m;
|
||||
std::vector<EB> seen;
|
||||
b.on_budget_changed([&](EB e) {
|
||||
std::lock_guard lk(m);
|
||||
seen.push_back(e);
|
||||
});
|
||||
|
||||
FakeTask a{&b, tid(1)}, c{&b, tid(2)};
|
||||
b.register_task(tid(1), {.host = "h", .per_task_cap = 4, .resumable = true},
|
||||
[&](std::uint32_t n) { a.on_target(n); });
|
||||
b.register_task(tid(2), {.host = "h", .per_task_cap = 4, .resumable = true},
|
||||
[&](std::uint32_t n) { c.on_target(n); });
|
||||
b.set_task_order(std::vector<TaskId>{tid(1), tid(2)});
|
||||
b.set_want(tid(1), 4);
|
||||
b.set_want(tid(2), 4); // budget is 1 -> tid(2) is starved: 0 -> nonzero edge
|
||||
|
||||
// the edge fire is synchronous on the triggering call
|
||||
bool saw_starved = false;
|
||||
{
|
||||
std::lock_guard lk(m);
|
||||
for (auto &e : seen)
|
||||
if (e.tasks_starved > 0)
|
||||
saw_starved = true;
|
||||
}
|
||||
VT_CHECK(saw_starved);
|
||||
|
||||
b.deregister_task(tid(1)); // frees the slot -> tid(2) no longer starved: edge back
|
||||
std::this_thread::sleep_for(std::chrono::milliseconds(120));
|
||||
bool saw_unstarved_after = false;
|
||||
{
|
||||
std::lock_guard lk(m);
|
||||
VT_CHECK(!seen.empty());
|
||||
saw_unstarved_after = seen.back().tasks_starved == 0;
|
||||
}
|
||||
VT_CHECK(saw_unstarved_after);
|
||||
}
|
||||
|
||||
VT_TEST(budget_concurrent_confirm_release_stays_consistent) {
|
||||
SegmentBudget b({.max_active_segments = 16});
|
||||
constexpr int kTasks = 6;
|
||||
std::vector<std::unique_ptr<FakeTask>> ts;
|
||||
for (int i = 0; i < kTasks; ++i) {
|
||||
ts.push_back(std::make_unique<FakeTask>());
|
||||
ts.back()->budget = &b;
|
||||
ts.back()->id = tid(i + 1);
|
||||
FakeTask *ft = ts.back().get();
|
||||
b.register_task(ft->id, {.host = "h", .per_task_cap = 6, .resumable = true},
|
||||
[ft](std::uint32_t n) { ft->on_target(n); });
|
||||
}
|
||||
std::vector<std::jthread> drivers;
|
||||
for (int i = 0; i < kTasks; ++i) {
|
||||
drivers.emplace_back([&, id = tid(i + 1)] {
|
||||
for (int r = 0; r < 4000; ++r)
|
||||
b.set_want(id, (r % 7));
|
||||
});
|
||||
}
|
||||
drivers.clear(); // join
|
||||
for (auto &ft : ts)
|
||||
b.set_want(ft->id, 0);
|
||||
|
||||
// With everyone wanting nothing, the budget must be fully released.
|
||||
VT_CHECK_EQ(b.budget().active, 0u);
|
||||
std::uint32_t sum = 0;
|
||||
for (auto &ft : ts)
|
||||
sum += b.segments_active(ft->id);
|
||||
VT_CHECK_EQ(sum, 0u);
|
||||
}
|
||||
@@ -0,0 +1,257 @@
|
||||
#include "vdm/segment/segmenter.hpp"
|
||||
|
||||
#include <atomic>
|
||||
#include <cstdint>
|
||||
#include <thread>
|
||||
#include <vector>
|
||||
|
||||
#include "vtest.hpp"
|
||||
|
||||
using namespace vdm::segment;
|
||||
|
||||
namespace {
|
||||
constexpr std::uint64_t MiB = 1u << 20;
|
||||
|
||||
// Assign `n` slots (bounded by what the segmenter hands out) and return the indices.
|
||||
std::vector<std::uint32_t> fill(Segmenter &s, int n) {
|
||||
std::vector<std::uint32_t> idx;
|
||||
for (int i = 0; i < n; ++i) {
|
||||
auto a = s.assign_slot();
|
||||
if (!a)
|
||||
break;
|
||||
idx.push_back(*a);
|
||||
}
|
||||
return idx;
|
||||
}
|
||||
|
||||
// Do the segments (by their [start,end] at this instant) tile [0,total) with no overlap?
|
||||
bool tiles_exactly(const Segmenter &s) {
|
||||
auto snap = s.snapshot();
|
||||
std::vector<std::pair<std::uint64_t, std::uint64_t>> r;
|
||||
for (auto &v : snap)
|
||||
if (v.state != SegState::failed)
|
||||
r.emplace_back(v.start, v.end);
|
||||
std::sort(r.begin(), r.end());
|
||||
std::uint64_t cursor = 0;
|
||||
for (auto [a, b] : r) {
|
||||
if (a != cursor)
|
||||
return false;
|
||||
cursor = b + 1;
|
||||
}
|
||||
return cursor == s.total_size();
|
||||
}
|
||||
} // namespace
|
||||
|
||||
VT_TEST(seg_target_count_clamps) {
|
||||
VT_CHECK_EQ(Segmenter(100 * MiB, 8, true).target_segment_count(), 8u);
|
||||
VT_CHECK_EQ(Segmenter(100 * MiB, 64, true).target_segment_count(), 32u); // max 32
|
||||
VT_CHECK_EQ(Segmenter(100 * MiB, 0, true).target_segment_count(), 8u); // default
|
||||
VT_CHECK_EQ(Segmenter(3 * MiB + 1, 8, true).target_segment_count(), 3u); // total/min
|
||||
VT_CHECK_EQ(Segmenter(100 * MiB, 8, false).target_segment_count(), 1u); // non-resumable
|
||||
VT_CHECK_EQ(Segmenter(0, 8, true).target_segment_count(), 1u); // chunked
|
||||
}
|
||||
|
||||
VT_TEST(seg_non_resumable_is_one_segment) {
|
||||
Segmenter s(50 * MiB, 8, false);
|
||||
auto idx = fill(s, 8);
|
||||
VT_REQUIRE(idx.size() == 1);
|
||||
auto snap = s.snapshot();
|
||||
VT_REQUIRE(snap.size() == 1);
|
||||
VT_CHECK_EQ(snap[0].start, 0u);
|
||||
VT_CHECK_EQ(snap[0].end, 50u * MiB - 1);
|
||||
}
|
||||
|
||||
VT_TEST(seg_initial_split_covers_range) {
|
||||
Segmenter s(80 * MiB, 8, true);
|
||||
auto idx = fill(s, 8);
|
||||
VT_CHECK_EQ(idx.size(), 8u);
|
||||
VT_CHECK(tiles_exactly(s));
|
||||
// no segment below the 1 MiB floor
|
||||
for (auto &v : s.snapshot())
|
||||
VT_CHECK(v.length() >= MiB);
|
||||
}
|
||||
|
||||
VT_TEST(seg_split_stops_at_min_floor) {
|
||||
// 5 MiB, floor 1 MiB, ask for 8: only ~5 splits possible (each half >= 1 MiB needs
|
||||
// the parent >= 2 MiB), so we get fewer than 8.
|
||||
Segmenter s(5 * MiB, 8, true);
|
||||
auto idx = fill(s, 8);
|
||||
VT_CHECK(idx.size() >= 1 && idx.size() <= 5);
|
||||
VT_CHECK(tiles_exactly(s));
|
||||
}
|
||||
|
||||
VT_TEST(seg_steal_takes_second_half_of_largest_remaining) {
|
||||
Segmenter s(80 * MiB, 4, true);
|
||||
auto idx = fill(s, 4);
|
||||
VT_REQUIRE(idx.size() == 4);
|
||||
|
||||
// Spread progress unevenly. Index != file position after splits, so identify the
|
||||
// largest-remaining segment by scanning the snapshot, not by index.
|
||||
s.advance(idx[1], 3 * MiB);
|
||||
s.advance(idx[2], 7 * MiB);
|
||||
s.advance(idx[3], 12 * MiB);
|
||||
|
||||
SegmentView pre_victim{};
|
||||
std::uint64_t worst = 0;
|
||||
for (auto &v : s.snapshot())
|
||||
if (v.index != idx[0] && v.remaining() > worst) {
|
||||
worst = v.remaining();
|
||||
pre_victim = v;
|
||||
}
|
||||
|
||||
auto cont = s.on_complete(idx[0], /*may_steal=*/true);
|
||||
VT_REQUIRE(cont.has_value());
|
||||
|
||||
SegmentView victim{}, fresh{};
|
||||
for (auto &v : s.snapshot()) {
|
||||
if (v.index == pre_victim.index)
|
||||
victim = v;
|
||||
if (v.index == *cont)
|
||||
fresh = v;
|
||||
}
|
||||
VT_CHECK_EQ(fresh.end, pre_victim.end); // fresh takes the tail of the victim's range
|
||||
VT_CHECK_EQ(victim.end + 1, fresh.start); // contiguous, no gap / no overlap
|
||||
VT_CHECK(victim.end < pre_victim.end); // the victim really did shrink
|
||||
VT_CHECK(fresh.length() >= MiB);
|
||||
VT_CHECK(victim.remaining() >= MiB);
|
||||
// fresh got roughly the back half of what was remaining
|
||||
VT_CHECK(fresh.length() >= worst / 2 - MiB && fresh.length() <= worst / 2 + MiB);
|
||||
VT_CHECK(tiles_exactly(s));
|
||||
}
|
||||
|
||||
VT_TEST(seg_complete_without_steal_releases) {
|
||||
Segmenter s(4 * MiB, 2, true);
|
||||
auto idx = fill(s, 2); // 2 x 2 MiB
|
||||
VT_REQUIRE(idx.size() == 2);
|
||||
s.advance(idx[1], 2 * MiB);
|
||||
s.set_segment_state(idx[1], SegState::complete);
|
||||
// idx[0] done, nothing left worth >= 1 MiB to steal -> release
|
||||
s.advance(idx[0], 2 * MiB);
|
||||
auto cont = s.on_complete(idx[0], true);
|
||||
VT_CHECK(!cont.has_value());
|
||||
}
|
||||
|
||||
VT_TEST(seg_yield_returns_nullopt) {
|
||||
Segmenter s(80 * MiB, 4, true);
|
||||
auto idx = fill(s, 4);
|
||||
s.advance(idx[0], 20 * MiB);
|
||||
auto cont = s.on_complete(idx[0], /*may_steal=*/false); // yielding
|
||||
VT_CHECK(!cont.has_value());
|
||||
}
|
||||
|
||||
VT_TEST(seg_third_connection_failure_with_mirror_requeues) {
|
||||
Segmenter s(40 * MiB, 2, true);
|
||||
auto idx = fill(s, 2);
|
||||
s.advance(idx[0], 4 * MiB);
|
||||
|
||||
VT_CHECK(s.on_failed(idx[0], /*conn=*/true, /*mirror=*/true) == FailAction::retry);
|
||||
VT_CHECK(s.on_failed(idx[0], true, true) == FailAction::retry);
|
||||
VT_CHECK(s.on_failed(idx[0], true, true) == FailAction::requeue); // 3rd
|
||||
|
||||
VT_CHECK_EQ(s.segment_state(idx[0]), SegState::failed);
|
||||
// the orphaned tail is now assignable again
|
||||
auto again = s.assign_slot();
|
||||
VT_REQUIRE(again.has_value());
|
||||
auto snap = s.snapshot();
|
||||
SegmentView reborn{};
|
||||
for (auto &v : snap)
|
||||
if (v.index == *again)
|
||||
reborn = v;
|
||||
VT_CHECK_EQ(reborn.start, 4u * MiB); // resumes where the failed one stopped
|
||||
VT_CHECK_EQ(reborn.end, 20u * MiB - 1); // its half of the file
|
||||
}
|
||||
|
||||
VT_TEST(seg_failure_without_mirror_always_retries) {
|
||||
Segmenter s(40 * MiB, 2, true);
|
||||
auto idx = fill(s, 2);
|
||||
for (int i = 0; i < 6; ++i)
|
||||
VT_CHECK(s.on_failed(idx[0], true, /*mirror=*/false) == FailAction::retry);
|
||||
// a non-connection error also retries regardless of count
|
||||
VT_CHECK(s.on_failed(idx[1], /*conn=*/false, /*mirror=*/true) == FailAction::retry);
|
||||
}
|
||||
|
||||
VT_TEST(seg_note_connected_resets_failure_count) {
|
||||
Segmenter s(40 * MiB, 2, true);
|
||||
auto idx = fill(s, 2);
|
||||
s.on_failed(idx[0], true, true);
|
||||
s.on_failed(idx[0], true, true);
|
||||
s.note_connected(idx[0]);
|
||||
VT_CHECK(s.on_failed(idx[0], true, true) == FailAction::retry); // count restarted
|
||||
}
|
||||
|
||||
VT_TEST(seg_resume_from_meta_table) {
|
||||
std::vector<ResumedRange> table = {
|
||||
{0, 9 * MiB - 1, 9 * MiB}, // fully done
|
||||
{9 * MiB, 19 * MiB - 1, 3 * MiB}, // partial
|
||||
{19 * MiB, 40 * MiB - 1, 0}, // untouched
|
||||
};
|
||||
Segmenter s(40 * MiB, 8, table, true);
|
||||
auto snap = s.snapshot();
|
||||
VT_REQUIRE(snap.size() == 3);
|
||||
VT_CHECK_EQ(snap[0].state, SegState::complete);
|
||||
VT_CHECK_EQ(snap[1].completed, 3u * MiB);
|
||||
VT_CHECK_EQ(s.downloaded(), 12u * MiB);
|
||||
VT_CHECK(tiles_exactly(s));
|
||||
|
||||
// assign hands out the two incomplete ranges before splitting
|
||||
auto a = s.assign_slot();
|
||||
auto b = s.assign_slot();
|
||||
VT_REQUIRE(a && b);
|
||||
}
|
||||
|
||||
VT_TEST(seg_resume_from_bad_table_falls_back) {
|
||||
std::vector<ResumedRange> gappy = {{0, 4 * MiB - 1, 0}, {8 * MiB, 40 * MiB - 1, 0}};
|
||||
Segmenter s(40 * MiB, 8, gappy, true);
|
||||
VT_CHECK(s.snapshot().empty()); // lazy fresh layout
|
||||
auto idx = fill(s, 8);
|
||||
VT_CHECK(idx.size() >= 1);
|
||||
VT_CHECK(tiles_exactly(s));
|
||||
}
|
||||
|
||||
VT_TEST(seg_all_complete_and_downloaded) {
|
||||
Segmenter s(8 * MiB, 4, true);
|
||||
auto idx = fill(s, 4);
|
||||
VT_CHECK(!s.all_complete());
|
||||
for (auto i : idx) {
|
||||
std::uint64_t len = s.segment_end(i) - s.segment_start(i) + 1;
|
||||
s.advance(i, len);
|
||||
s.set_segment_state(i, SegState::complete);
|
||||
}
|
||||
VT_CHECK(s.all_complete());
|
||||
VT_CHECK_EQ(s.downloaded(), 8u * MiB);
|
||||
}
|
||||
|
||||
// --- the steal path under the sanitizers -----------------------------------------------
|
||||
|
||||
VT_TEST(seg_concurrent_steal_and_advance) {
|
||||
constexpr std::uint64_t total = 64 * MiB;
|
||||
Segmenter s(total, 8, true);
|
||||
auto idx = fill(s, 8);
|
||||
VT_REQUIRE(idx.size() == 8);
|
||||
|
||||
std::vector<std::jthread> workers;
|
||||
for (std::uint32_t w = 0; w < 8; ++w) {
|
||||
workers.emplace_back([&s, seg = idx[w]]() mutable {
|
||||
std::uint32_t cur = seg;
|
||||
for (int guard = 0; guard < 200000; ++guard) {
|
||||
const std::uint64_t start = s.segment_start(cur);
|
||||
const std::uint64_t end = s.segment_end(cur); // may shrink under a steal
|
||||
const std::uint64_t len = end - start + 1;
|
||||
const std::uint64_t done = s.segment_completed(cur);
|
||||
if (done >= len) {
|
||||
auto nxt = s.on_complete(cur, /*may_steal=*/true);
|
||||
if (!nxt)
|
||||
return; // nothing left to steal — this worker is finished
|
||||
cur = *nxt;
|
||||
continue;
|
||||
}
|
||||
s.advance(cur, std::min(done + 128 * 1024, len));
|
||||
}
|
||||
});
|
||||
}
|
||||
workers.clear(); // join
|
||||
|
||||
VT_CHECK(s.all_complete());
|
||||
VT_CHECK_EQ(s.downloaded(), total);
|
||||
VT_CHECK(tiles_exactly(s));
|
||||
}
|
||||
@@ -81,9 +81,17 @@ std::string show(const T &v) {
|
||||
}
|
||||
}
|
||||
|
||||
inline int run_all() {
|
||||
inline int run_all(const std::vector<std::string> &filters = {}) {
|
||||
int failed_cases = 0;
|
||||
for (const auto &c : registry()) {
|
||||
if (!filters.empty()) {
|
||||
bool match = false;
|
||||
for (const auto &f : filters)
|
||||
if (std::string_view(c.name).find(f) != std::string_view::npos)
|
||||
match = true;
|
||||
if (!match)
|
||||
continue;
|
||||
}
|
||||
int before = stats().failures;
|
||||
stats().current_fatal = false;
|
||||
std::fprintf(stderr, "[ RUN ] %s\n", c.name);
|
||||
@@ -129,11 +137,14 @@ inline int run_all() {
|
||||
::vt::report(__FILE__, __LINE__, #COND, {}, /*fatal=*/true); \
|
||||
} while (0)
|
||||
|
||||
// NOTE: operands are copied (auto, not auto&&). A test assertion must never outlive a
|
||||
// temporary the expression returned a reference into — the copy makes that safe. All
|
||||
// compared types here are cheap to copy.
|
||||
#define VT_CHECK_EQ(A, B) \
|
||||
do { \
|
||||
::vt::stats().checks++; \
|
||||
auto &&_a = (A); \
|
||||
auto &&_b = (B); \
|
||||
auto _a = (A); \
|
||||
auto _b = (B); \
|
||||
if (!(_a == _b)) \
|
||||
::vt::report(__FILE__, __LINE__, #A " == " #B, \
|
||||
::vt::show(_a) + " vs " + ::vt::show(_b), false); \
|
||||
@@ -142,8 +153,8 @@ inline int run_all() {
|
||||
#define VT_CHECK_NE(A, B) \
|
||||
do { \
|
||||
::vt::stats().checks++; \
|
||||
auto &&_a = (A); \
|
||||
auto &&_b = (B); \
|
||||
auto _a = (A); \
|
||||
auto _b = (B); \
|
||||
if (!(_a != _b)) \
|
||||
::vt::report(__FILE__, __LINE__, #A " != " #B, \
|
||||
::vt::show(_a) + " vs " + ::vt::show(_b), false); \
|
||||
|
||||
@@ -1,6 +1,29 @@
|
||||
// vtest_main.cpp — shared entry point for every CORE test binary.
|
||||
#include <cstdlib>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "vtest.hpp"
|
||||
|
||||
int main() {
|
||||
return ::vt::run_all();
|
||||
// Optional filters: each argv argument (or a comma-separated entry in $VT_ONLY) is a
|
||||
// substring; a test runs only if its name contains one of them. No filters => run all.
|
||||
int main(int argc, char **argv) {
|
||||
std::vector<std::string> filters;
|
||||
for (int i = 1; i < argc; ++i)
|
||||
filters.emplace_back(argv[i]);
|
||||
if (const char *env = std::getenv("VT_ONLY")) {
|
||||
std::string cur;
|
||||
for (const char *p = env;; ++p) {
|
||||
if (*p == ',' || *p == '\0') {
|
||||
if (!cur.empty())
|
||||
filters.push_back(cur);
|
||||
cur.clear();
|
||||
if (*p == '\0')
|
||||
break;
|
||||
} else {
|
||||
cur.push_back(*p);
|
||||
}
|
||||
}
|
||||
}
|
||||
return ::vt::run_all(filters);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
// The engine API sketch must compile and its value types must behave. Engine /
|
||||
// DownloadHandle bodies land in stage 8; this only exercises the data shapes DAEMON
|
||||
// builds against.
|
||||
|
||||
#include "vdm/engine.hpp"
|
||||
#include "vdm/task/download.hpp"
|
||||
|
||||
#include <type_traits>
|
||||
|
||||
#include "vtest.hpp"
|
||||
|
||||
using namespace vdm;
|
||||
using namespace vdm::task;
|
||||
|
||||
VT_TEST(api_download_spec_defaults) {
|
||||
DownloadSpec s;
|
||||
s.url = "https://example.com/big.iso";
|
||||
s.save_path = "/home/u/Downloads/big.iso";
|
||||
VT_CHECK(s.mirrors.empty());
|
||||
VT_CHECK(!s.segments.has_value());
|
||||
VT_CHECK(!s.buffer_bytes.has_value());
|
||||
VT_CHECK(!s.checksum.has_value());
|
||||
VT_CHECK(!s.probe_hint.has_value());
|
||||
VT_CHECK(s.allow_resume);
|
||||
VT_CHECK(s.proxy.kind == net::ProxyKind::none);
|
||||
VT_CHECK(s.auth.scheme == net::AuthScheme::none);
|
||||
}
|
||||
|
||||
VT_TEST(api_state_helpers) {
|
||||
VT_CHECK(is_terminal(EngineState::complete));
|
||||
VT_CHECK(is_terminal(EngineState::failed));
|
||||
VT_CHECK(is_terminal(EngineState::cancelled));
|
||||
VT_CHECK(!is_terminal(EngineState::paused));
|
||||
VT_CHECK(!is_terminal(EngineState::downloading));
|
||||
}
|
||||
|
||||
VT_TEST(api_callbacks_are_all_optional) {
|
||||
DownloadCallbacks cb; // every std::function default-constructs empty
|
||||
VT_CHECK(!cb.on_progress);
|
||||
VT_CHECK(!cb.on_state);
|
||||
VT_CHECK(!cb.on_auth_required);
|
||||
VT_CHECK(!cb.on_decision_needed);
|
||||
VT_CHECK(!cb.on_finished);
|
||||
|
||||
cb.on_state = [](EngineState, EngineState, const std::optional<vdm::ErrorInfo> &) {};
|
||||
cb.on_finished = [](Result<DownloadOutcome>) {};
|
||||
VT_CHECK(cb.on_state && cb.on_finished);
|
||||
}
|
||||
|
||||
VT_TEST(api_value_types_roundtrip) {
|
||||
Progress p;
|
||||
p.downloaded = 1234;
|
||||
p.total = 5000;
|
||||
p.effective_segments = 4;
|
||||
SegmentProgress sp;
|
||||
sp.index = 0;
|
||||
sp.end = 1249;
|
||||
sp.completed = 1234;
|
||||
p.segments.push_back(sp);
|
||||
VT_CHECK_EQ(p.segments.size(), 1u);
|
||||
VT_CHECK_EQ(p.segments[0].end, 1249u);
|
||||
|
||||
DownloadOutcome o;
|
||||
o.final_path = "/x";
|
||||
o.bytes = 5000;
|
||||
VT_CHECK_EQ(o.bytes, 5000u);
|
||||
|
||||
AuthChallenge a;
|
||||
a.host = "h";
|
||||
a.scheme = AuthChallenge::Scheme::digest;
|
||||
VT_CHECK(a.scheme == AuthChallenge::Scheme::digest);
|
||||
|
||||
DecisionRequest d;
|
||||
d.kind = DecisionRequest::Kind::server_file_changed;
|
||||
d.detail = "changed";
|
||||
VT_CHECK(d.kind == DecisionRequest::Kind::server_file_changed);
|
||||
}
|
||||
|
||||
VT_TEST(api_handle_and_engine_are_move_only_shaped) {
|
||||
static_assert(!std::is_copy_constructible_v<Engine>, "Engine is non-copyable");
|
||||
static_assert(std::is_copy_constructible_v<DownloadHandle>, "handle is a shared handle");
|
||||
DownloadHandle h; // default handle is invalid until Engine::start() fills it
|
||||
VT_CHECK(!h.valid());
|
||||
}
|
||||
@@ -0,0 +1,328 @@
|
||||
// End-to-end: a real Engine against tools/testserver, covering the CORE M1 DoD paths.
|
||||
|
||||
#include "vdm/engine.hpp"
|
||||
|
||||
#include <fcntl.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#include <atomic>
|
||||
#include <chrono>
|
||||
#include <cstdlib>
|
||||
#include <future>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "task/digest.hpp"
|
||||
#include "testserver_fixture.hpp"
|
||||
#include "vtest.hpp"
|
||||
|
||||
using namespace vdm;
|
||||
using namespace vdm::task;
|
||||
using vdm::testing::TestServer;
|
||||
using namespace std::chrono_literals;
|
||||
|
||||
namespace {
|
||||
|
||||
struct TmpDir {
|
||||
std::string path;
|
||||
TmpDir() {
|
||||
const char *d = std::getenv("TMPDIR");
|
||||
path = (d ? d : "/tmp");
|
||||
path += "/vdm_engine_XXXXXX";
|
||||
path = ::mkdtemp(path.data()) ? path : "";
|
||||
}
|
||||
~TmpDir() {
|
||||
// best-effort recursive cleanup of our flat dir
|
||||
if (path.empty())
|
||||
return;
|
||||
std::string cmd = "rm -rf '" + path + "'";
|
||||
(void)std::system(cmd.c_str());
|
||||
}
|
||||
std::string file(const std::string &name) const { return path + "/" + name; }
|
||||
};
|
||||
|
||||
struct Recorder {
|
||||
std::promise<Result<DownloadOutcome>> done;
|
||||
std::future<Result<DownloadOutcome>> fut = done.get_future();
|
||||
std::atomic<bool> fired{false};
|
||||
std::vector<EngineState> states;
|
||||
std::mutex mu;
|
||||
std::atomic<int> auth_calls{0};
|
||||
std::atomic<int> decision_calls{0};
|
||||
// A probe callback can fire before the caller has stored the handle returned by
|
||||
// eng.start(). Callbacks that reach back into the handle wait on this.
|
||||
std::atomic<bool> handle_ready{false};
|
||||
|
||||
void arm(DownloadHandle &) { handle_ready.store(true, std::memory_order_release); }
|
||||
|
||||
DownloadCallbacks cbs(DownloadHandle *h = nullptr, std::string user = "",
|
||||
std::string pass = "") {
|
||||
DownloadCallbacks c;
|
||||
c.on_state = [this](EngineState, EngineState to, const std::optional<ErrorInfo> &) {
|
||||
std::lock_guard lk(mu);
|
||||
states.push_back(to);
|
||||
};
|
||||
c.on_finished = [this](Result<DownloadOutcome> r) {
|
||||
if (!fired.exchange(true))
|
||||
done.set_value(std::move(r));
|
||||
};
|
||||
if (h) {
|
||||
c.on_auth_required = [this, h, user, pass](const AuthChallenge &) {
|
||||
auth_calls.fetch_add(1);
|
||||
while (!handle_ready.load(std::memory_order_acquire))
|
||||
std::this_thread::sleep_for(1ms);
|
||||
h->provide_auth(user, pass, false);
|
||||
};
|
||||
}
|
||||
return c;
|
||||
}
|
||||
Result<DownloadOutcome> wait(std::chrono::seconds to = 40s) {
|
||||
if (fut.wait_for(to) != std::future_status::ready)
|
||||
return Err{Error::timeout, "engine test wait"};
|
||||
return fut.get();
|
||||
}
|
||||
bool saw(EngineState s) {
|
||||
std::lock_guard lk(mu);
|
||||
for (auto x : states)
|
||||
if (x == s)
|
||||
return true;
|
||||
return false;
|
||||
}
|
||||
};
|
||||
|
||||
std::uint64_t file_size(const std::string &p) {
|
||||
int fd = ::open(p.c_str(), O_RDONLY);
|
||||
if (fd < 0)
|
||||
return ~0ull;
|
||||
off_t e = ::lseek(fd, 0, SEEK_END);
|
||||
::close(fd);
|
||||
return e < 0 ? ~0ull : static_cast<std::uint64_t>(e);
|
||||
}
|
||||
|
||||
// The reference SHA-256 the testserver will report for a given path.
|
||||
std::string server_sha(TestServer &srv, const std::string &mode, const std::string &size) {
|
||||
// one-shot GET /<mode>/sha256/<size> via a throwaway Engine::probe? no — use raw curl
|
||||
// through a small helper. Simplest: shell out.
|
||||
std::string url = srv.url("/" + mode + "/sha256/" + size);
|
||||
std::string cmd = "curl -s '" + url + "'";
|
||||
std::string out;
|
||||
if (FILE *f = ::popen(cmd.c_str(), "r")) {
|
||||
char buf[512];
|
||||
while (std::fgets(buf, sizeof buf, f))
|
||||
out += buf;
|
||||
::pclose(f);
|
||||
}
|
||||
auto q = out.find("\"sha256\"");
|
||||
if (q == std::string::npos)
|
||||
return {};
|
||||
auto colon = out.find(':', q);
|
||||
auto open = out.find('"', colon);
|
||||
auto close = out.find('"', open + 1);
|
||||
if (open == std::string::npos || close == std::string::npos)
|
||||
return {};
|
||||
return out.substr(open + 1, close - open - 1);
|
||||
}
|
||||
|
||||
DownloadSpec spec_for(TestServer &srv, const std::string &urlpath, const std::string &save) {
|
||||
DownloadSpec s;
|
||||
s.url = srv.url(urlpath);
|
||||
s.save_path = save;
|
||||
return s;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
VT_TEST(engine_plain_multisegment_download) {
|
||||
TestServer srv;
|
||||
VT_REQUIRE(srv.available());
|
||||
TmpDir td;
|
||||
VT_REQUIRE(!td.path.empty());
|
||||
|
||||
Recorder rec;
|
||||
Engine eng;
|
||||
auto h = eng.start(spec_for(srv, "/plain/file/4M", td.file("a.bin")), rec.cbs());
|
||||
|
||||
auto r = rec.wait();
|
||||
VT_REQUIRE(r.has_value());
|
||||
VT_CHECK_EQ(r.value().final_path, td.file("a.bin"));
|
||||
VT_CHECK_EQ(r.value().bytes, 4u * 1024 * 1024);
|
||||
VT_CHECK_EQ(file_size(td.file("a.bin")), 4u * 1024 * 1024);
|
||||
VT_CHECK(rec.saw(EngineState::downloading));
|
||||
VT_CHECK(rec.saw(EngineState::complete));
|
||||
|
||||
auto got = hash_file(td.file("a.bin"), Checksum::Algo::sha256);
|
||||
VT_REQUIRE(got.has_value());
|
||||
VT_CHECK_EQ(got.value(), server_sha(srv, "plain", "4M"));
|
||||
// the sidecar is gone on success
|
||||
VT_CHECK_EQ(::access((td.file("a.bin") + ".veloxpart.meta").c_str(), F_OK), -1);
|
||||
}
|
||||
|
||||
VT_TEST(engine_checksum_pass_and_mismatch) {
|
||||
TestServer srv;
|
||||
VT_REQUIRE(srv.available());
|
||||
TmpDir td;
|
||||
Engine eng;
|
||||
|
||||
std::string want = server_sha(srv, "plain", "1M");
|
||||
VT_REQUIRE(!want.empty());
|
||||
|
||||
{
|
||||
Recorder rec;
|
||||
auto s = spec_for(srv, "/plain/file/1M", td.file("ok.bin"));
|
||||
s.checksum = Checksum{Checksum::Algo::sha256, want};
|
||||
auto h = eng.start(std::move(s), rec.cbs());
|
||||
auto r = rec.wait();
|
||||
VT_REQUIRE(r.has_value());
|
||||
VT_CHECK(rec.saw(EngineState::verifying));
|
||||
}
|
||||
{
|
||||
Recorder rec;
|
||||
auto s = spec_for(srv, "/plain/file/1M", td.file("bad.bin"));
|
||||
s.checksum = Checksum{Checksum::Algo::sha256, std::string(64, 'a')};
|
||||
auto h = eng.start(std::move(s), rec.cbs());
|
||||
auto r = rec.wait();
|
||||
VT_REQUIRE(!r.has_value());
|
||||
VT_CHECK_EQ(r.error().code, Error::checksum_mismatch);
|
||||
}
|
||||
}
|
||||
|
||||
VT_TEST(engine_non_resumable_single_segment) {
|
||||
TestServer srv;
|
||||
VT_REQUIRE(srv.available());
|
||||
TmpDir td;
|
||||
Recorder rec;
|
||||
Engine eng;
|
||||
auto h = eng.start(spec_for(srv, "/no-range/file/2M", td.file("nr.bin")), rec.cbs());
|
||||
auto r = rec.wait();
|
||||
VT_REQUIRE(r.has_value());
|
||||
VT_CHECK_EQ(file_size(td.file("nr.bin")), 2u * 1024 * 1024);
|
||||
auto got = hash_file(td.file("nr.bin"), Checksum::Algo::sha256);
|
||||
VT_CHECK_EQ(got.value(), server_sha(srv, "no-range", "2M"));
|
||||
}
|
||||
|
||||
VT_TEST(engine_404_is_an_error) {
|
||||
TestServer srv;
|
||||
VT_REQUIRE(srv.available());
|
||||
TmpDir td;
|
||||
Recorder rec;
|
||||
Engine eng;
|
||||
auto h = eng.start(spec_for(srv, "/plain/nope", td.file("x.bin")), rec.cbs());
|
||||
auto r = rec.wait();
|
||||
VT_REQUIRE(!r.has_value());
|
||||
VT_CHECK_EQ(r.error().code, Error::not_found);
|
||||
VT_CHECK(rec.saw(EngineState::failed));
|
||||
}
|
||||
|
||||
VT_TEST(engine_cancel_mid_download) {
|
||||
TestServer srv;
|
||||
VT_REQUIRE(srv.available());
|
||||
TmpDir td;
|
||||
Recorder rec;
|
||||
Engine eng;
|
||||
auto h = eng.start(spec_for(srv, "/throttled/file/8M", td.file("c.bin")), rec.cbs());
|
||||
|
||||
for (int i = 0; i < 200 && !rec.saw(EngineState::downloading); ++i)
|
||||
std::this_thread::sleep_for(10ms);
|
||||
VT_REQUIRE(rec.saw(EngineState::downloading));
|
||||
h.cancel(/*discard_partial=*/true);
|
||||
|
||||
auto r = rec.wait();
|
||||
VT_REQUIRE(!r.has_value());
|
||||
VT_CHECK_EQ(r.error().code, Error::canceled);
|
||||
VT_CHECK(rec.saw(EngineState::cancelled));
|
||||
VT_CHECK_EQ(::access((td.file("c.bin") + ".veloxpart").c_str(), F_OK), -1); // discarded
|
||||
}
|
||||
|
||||
VT_TEST(engine_pause_resume_completes) {
|
||||
TestServer srv;
|
||||
VT_REQUIRE(srv.available());
|
||||
TmpDir td;
|
||||
Recorder rec;
|
||||
Engine eng;
|
||||
auto h = eng.start(spec_for(srv, "/throttled/file/2M", td.file("pr.bin")), rec.cbs());
|
||||
|
||||
for (int i = 0; i < 200 && !rec.saw(EngineState::downloading); ++i)
|
||||
std::this_thread::sleep_for(10ms);
|
||||
h.pause();
|
||||
for (int i = 0; i < 100 && h.state() != EngineState::paused; ++i)
|
||||
std::this_thread::sleep_for(20ms);
|
||||
VT_CHECK_EQ(h.state(), EngineState::paused);
|
||||
h.resume();
|
||||
|
||||
auto r = rec.wait(90s);
|
||||
VT_REQUIRE(r.has_value());
|
||||
VT_CHECK_EQ(file_size(td.file("pr.bin")), 2u * 1024 * 1024);
|
||||
auto got = hash_file(td.file("pr.bin"), Checksum::Algo::sha256);
|
||||
VT_CHECK_EQ(got.value(), server_sha(srv, "throttled", "2M"));
|
||||
}
|
||||
|
||||
VT_TEST(engine_resume_after_a_fresh_task) {
|
||||
// Simulates kill -9: cancel WITHOUT discard, then a new task with allow_resume picks
|
||||
// up the .veloxpart[.meta] and finishes with a byte-identical file.
|
||||
TestServer srv;
|
||||
VT_REQUIRE(srv.available());
|
||||
TmpDir td;
|
||||
std::string save = td.file("resume.bin");
|
||||
std::string want = server_sha(srv, "throttled", "3M");
|
||||
|
||||
{
|
||||
Recorder rec;
|
||||
Engine eng;
|
||||
auto h = eng.start(spec_for(srv, "/throttled/file/3M", save), rec.cbs());
|
||||
for (int i = 0; i < 800 && (h.progress().downloaded < 512u * 1024); ++i)
|
||||
std::this_thread::sleep_for(10ms);
|
||||
VT_REQUIRE(h.progress().downloaded >= 512u * 1024);
|
||||
h.cancel(/*discard_partial=*/false);
|
||||
(void)rec.wait();
|
||||
VT_CHECK_EQ(::access((save + ".veloxpart.meta").c_str(), F_OK), 0); // sidecar kept
|
||||
}
|
||||
{
|
||||
Recorder rec;
|
||||
Engine eng;
|
||||
auto s = spec_for(srv, "/throttled/file/3M", save); // same source -> sidecar validates
|
||||
s.allow_resume = true;
|
||||
auto h = eng.start(std::move(s), rec.cbs());
|
||||
auto r = rec.wait(120s);
|
||||
VT_REQUIRE(r.has_value());
|
||||
VT_CHECK_EQ(file_size(save), 3u * 1024 * 1024);
|
||||
auto got = hash_file(save, Checksum::Algo::sha256);
|
||||
VT_CHECK_EQ(got.value(), want); // byte-identical after resume
|
||||
}
|
||||
}
|
||||
|
||||
VT_TEST(engine_flaky_reset_retries_to_completion) {
|
||||
TestServer srv;
|
||||
VT_REQUIRE(srv.available());
|
||||
TmpDir td;
|
||||
Recorder rec;
|
||||
Engine eng;
|
||||
// flaky-reset RSTs the first two attempts per (path,range); a single-segment request
|
||||
// therefore needs the retry loop.
|
||||
auto s = spec_for(srv, "/flaky-reset/file/256K", td.file("fl.bin"));
|
||||
s.segments = 1;
|
||||
s.max_retries = 40; // the server RSTs at the halfway point every attempt -> ~18 halvings
|
||||
auto h = eng.start(std::move(s), rec.cbs());
|
||||
auto r = rec.wait(120s);
|
||||
VT_REQUIRE(r.has_value());
|
||||
VT_CHECK_EQ(file_size(td.file("fl.bin")), 256u * 1024);
|
||||
auto got = hash_file(td.file("fl.bin"), Checksum::Algo::sha256);
|
||||
VT_CHECK_EQ(got.value(), server_sha(srv, "flaky-reset", "256K"));
|
||||
VT_CHECK(rec.saw(EngineState::retry_wait) || rec.saw(EngineState::connecting));
|
||||
}
|
||||
|
||||
VT_TEST(engine_401_then_provide_auth_completes) {
|
||||
TestServer srv;
|
||||
VT_REQUIRE(srv.available());
|
||||
TmpDir td;
|
||||
Recorder rec;
|
||||
Engine eng;
|
||||
DownloadHandle h;
|
||||
auto cbs = rec.cbs(&h, "test", "test");
|
||||
h = eng.start(spec_for(srv, "/401-basic/file/1M", td.file("au.bin")), std::move(cbs));
|
||||
rec.arm(h); // publish h to the auth callback (which may already be waiting)
|
||||
|
||||
auto r = rec.wait();
|
||||
VT_REQUIRE(r.has_value());
|
||||
VT_CHECK(rec.auth_calls.load() >= 1);
|
||||
VT_CHECK_EQ(file_size(td.file("au.bin")), 1u * 1024 * 1024);
|
||||
}
|
||||
@@ -0,0 +1,98 @@
|
||||
# daemon/ produces the veloxd binary and the static libraries it is built from.
|
||||
# Owned by lane DAEMON. Wired in by PKG via add_subdirectory(daemon) in the root file,
|
||||
# guarded on this file existing.
|
||||
#
|
||||
# Layering (CLAUDE.md §3): depends on velox::core and velox::proto. No Qt. The engine
|
||||
# (velox::core) is not linked yet — it arrives when sched/ and the task glue land. This
|
||||
# drop is the RPC transports (Unix socket + loopback WebSocket), the SQLite store, and a
|
||||
# dispatcher skeleton so the CLI and GUI have a real server to talk to.
|
||||
|
||||
if(NOT TARGET nlohmann_json::nlohmann_json)
|
||||
find_package(nlohmann_json 3.11 REQUIRED)
|
||||
endif()
|
||||
find_package(Threads REQUIRED)
|
||||
find_package(SQLite3 REQUIRED)
|
||||
find_package(OpenSSL REQUIRED) # libcrypto: WebSocket accept hash, pairing token hash
|
||||
|
||||
# --- generated: migrations_embedded.hpp from src/store/migrations/*.sql ---------------
|
||||
set(_mig_dir ${CMAKE_CURRENT_SOURCE_DIR}/src/store/migrations)
|
||||
set(_mig_hdr ${CMAKE_CURRENT_BINARY_DIR}/generated/migrations_embedded.hpp)
|
||||
file(GLOB _mig_srcs ${_mig_dir}/*.sql)
|
||||
add_custom_command(
|
||||
OUTPUT ${_mig_hdr}
|
||||
COMMAND ${CMAKE_COMMAND} -DMIG_DIR=${_mig_dir} -DOUT=${_mig_hdr}
|
||||
-P ${CMAKE_CURRENT_SOURCE_DIR}/cmake/embed_migrations.cmake
|
||||
DEPENDS ${_mig_srcs} ${CMAKE_CURRENT_SOURCE_DIR}/cmake/embed_migrations.cmake
|
||||
COMMENT "Embedding SQL migrations"
|
||||
VERBATIM)
|
||||
add_custom_target(veloxd_migrations_hdr DEPENDS ${_mig_hdr})
|
||||
|
||||
# --- veloxd_store — SQLite store, migrations, crypto helpers --------------------------
|
||||
add_library(veloxd_store STATIC
|
||||
src/util/crypto.cpp
|
||||
src/store/sqlite.cpp
|
||||
src/store/migrations.cpp
|
||||
src/store/pairings.cpp
|
||||
src/store/settings.cpp
|
||||
src/store/tasks.cpp
|
||||
${_mig_hdr}
|
||||
)
|
||||
add_library(velox::daemon_store ALIAS veloxd_store)
|
||||
target_include_directories(veloxd_store
|
||||
PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/src
|
||||
PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/generated
|
||||
)
|
||||
target_compile_features(veloxd_store PUBLIC cxx_std_23)
|
||||
target_compile_options(veloxd_store PRIVATE -Wall -Wextra -Wpedantic -Werror)
|
||||
target_link_libraries(veloxd_store PUBLIC SQLite::SQLite3 velox::proto nlohmann_json::nlohmann_json PRIVATE OpenSSL::Crypto)
|
||||
|
||||
# --- veloxd_fs — the saveDir/filename path-traversal boundary (security) -----------
|
||||
# daemon/docs/safepath-adversarial.md is the spec; safepath_test.cpp is that table.
|
||||
add_library(veloxd_fs STATIC src/fs/safepath.cpp)
|
||||
add_library(velox::daemon_fs ALIAS veloxd_fs)
|
||||
target_include_directories(veloxd_fs PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/src)
|
||||
target_compile_features(veloxd_fs PUBLIC cxx_std_23)
|
||||
target_compile_options(veloxd_fs PRIVATE -Wall -Wextra -Wpedantic -Werror)
|
||||
|
||||
# --- veloxd_sched — the concurrency governor (pure; no engine link yet, see
|
||||
# daemon/docs/deferrals.md D4) ---------------------------------------------------
|
||||
add_library(veloxd_sched STATIC
|
||||
src/sched/schedule_window.cpp
|
||||
src/sched/governor.cpp
|
||||
src/sched/scheduler.cpp
|
||||
)
|
||||
add_library(velox::daemon_sched ALIAS veloxd_sched)
|
||||
target_include_directories(veloxd_sched PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/src)
|
||||
target_compile_features(veloxd_sched PUBLIC cxx_std_23)
|
||||
target_compile_options(veloxd_sched PRIVATE -Wall -Wextra -Wpedantic -Werror)
|
||||
target_link_libraries(veloxd_sched PUBLIC velox::proto velox::core veloxd_store nlohmann_json::nlohmann_json)
|
||||
|
||||
# --- veloxd_rpc — the RPC transports + dispatcher ------------------------------------
|
||||
add_library(veloxd_rpc STATIC
|
||||
src/rpc/runtime_dir.cpp
|
||||
src/rpc/event_loop.cpp
|
||||
src/rpc/uds_server.cpp
|
||||
src/rpc/ws_frame.cpp
|
||||
src/rpc/ws_handshake.cpp
|
||||
src/rpc/ws_server.cpp
|
||||
src/rpc/pairing.cpp
|
||||
src/rpc/dispatcher.cpp
|
||||
)
|
||||
add_library(velox::daemon_rpc ALIAS veloxd_rpc)
|
||||
|
||||
target_include_directories(veloxd_rpc PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/src)
|
||||
target_compile_features(veloxd_rpc PUBLIC cxx_std_23)
|
||||
target_compile_options(veloxd_rpc PRIVATE -Wall -Wextra -Wpedantic -Werror)
|
||||
target_link_libraries(veloxd_rpc
|
||||
PUBLIC velox::proto veloxd_store veloxd_fs nlohmann_json::nlohmann_json Threads::Threads
|
||||
)
|
||||
|
||||
# --- veloxd — the daemon binary -------------------------------------------------------
|
||||
add_executable(veloxd src/main.cpp)
|
||||
target_compile_features(veloxd PRIVATE cxx_std_23)
|
||||
target_compile_options(veloxd PRIVATE -Wall -Wextra -Wpedantic -Werror)
|
||||
target_link_libraries(veloxd PRIVATE veloxd_rpc veloxd_sched)
|
||||
|
||||
if(VELOX_BUILD_TESTS AND EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/tests/CMakeLists.txt)
|
||||
add_subdirectory(tests)
|
||||
endif()
|
||||
@@ -0,0 +1,52 @@
|
||||
# Generates migrations_embedded.hpp from daemon/src/store/migrations/*.sql.
|
||||
#
|
||||
# cmake -DMIG_DIR=<dir> -DOUT=<file> -P embed_migrations.cmake
|
||||
#
|
||||
# Each NNNN_name.sql becomes a Migration{ version = NNNN, name = "NNNN_name", sql = R"..." }.
|
||||
# The delimiter for the raw string literal is chosen to not collide with the file body.
|
||||
|
||||
file(GLOB _sql_files "${MIG_DIR}/*.sql")
|
||||
list(SORT _sql_files)
|
||||
|
||||
set(_entries "")
|
||||
foreach(_f ${_sql_files})
|
||||
get_filename_component(_stem "${_f}" NAME_WE) # 0001_initial
|
||||
string(REGEX MATCH "^([0-9]+)_" _m "${_stem}")
|
||||
if(NOT _m)
|
||||
message(FATAL_ERROR "migration file '${_f}' does not start with NNNN_")
|
||||
endif()
|
||||
string(REGEX REPLACE "^0*([0-9]+)_.*$" "\\1" _ver "${_stem}")
|
||||
|
||||
file(READ "${_f}" _body)
|
||||
# Pick a raw-string delimiter guaranteed absent from the body.
|
||||
set(_delim "MIGSQL")
|
||||
while(_body MATCHES "\\)${_delim}\"")
|
||||
set(_delim "${_delim}X")
|
||||
endwhile()
|
||||
|
||||
string(APPEND _entries
|
||||
" Migration{ ${_ver}, \"${_stem}\", R\"${_delim}(\n${_body}\n)${_delim}\" },\n")
|
||||
endforeach()
|
||||
|
||||
list(LENGTH _sql_files _count)
|
||||
|
||||
set(_out "// GENERATED by embed_migrations.cmake — do not edit. Source: src/store/migrations/*.sql
|
||||
#pragma once
|
||||
#include <array>
|
||||
#include \"store/migrations.hpp\"
|
||||
|
||||
namespace velox::daemon::store {
|
||||
|
||||
inline constexpr std::array<Migration, ${_count}> kEmbeddedMigrations = {{
|
||||
${_entries}}};
|
||||
|
||||
} // namespace velox::daemon::store
|
||||
")
|
||||
|
||||
if(EXISTS "${OUT}")
|
||||
file(READ "${OUT}" _existing)
|
||||
if(_existing STREQUAL "${_out}")
|
||||
return() # unchanged — do not rewrite, keeps the build stable
|
||||
endif()
|
||||
endif()
|
||||
file(WRITE "${OUT}" "${_out}")
|
||||
@@ -0,0 +1,67 @@
|
||||
# DAEMON → CORE — the engine API `sched/` needs before it can be written
|
||||
|
||||
Status: **resolved**. CORE answered in full: `core/docs/adr-0011-core-response.md`
|
||||
(`lane/core`, commit `7bf5cb5`). Kept as a record of what was asked and the shape of the
|
||||
answer; the API itself is now specified in
|
||||
`docs/adr/0011-admission-control-and-the-segment-budget.md` §"Engine API `sched/` is built
|
||||
against". `sched/` may be written against it.
|
||||
|
||||
Ranking followed `contracts/README.md` rule 4 conventions even though this wasn't a
|
||||
`contracts/` change: new API surface = cheap, land anytime; a behavioural promise (min-1
|
||||
fairness) = needed CORE's explicit sign-off before DAEMON built on the assumption. That
|
||||
sign-off is in.
|
||||
|
||||
---
|
||||
|
||||
## C1. Occupancy read-out, not inference — **resolved**
|
||||
|
||||
Requested `budget()`, `segments_active(TaskId)`, `on_budget_changed`. CORE's answer adds
|
||||
`starved_tasks()` and `starved_since(TaskId)` (amendment A3) and pins two definitions:
|
||||
`segments_active(id)` counts a segment in `connecting` state as held (it is progress, not
|
||||
starvation), and `tasks_starved` counts only `segments_active == 0`. See ADR 0011 §3.6.
|
||||
|
||||
## C2. `set_max_active_segments(uint32_t)` live-apply — **resolved: drain, never kill**
|
||||
|
||||
Confirmed DAEMON's assumption. Lowering runs in-flight segments to their next boundary; no
|
||||
new segment starts while over the new ceiling; nothing is aborted, no partial range lost.
|
||||
If the new ceiling is below the running-task count, CORE honours min-1 for the top-priority
|
||||
subset and reports the rest via `tasks_starved` — DAEMON's governor must reconcile and
|
||||
pause the lowest-priority excess itself (CORE does not auto-pause). See ADR 0011 §2.
|
||||
|
||||
## C3. `set_host_segment_cap(host, uint32_t)` — **resolved, confirmed as proposed**
|
||||
|
||||
CORE keeps the `host → cap` map and derives a task's host from its URL/mirror set; DAEMON
|
||||
owns the table and pushes it. See ADR 0011 §4.
|
||||
|
||||
## C4. Contract gap: `connection.maxActiveSegments` — **resolved by PROTO**
|
||||
|
||||
PROTO landed it (ADR 0012, `connection.maxActiveSegments` default 32,
|
||||
`connection.maxTotalBufferBytes` default 128 MiB, `TaskDetail.effectiveBufferBytes`) while
|
||||
this was in flight. No daemon-local stopgap needed — `sched/` reads the wire field
|
||||
directly. See ADR 0011 §6.
|
||||
|
||||
## C5. Fairness rule sign-off — **resolved, with two amendments**
|
||||
|
||||
CORE confirmed min-1-before-seconds is implementable without a priority-inversion at slot
|
||||
release (two-pass allocator: guarantee pass over zero-slot tasks in DAEMON's priority
|
||||
order, then a growth pass; a released slot always re-enters the pool at pass 1, never
|
||||
handed back locally). Two amendments to what DAEMON assumed:
|
||||
|
||||
- **A1** — "steal" (slot-neutral, unchanged) isn't the whole mechanism; **"yield"** is the
|
||||
slot-transfer operation that actually satisfies min-1 out of a full budget: an
|
||||
over-quota task releases one slot at its next segment boundary, bounded by that
|
||||
segment's remaining bytes.
|
||||
- **A2** — "admission implies progress" is **bounded-delay**, not immediate:
|
||||
`time_to_first_slot ≤ min(next yield boundary, low_speed_secs) + connect_timeout`, not
|
||||
"connect timeout + per-host cap" alone. DAEMON's starvation-invariant assertion window
|
||||
widened from the originally proposed 2 s to `low_speed_secs + connect_timeout` (~45 s)
|
||||
accordingly.
|
||||
|
||||
Priority order (open item 4) is an ordered `TaskId` list pushed via `set_task_order` on
|
||||
change — not an integer, not per-tick. See ADR 0011 §3.
|
||||
|
||||
## C6. Probe pool sized outside the segment budget — **resolved, confirmed**
|
||||
|
||||
Dedicated pool, default size 4, `set_probe_pool_size(uint32_t)`, independent of
|
||||
`maxActiveSegments`; probe cancellation is immediate. DAEMON still bounds probe
|
||||
*submission* on its own side. See ADR 0011 §5.
|
||||
@@ -0,0 +1,13 @@
|
||||
# DAEMON — deferred work, tracked
|
||||
|
||||
Things that are deliberately incomplete in `daemon/` right now, with why and when they
|
||||
close. Kept here (not buried in commit messages) so the next pass can see them at a glance.
|
||||
|
||||
| # | What | Where | Why deferred | Closes when |
|
||||
|---|---|---|---|---|
|
||||
| D1 | Pairing prompt is `EnvAutoApprover` (needs `VELOX_PAIR_AUTO=1`) | `rpc/pairing.hpp`, `main.cpp` | A GUI dialog / `org.freedesktop.Notifications` approver is integration work | Build step 7 (systemd + notifications) |
|
||||
| D2 | `download.probe` → `-32603` | `rpc/dispatcher.cpp` | `download.add` is wired (`fs/safepath` + store, real `-32011`); `download.probe` needs the engine's probe path for `-32013` | probe with the engine link (CORE stage 3 is landed; wire `Engine::probe`) |
|
||||
| D3 | Stub handlers for everything except `session.*`, `download.add/list/get` | `rpc/dispatcher.cpp` | No store behind them yet (categories/queues/rules/settings/limiter/schedule) | Per method, as the store query modules land behind them |
|
||||
| ~~D4a~~ | **Closed** — `sched/engine_port_core.hpp` wraps `vdm::Engine` + `segment_budget()`; `main.cpp` constructs `Engine` + `Scheduler`, calls `reconcile_after_restart` / `reload_config` / `tick` at startup | — | — | done (`lane/core` stage 8 merged) |
|
||||
| D4b | timer + nudges: a 1 s `timerfd` re-runs `Scheduler::tick()` and `download.add` nudges via `on_mutation`. `download.pause`/`resume`/`start`/`cancel` and the queue.* handlers still don't touch the scheduler | `rpc/dispatcher.cpp` | those handlers are still stubs (D3) | as each handler is implemented behind the store, it calls `on_mutation` / drives the scheduler |
|
||||
| D5 | `event.*` fan-out not implemented; `session.subscribe` accepts and echoes but nothing is emitted | `rpc/uds_server.cpp`, `rpc/ws_server.cpp` | No task state to broadcast until the engine is wired. `Scheduler::on_engine_state` is the hook it will fire from | With D4a — the same engine-state callback feeds both the store and `event.task.state` |
|
||||
@@ -0,0 +1,94 @@
|
||||
# DAEMON review — CORE's engine API (`core/docs/engine-api-m1.md`, `lane/core@d6cf1fe`)
|
||||
|
||||
**Verdict: sign off.** Nothing here forces a `daemon/src/sched/` or RPC-dispatch rewrite.
|
||||
The value types are final enough to build `sched/` against now; `Engine` / `DownloadHandle`
|
||||
bodies landing in stage 8 is fine. The split matches ADR 0011 and ADR 0013 exactly.
|
||||
|
||||
Answers to the five open questions, then the small things to confirm.
|
||||
|
||||
## Answers
|
||||
|
||||
### Q1 — `probe_hint`: keep it optional, as sketched
|
||||
|
||||
DAEMON has a `ProbeResult` on exactly one path: the File Info dialog, where the user
|
||||
already waited for `download.probe` and then clicked Download. Every other entry —
|
||||
`capture.offer` → take, `velox add`, `download.addBatch`, the restart flow — has no probe
|
||||
in hand. Forcing the engine to always probe adds a round trip to the one case where the
|
||||
user just sat through one; forcing DAEMON to always probe first means reimplementing the
|
||||
engine's probe on the daemon side. Pass `probe_hint` when we have it, omit it otherwise —
|
||||
the two code paths are worth keeping.
|
||||
|
||||
### Q2 — one `cancel(discard_partial)`, not a separate `remove()`
|
||||
|
||||
The two wire methods map cleanly onto the one call:
|
||||
|
||||
| wire | live task | already terminal |
|
||||
|---|---|---|
|
||||
| `download.cancel` | `handle.cancel(discard_partial=false)` — keeps `.veloxpart` | no-op on the handle; DAEMON marks the row `cancelled` |
|
||||
| `download.remove {deleteFile}` | `handle.cancel(discard_partial=true)` | no-op on the handle; DAEMON deletes the row and, if `deleteFile`, the finished file |
|
||||
|
||||
`download.remove` on a completed task is pure DAEMON-side work (row + optional file); the
|
||||
handle is already terminal so `cancel()` no-ops, which is exactly what we want. No
|
||||
`handle.remove()` needed.
|
||||
|
||||
### Q3 — `{restart, keep_partial, abort}` is enough, if the engine owns the mechanical 416 retry
|
||||
|
||||
For `server_file_changed` the three options are right and complete. For
|
||||
`range_metadata_stale` (416): `docs/04` §7 already has the engine re-probe and re-split
|
||||
automatically. Keep that — DAEMON does not want to be in the loop for a routine 416. Only
|
||||
escalate to `on_decision_needed{range_metadata_stale}` when the automatic re-probe/re-split
|
||||
*also* fails to reconcile, and at that point "retry the same range once more" is not a
|
||||
useful fourth option (the engine already exhausted it). So: no fourth value, provided the
|
||||
engine handles the common 416 without a callback.
|
||||
|
||||
### Q4 — per-task 4 Hz `on_progress` is fine; `on_progress_batch` optional
|
||||
|
||||
DAEMON already has to coalesce across tasks: `event.task.progress` is "batched ≤ 4 Hz into
|
||||
a single array message" (AGENT-DAEMON.md item 5). So 80 per-task callbacks/s land in
|
||||
DAEMON's fan-out queue and are re-emitted as one array at 4 Hz regardless. Per-task keeps
|
||||
the handle↔callback correspondence simple and is not a bottleneck. If the engine's timer
|
||||
thread is already walking every task to build those callbacks, a
|
||||
`on_progress_batch(span<Progress>)` is strictly less work for both sides and we'd take it —
|
||||
but it is not needed for M1 and should not hold stage 8.
|
||||
|
||||
### Q5 — `refresh_url` while `downloading`: restart all segments on the new URL
|
||||
|
||||
`download.refreshUrl`'s contract is the signed-URL-expiry case: "re-probes and compares
|
||||
size and validator; if they still match, the transfer resumes from where it stopped." That
|
||||
wants consistency — every segment on the new URL once the re-probe validates. Mirror
|
||||
rotation (finish in-flight on the old host, new work elsewhere) is a different mechanism
|
||||
and it is already `DownloadSpec.mirrors` + the segmenter's 3-failure requeue, not
|
||||
`refresh_url`. So: on `refresh_url`, re-probe, and if size+validator match, move all
|
||||
segments to the new URL from their current offsets; if they don't match, surface it
|
||||
(`on_decision_needed` or a `refresh_url` error) rather than silently restarting.
|
||||
|
||||
## Confirmed by CORE (`lane/core@3da4cd6`)
|
||||
|
||||
1. **`vdm::TaskId`** — cheap-copy and `std::hash`-able; DAEMON never constructs one, only
|
||||
receives it from `start()` / callbacks and passes it back to `set_task_order()`.
|
||||
`sched/` keeps the `vdm::TaskId → wire taskId` map keyed off `handle.id()`.
|
||||
2. **Parent directory** — DAEMON `mkdir -p`s `save_path`'s parent before `start()`; the
|
||||
engine opens the file and fails the task with `Error::path_rejected` if it is missing.
|
||||
Now explicit on `DownloadSpec`'s doc comment.
|
||||
3. **`sha512`** — added as the fourth `Checksum::Algo`, matching the wire set. No `-32602`
|
||||
at the RPC edge; DAEMON passes it straight through.
|
||||
4. **Cancel ordering** — `on_state(_, cancelled, nullopt)` then `on_finished`, always in
|
||||
that order. Note the taxonomy value is spelled `canceled` (one L): the finish is
|
||||
`on_finished(Err{Error::canceled})`, `error.code == canceled`. `sched/` keys on that
|
||||
to write a `cancelled` history row rather than a user-facing failure. (The wire
|
||||
`TaskState` / `EngineState` spelling stays `cancelled`; only `vdm::Error` is one-L.)
|
||||
|
||||
## Related, landed the same pass
|
||||
|
||||
`rate/token_bucket` — the global → queue → task speed-limiter hierarchy (`docs/04` §6),
|
||||
reached via `Engine::rate_limiter()`. DAEMON's `limiter.set {globalBps, enabled}` maps to
|
||||
`rate_limiter().set_global_limit(...)`; `limiter.get` reads it back. Wire this alongside
|
||||
the `download.add` → `start()` glue.
|
||||
|
||||
## Integration timing
|
||||
|
||||
Wire it after `sched/` lands — the scheduler is what calls `start()` / `pause()` /
|
||||
`resume()` / `cancel()` and drives `segment_budget().set_task_order()`. Order:
|
||||
`sched/` (against these headers) → `download.add` → `start()` glue → the callback→wire
|
||||
projection. `sched/` does not need the `Engine` bodies, only the signatures in this doc,
|
||||
so stage 8 and `sched/` proceed in parallel.
|
||||
@@ -0,0 +1,92 @@
|
||||
# DAEMON → PROTO — requests against `contracts/` (and its codegen)
|
||||
|
||||
Status: **open**. Raised by lane DAEMON while building `rpc/` against `1.3.0`.
|
||||
PROTO owns `contracts/`, including `contracts/codegen/`. Ranking per
|
||||
`contracts/README.md` rule 4: a codegen output-shape change that every server must
|
||||
adopt is effectively **major for the C++ binding** even when the wire is untouched —
|
||||
it needs a version note and a regen, not a silent change.
|
||||
|
||||
---
|
||||
|
||||
## Status
|
||||
|
||||
- **P1 — landed** on `lane/proto` as `contracts/` **1.4.0** (commit `5e3e215`), as the
|
||||
`HandlerError` / `HandlerResult<T>` sketch below. Wire is byte-identical; C++-binding
|
||||
bump only. `rpc/` adopts it (the predicted `Result<T>` → `HandlerResult<T>` swap on the
|
||||
`on_*` overrides) **once `lane/proto` merges to `main`** — not against the unmerged
|
||||
branch. `uds_roundtrip`'s `-32603`-collapse guard flips to `-32010` in the same change.
|
||||
- **P2 — resolved.** `session.hello.version-mismatch`'s `data.expected` is now `$any`;
|
||||
the error-fixture compare is on `code` only, so `rpc/` echoes `kProtocolVersion` there.
|
||||
PROTO's writeup: `contracts/proto-answers-daemon-m1.md`.
|
||||
|
||||
---
|
||||
|
||||
## P1. The generated `Dispatcher` has no error channel below `-32603` — **blocking a conformant server**
|
||||
|
||||
`velox::proto::Dispatcher`'s 39 methods each return `Result<T>` =
|
||||
`std::expected<T, ParseError>`, and `dispatch()` maps **every** handler error to
|
||||
`ErrorCode::InternalError` (`-32603`):
|
||||
|
||||
```cpp
|
||||
auto r = handler.on_download_get(*p);
|
||||
if (!r)
|
||||
return make_error(id, ErrorCode::InternalError, r.error().message,
|
||||
nlohmann::json{{"path", r.error().path}});
|
||||
```
|
||||
|
||||
So a handler cannot return any of the contract's own error codes. The error fixtures
|
||||
in `contracts/fixtures/errors/` that a live server must satisfy (DAEMON DoD: "passes
|
||||
the full conformance suite as a server, over both transports") include:
|
||||
|
||||
| Fixture | Expected code | `data` | Originates |
|
||||
|---|---|---|---|
|
||||
| `download.get.not-found` | `-32010` | `{taskId}` | inside the handler |
|
||||
| `download.add.invalid-path` | `-32011` | `{path}` | inside the handler (after canonicalization) |
|
||||
| `download.probe.probe-failed` | `-32013` | `{httpStatus}` | inside the handler |
|
||||
| `session.pair.rate-limited` | `-32014` | `{retryAfterSec}` | server-side gate, but cleanest expressed as a handler result |
|
||||
| `session.hello.version-mismatch` | `-32001` | `{expected, actual}` | can be done server-side around `dispatch()` |
|
||||
| `session.hello.not-paired` | `-32002` | — | server-side WS auth gate, around `dispatch()` |
|
||||
|
||||
`-32001`, `-32002`, `-32003` DAEMON can and will handle in the server layer that wraps
|
||||
`dispatch()` (`-32003` is already in `dispatch()` itself). But `-32010`, `-32011`,
|
||||
`-32013` are per-method **handler outcomes** — the daemon knows "no such task" only
|
||||
after the store lookup, "outside allowed roots" only after `realpath()`. There is no
|
||||
correct way to surface them today except misreporting as `-32603`, which the TS
|
||||
conformance replay will reject on the `code` compare.
|
||||
|
||||
**Requested:** give the generated handler methods an error return that carries an
|
||||
`ErrorCode`, a message, and a free-form `data` object. Shape is PROTO's call; a
|
||||
minimal one that keeps `ParseError` for the parse path and adds a handler-error type:
|
||||
|
||||
```cpp
|
||||
struct HandlerError {
|
||||
ErrorCode code{ErrorCode::InternalError};
|
||||
std::string message;
|
||||
nlohmann::json data{nullptr};
|
||||
};
|
||||
template <class T> using HandlerResult = std::expected<T, HandlerError>;
|
||||
// Dispatcher::on_* return HandlerResult<T>; dispatch() forwards code/message/data
|
||||
// straight into make_error() instead of hard-coding InternalError.
|
||||
```
|
||||
|
||||
`FixtureDispatcher` and `conformance_main.cpp` would need the trivial follow-on edit
|
||||
(they only ever return success today, so it is a type-name swap).
|
||||
|
||||
Until this lands, DAEMON's `rpc/` server layer handles `-3200x` around `dispatch()`
|
||||
where it can, and every genuine in-handler failure collapses to `-32603` with a clear
|
||||
message — visibly non-conformant on three error fixtures, tracked here, not worked
|
||||
around by inventing a side channel.
|
||||
|
||||
---
|
||||
|
||||
## P2. `SessionHelloResult.transport` and `-32001` `data.expected` — minor clarifications
|
||||
|
||||
- `session.hello.version-mismatch`'s `data.expected` is `"1.0.0"` in the fixture, i.e.
|
||||
the daemon's *current* protocol version string, not a bare major. DAEMON will echo
|
||||
`kProtocolVersion` (`"1.3.0"`) there unless PROTO wants the fixture's literal
|
||||
`"1.0.0"` preserved — flag if the conformance compare is exact on that field rather
|
||||
than structural.
|
||||
- `SessionHelloResult.transport` is `std::optional` — DAEMON intends to always populate
|
||||
it (`"uds"` / `"ws"`) so a client knows its privilege level up front, as the field's
|
||||
own description invites. No change requested; noting the intent so a later "why is
|
||||
this always set" review has the answer.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user