Stage 8 of the CORE build order: the bodies behind the DownloadSpec / callback API reviewed in core/docs/engine-api-m1.md. Wires probe -> segment workers -> WriteBuffer -> SparseFile -> .veloxpart.meta -> retry/backoff -> SegmentBudget -> RateLimiter -> callbacks into one event-driven machine. - Engine (src/engine.cpp): owns HttpClient, Prober, SegmentBudget, RateLimiter and one timer jthread (min-heap of scheduled fns). start() builds a task and returns a DownloadHandle; ~Impl quiesces every task before joining the timer so no callback fires during teardown. - DownloadTaskState (src/task/download_task.cpp): one `mu` task lock; a shared_mutex over the worker map for the curl write path; callbacks collected under `mu` and fired after release via a separate deferred queue; weak_from_this() in every async hop. State machine over the CORE-owned EngineState subset, auto-pause on 401/407 and on a 200 where 206 was expected, validated resume via If-Range. - digest (src/task/digest.cpp): OpenSSL EVP hash_file() for the optional post-download checksum; links OpenSSL::Crypto PRIVATE. - Segmenter::release_segment(): hand a paused segment back to the pool unassigned so resume's assign_slot() picks it up instead of splitting a still-"assigned" range and orphaning its front half. - DownloadHandle now names the real control block (vdm::task:: DownloadTaskState, defined only in the engine TU) via a namespace-scope fwd decl and a public-but-effectively-engine-only ctor, replacing the nested State/friend pair. Every public signature is unchanged; DAEMON (vdm-79) confirmed sched/ names only the public API. Fixes found while building the end-to-end suite (tests/task/engine_test.cpp, 9 cases against tools/testserver, green under ASan/UBSan and TSan): - a dropped connection lost its unflushed WriteBuffer tail while advance() had already counted those bytes as done -> a retry resumed past an unwritten hole. Flush on the failure path. - when the byte counters hit total while other workers were still live, teardown dropped their buffered tails. Now: cancel them and let each worker's own seg_finished drain it (the `assembling` state), last one starts verification -- no cross-thread buffer access. - seg_head() let a 401 with credentials present abort before libcurl's resend; now it proceeds once and acts on the final status. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01HPPSGhiArbvQgwC2DNiURS
224 lines
9.3 KiB
C++
224 lines
9.3 KiB
C++
// vdm/task/download.hpp — the public download API: what DAEMON hands the engine and how
|
|
// the engine reports back. REVIEW SKETCH (stage 7 pre-work) — value types are final
|
|
// enough to build against; Engine/DownloadHandle bodies land in stage 8.
|
|
//
|
|
// See core/docs/engine-api-m1.md for the threading, lifetime, and pause/resume/cancel
|
|
// contract that goes with these signatures.
|
|
//
|
|
// This header compiles standalone.
|
|
|
|
#ifndef VDM_TASK_DOWNLOAD_HPP
|
|
#define VDM_TASK_DOWNLOAD_HPP
|
|
|
|
#include <chrono>
|
|
#include <cstdint>
|
|
#include <functional>
|
|
#include <memory>
|
|
#include <optional>
|
|
#include <string>
|
|
#include <vector>
|
|
|
|
#include "vdm/ids.hpp"
|
|
#include "vdm/net/http_types.hpp"
|
|
#include "vdm/net/probe.hpp"
|
|
#include "vdm/segment/segmenter.hpp"
|
|
#include "vdm/util/error.hpp"
|
|
#include "vdm/util/result.hpp"
|
|
|
|
namespace vdm {
|
|
class Engine; // owns and fills DownloadHandle (see vdm/engine.hpp)
|
|
} // namespace vdm
|
|
|
|
namespace vdm::task {
|
|
|
|
// The task control block. Opaque: defined only in the engine's translation unit. A handle
|
|
// holds a shared_ptr to one; the engine keeps its own copy so the task outlives a caller
|
|
// that drops its handle.
|
|
struct DownloadTaskState;
|
|
|
|
// --- input ---------------------------------------------------------------------------
|
|
|
|
struct Checksum {
|
|
enum class Algo { md5, sha1, sha256, sha512 }; // matches the wire Checksum set
|
|
Algo algo = Algo::sha256;
|
|
std::string hex; // lower-case, no separators
|
|
};
|
|
|
|
// Everything the engine needs to run ONE download. DAEMON has already run the rules
|
|
// engine, canonicalised the path, checked it against the allowed roots, resolved the
|
|
// filename, and created the parent directory — `save_path` is absolute and final and its
|
|
// directory exists. `<save_path>.veloxpart` and `<save_path>.veloxpart.meta` live beside
|
|
// it during the transfer; on success the part file is renamed in place. If the directory
|
|
// is missing at open time the task fails with Error::path_rejected.
|
|
struct DownloadSpec {
|
|
std::string url;
|
|
std::vector<std::string> mirrors; // alternative URLs for the same bytes
|
|
|
|
std::vector<net::HeaderField> headers; // the browser's, verbatim
|
|
std::vector<net::Cookie> cookies;
|
|
std::string referrer;
|
|
std::string user_agent;
|
|
|
|
std::string save_path; // absolute; the engine never canonicalises or root-checks
|
|
|
|
std::optional<std::uint32_t> segments; // requested 1..32; nullopt => engine default
|
|
std::optional<std::uint64_t> buffer_bytes; // requested per segment; nullopt => default
|
|
|
|
net::ProxyConfig proxy;
|
|
net::AuthConfig auth; // credentials known up front (e.g. from the Secret Service);
|
|
// leave scheme == none to be prompted on a 401/407
|
|
|
|
std::optional<Checksum> checksum; // verified during `verifying`; mismatch => failed
|
|
|
|
// DAEMON usually probed already for the File Info dialog. Pass it to skip a second
|
|
// probe; the engine still revalidates on resume. nullopt => the engine probes.
|
|
std::optional<net::ProbeResult> probe_hint;
|
|
|
|
bool allow_resume = true; // if a valid .veloxpart.meta sits beside save_path, resume
|
|
// from it; false starts fresh and overwrites
|
|
std::optional<long> max_retries; // per-segment; nullopt => engine default (10)
|
|
};
|
|
|
|
// --- lifecycle (the CORE-owned subset of the wire TaskState; ADR 0013 §1) -------------
|
|
|
|
enum class EngineState {
|
|
probing,
|
|
connecting,
|
|
downloading,
|
|
paused, // shared with DAEMON; entered by either side, idempotently
|
|
retry_wait, // the engine's own backoff timer
|
|
assembling, // no-op rename in M1; a real mux step for HLS/DASH (M4)
|
|
verifying, // checksum
|
|
complete, // terminal
|
|
failed, // terminal
|
|
cancelled, // terminal; always DAEMON- or user-initiated
|
|
};
|
|
|
|
[[nodiscard]] constexpr bool is_terminal(EngineState s) noexcept {
|
|
return s == EngineState::complete || s == EngineState::failed || s == EngineState::cancelled;
|
|
}
|
|
|
|
// --- progress ---------------------------------------------------------------------------
|
|
|
|
struct SegmentProgress {
|
|
std::uint32_t index = 0;
|
|
std::uint64_t start = 0;
|
|
std::uint64_t end = 0; // inclusive
|
|
std::uint64_t completed = 0;
|
|
std::uint64_t speed_bps = 0;
|
|
segment::SegState state = segment::SegState::idle;
|
|
};
|
|
|
|
struct Progress {
|
|
std::uint64_t downloaded = 0;
|
|
std::optional<std::uint64_t> total; // absent for a chunked source until it ends
|
|
std::uint64_t speed_bps = 0; // aggregate over the last window
|
|
std::optional<std::uint32_t> eta_seconds;
|
|
|
|
std::uint32_t effective_segments = 0; // slots the budget granted (held)
|
|
std::uint64_t effective_buffer_bytes = 0; // per segment, after the maxTotal clamp
|
|
std::vector<SegmentProgress> segments;
|
|
};
|
|
|
|
// --- interaction callbacks ----------------------------------------------------------
|
|
|
|
// A 401/407. The task has already auto-paused (state -> paused, error == auth_required).
|
|
// DAEMON collects credentials and calls handle.provide_auth().
|
|
struct AuthChallenge {
|
|
std::string host;
|
|
std::string realm;
|
|
enum class Scheme { basic, digest, ntlm, negotiate, unknown };
|
|
Scheme scheme = Scheme::unknown;
|
|
};
|
|
|
|
// The server's copy changed under us (a 200 where a 206 was expected, or an If-Range /
|
|
// ETag mismatch on resume — docs/04 §5), or the range metadata went stale (416). The
|
|
// task has auto-paused. DAEMON asks the user and calls handle.decide().
|
|
struct DecisionRequest {
|
|
enum class Kind { server_file_changed, range_metadata_stale };
|
|
Kind kind = Kind::server_file_changed;
|
|
std::string detail; // human-readable, for the dialog body
|
|
};
|
|
|
|
enum class Decision {
|
|
restart, // discard the partial file, download again from scratch
|
|
keep_partial, // trust what is on disk and continue (the user's risk)
|
|
abort, // give up: the task goes to `failed`
|
|
};
|
|
|
|
struct DownloadOutcome {
|
|
std::string final_path;
|
|
std::uint64_t bytes = 0;
|
|
std::optional<std::string> sha256_hex; // present when a checksum was requested/derived
|
|
std::chrono::milliseconds elapsed{0};
|
|
};
|
|
|
|
// All callbacks are optional. See core/docs/engine-api-m1.md for the rules; in short:
|
|
// they arrive on an engine thread, are serialised per task, must not block, and must not
|
|
// re-enter THIS task's handle synchronously.
|
|
struct DownloadCallbacks {
|
|
// Coalesced to <= 4 Hz per task (matches the wire event.task.progress cadence).
|
|
std::function<void(const Progress &)> on_progress;
|
|
|
|
// Every lifecycle transition, including the auto-pauses above (to == paused with a
|
|
// populated ErrorInfo) and terminals.
|
|
std::function<void(EngineState from, EngineState to, const std::optional<ErrorInfo> &)>
|
|
on_state;
|
|
|
|
std::function<void(const AuthChallenge &)> on_auth_required;
|
|
std::function<void(const DecisionRequest &)> on_decision_needed;
|
|
|
|
// Fired exactly once, last. Success carries the outcome; failure carries the mapped
|
|
// ErrorInfo. After it returns the engine makes no further callbacks for this task and
|
|
// the handle's control methods become no-ops.
|
|
std::function<void(Result<DownloadOutcome>)> on_finished;
|
|
};
|
|
|
|
// --- the handle -------------------------------------------------------------------------
|
|
|
|
// Copyable (shared state). Every method is safe to call from any thread; each posts to
|
|
// the engine and returns immediately. Dropping the last handle does NOT cancel the task —
|
|
// call cancel() for that. Bodies land in stage 8.
|
|
class DownloadHandle {
|
|
public:
|
|
DownloadHandle() = default;
|
|
// The engine builds handles; `DownloadTaskState` is incomplete everywhere else, so
|
|
// this is effectively engine-only without a friend declaration.
|
|
explicit DownloadHandle(std::shared_ptr<DownloadTaskState> s) : state_(std::move(s)) {}
|
|
|
|
[[nodiscard]] TaskId id() const noexcept;
|
|
[[nodiscard]] bool valid() const noexcept { return static_cast<bool>(state_); }
|
|
|
|
// Idempotent. pause() on an already-paused or terminal task is a no-op (no error);
|
|
// likewise resume() on a task that is not paused. The resulting state is observed via
|
|
// on_state / this->state(), never a return value (ADR 0013 §2).
|
|
void pause();
|
|
void resume();
|
|
// Idempotent, terminal. discard_partial also removes the .veloxpart[.meta] files.
|
|
// Always fires on_state(_, cancelled, nullopt) then on_finished(Err{Error::canceled}),
|
|
// in that order. `download.cancel` == cancel(false); `download.remove` == cancel(true)
|
|
// (plus DAEMON's own row/file cleanup).
|
|
void cancel(bool discard_partial = false);
|
|
|
|
// Only act while the task is awaiting the matching input (auto-paused for auth /
|
|
// decision); otherwise a no-op. `remember` asks DAEMON to persist to the Secret
|
|
// Service — the engine never stores a credential.
|
|
void provide_auth(std::string username, std::string password, bool remember);
|
|
void decide(Decision d);
|
|
|
|
// IDM's "Refresh Download Address": swap the URL (e.g. a fresh signed URL) on a live
|
|
// or paused task without losing progress. Empty `headers` keeps the current ones.
|
|
void refresh_url(std::string url, std::vector<net::HeaderField> headers = {});
|
|
|
|
// Synchronous snapshots — cheap, lock-guarded, safe any time.
|
|
[[nodiscard]] EngineState state() const;
|
|
[[nodiscard]] Progress progress() const;
|
|
|
|
private:
|
|
std::shared_ptr<DownloadTaskState> state_;
|
|
};
|
|
|
|
} // namespace vdm::task
|
|
|
|
#endif // VDM_TASK_DOWNLOAD_HPP
|