daemon: D3's remainder — rules.*, queue.reorder, schedule.*, limiter.*, download.update/refreshUrl

rules.list/rules.upsert: new store/rules.{hpp,cpp} (list in priority order; apply()
upserts+removes atomically, generating an id when absent, per the schema's own "never
leaves the table in a half-valid state"). Found and fixed along the way: migration
0001's rules table had no column for Rule.name at all — every rules.list/.upsert call
failed outright ("no such column: name"), unit tests included, since :memory: migrates
through the same path. Migration 0004 adds it.

queue.reorder: new store::Queues::reorder — taskIds must be an exact permutation of
the queue's current membership (compared as sorted sets) or nothing is written and
-32602 names the queue; a valid permutation rewrites every member's queue_position in
one transaction.

schedule.get/schedule.set: a thin wrapper over the queues.schedule column (already
read since D3b, never independently settable). nextRunAt is deliberately left unset —
computing it needs the same local-time, DST-aware window logic
sched/schedule_window.hpp's window_open() only has half of; called out rather than
approximated, and the field is optional.

limiter.get/limiter.set: backed by the same downloads.speedLimitEnabled/
downloads.speedLimitBps settings keys D9 already wired — one bag of truth, not two.
The new part is reaching the engine: EnginePort/TaskActionPort gain
set_global_speed_limit(bps) (0 = unlimited, TokenBucket's own convention), wired to
Engine::rate_limiter().set_global_limit(). Pushed live on every limiter.set *and* on
Scheduler::reload_config() so a limit from a previous run isn't silently unlimited
again after a restart. applyToRunning is accepted but has no lever to pull
differently — a single shared global bucket has no "next task only" variant. Also
found, not chased further: the schema's "globalBps:0 with enabled:true means 'stop
everything'" is the opposite of what TokenBucket does with rate_bps==0 (unlimited) —
a real discrepancy, but the schema says the GUI must not offer that combination.

download.update: "moving saveDir or filename moves the file on disk in the same
operation" — resolved and root-checked like download.add's destination, then the
.veloxpart/.veloxpart.meta pair (or the finished file, if complete) is moved via
rename, falling back to copy+remove across filesystems, only when the resolved
location actually differs. categoryId/queueId(appended to the new queue's run
order)/description/segments/bufferBytes/checksum apply through new
store::Tasks::apply_update.

download.refreshUrl: same async server-layer special-case as download.probe (a real
network round trip, same 30s deadline). Re-probes, flags contentChanged only when
size or validator are both known and actually differ, persists the new URL and probe
result, and swaps the URL on a live engine handle via a newly-widened
EnginePort::refresh_url (now takes headers too, matching DownloadHandle's real
signature — the seam had silently dropped them).

Found and documented, not fixed: the generated parser collapses "field absent" and
"field explicitly null" to the same nullopt for every optional<T> patch field
(DownloadUpdateParamsPatch, Settings) — both schemas document "an explicit null
clears the field" but neither handler can act on it because the wire distinction is
already gone by the time either sees the parsed struct. A generator-level gap
(PROTO's), not something to hand-route around locally.

Verified against real veloxd + tools/testserver: rules create/list, limiter.set
takes effect and reads back, schedule.set/get round-trips, queue.reorder against real
membership (and rejects a non-permutation), download.update renames+recategorizes a
task, download.refreshUrl swaps a paused task's URL and reports contentChanged
correctly. Full ctest: 55/55 (excluding the pre-existing, unrelated conformance
failure noted two commits back).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GRDjHGgpYmMoPE2UFbe7pP
This commit is contained in:
2026-09-12 14:26:18 +04:00
co-authored by Claude Sonnet 5
parent 6632b75099
commit 4f6c0cc9d2
20 changed files with 948 additions and 19 deletions
+233 -14
View File
@@ -819,8 +819,120 @@ VeloxDispatcher::on_download_start(const proto::DownloadStartParams& params) {
return result;
}
proto::HandlerResult<proto::TaskSummary>
VeloxDispatcher::on_download_update(const proto::DownloadUpdateParams&) {
return not_implemented<proto::TaskSummary>("download.update");
VeloxDispatcher::on_download_update(const proto::DownloadUpdateParams& params) {
store::Tasks tasks(db_);
auto got = tasks.get(params.taskId);
if (!got)
return std::unexpected(proto::HandlerError{proto::ErrorCode::InternalError,
"download.update: " + got.error().message});
if (!got->has_value())
return std::unexpected(proto::HandlerError{proto::ErrorCode::TaskNotFound, "no such task",
nlohmann::json{{"taskId", params.taskId}}});
const store::TaskRow& row = **got;
const auto& patch = params.patch;
if (patch.segments && (*patch.segments < 1 || *patch.segments > 32)) {
return std::unexpected(proto::HandlerError{
proto::ErrorCode::InvalidParams, "segments must be between 1 and 32",
nlohmann::json{{"value", *patch.segments}}});
}
if (patch.bufferBytes && (*patch.bufferBytes < 65536 || *patch.bufferBytes > 16777216)) {
return std::unexpected(proto::HandlerError{
proto::ErrorCode::InvalidParams, "bufferBytes must be between 65536 and 16777216",
nlohmann::json{{"value", *patch.bufferBytes}}});
}
store::Tasks::UpdatePatch update;
// "Moving saveDir or filename moves the file on disk in the same operation" (the
// schema's own words). Resolved and root-checked exactly like download.add's own
// destination; only actually touches the filesystem if the resolved location differs
// from where the task already is.
if (patch.filename || patch.saveDir) {
store::Settings settings(db_);
std::string dir = expand_tilde(patch.saveDir.value_or(row.save_dir));
std::string leaf = patch.filename.value_or(row.filename);
std::vector<std::string> roots;
for (const auto& r : settings.get_string_array("saveTo.allowedRoots"))
if (auto c = fs::canonicalize_root(r)) roots.push_back(*c);
auto target = fs::resolve_target(dir, leaf, roots);
if (!target) {
return std::unexpected(proto::HandlerError{
proto::ErrorCode::InvalidPath, target.error().message,
nlohmann::json{{"path", patch.saveDir.value_or(dir)}}});
}
if (target->dir != row.save_dir || target->leaf != row.filename) {
namespace fsn = std::filesystem;
const std::string old_base = row.save_dir + "/" + row.filename;
const std::string new_base = target->dir + "/" + target->leaf;
// move_one: rename, falling back to copy+remove across filesystems (EXDEV).
// Missing source (nothing to move at this path yet) is not an error.
auto move_one = [](const std::string& from, const std::string& to) -> bool {
std::error_code ec;
if (!fsn::exists(from, ec)) return true;
fsn::rename(from, to, ec);
if (!ec) return true;
ec.clear();
fsn::copy_file(from, to, fsn::copy_options::overwrite_existing, ec);
if (ec) return false;
fsn::remove(from, ec);
return true;
};
const bool moved = row.state == "complete"
? move_one(old_base, new_base)
: (move_one(old_base + ".veloxpart", new_base + ".veloxpart") &&
move_one(old_base + ".veloxpart.meta", new_base + ".veloxpart.meta"));
if (!moved) {
return std::unexpected(proto::HandlerError{
proto::ErrorCode::InternalError, "could not move the file to its new location",
nlohmann::json{{"path", new_base}}});
}
update.save_dir = target->dir;
update.filename = target->leaf;
}
}
if (patch.categoryId) update.category_id = patch.categoryId;
if (patch.queueId) {
update.queue_id = patch.queueId;
// Appended to the end of the new queue's run order.
auto page = tasks.list(
[&] {
proto::TaskFilter f;
f.queueId = *patch.queueId;
return f;
}(),
std::nullopt, 0, 10000);
std::int64_t next_pos = 0;
if (page)
for (const auto& r : page->rows)
if (r.queue_position) next_pos = std::max<std::int64_t>(next_pos, *r.queue_position + 1);
update.queue_position = next_pos;
}
if (patch.description) update.description = patch.description;
if (patch.segments) update.req_segments = patch.segments;
if (patch.bufferBytes) update.req_buffer_bytes = patch.bufferBytes;
if (patch.checksum) {
update.checksum_algo = std::string(proto::to_string(patch.checksum->algorithm));
update.checksum_value = patch.checksum->value;
}
if (auto r = tasks.apply_update(params.taskId, update); !r) {
return std::unexpected(proto::HandlerError{proto::ErrorCode::InternalError,
"download.update: " + r.error().message});
}
auto after = tasks.get(params.taskId);
if (!after || !after->has_value()) {
return std::unexpected(
proto::HandlerError{proto::ErrorCode::InternalError, "download.update: row vanished"});
}
if (on_mutation_) on_mutation_();
return store::to_summary(**after);
}
proto::HandlerResult<proto::GrabberHarvestResult>
VeloxDispatcher::on_grabber_harvest(const proto::GrabberHarvestParams&) {
@@ -835,10 +947,24 @@ VeloxDispatcher::on_grabber_status(const proto::GrabberStatusParams&) {
return not_implemented<proto::GrabberStatusResult>("grabber.status");
}
proto::HandlerResult<proto::Limiter> VeloxDispatcher::on_limiter_get(const proto::LimiterGetParams&) {
return not_implemented<proto::Limiter>("limiter.get");
store::Settings settings(db_);
proto::Limiter r;
r.enabled = settings.get_bool("downloads.speedLimitEnabled");
r.globalBps = settings.get_int("downloads.speedLimitBps");
return r;
}
proto::HandlerResult<proto::Limiter> VeloxDispatcher::on_limiter_set(const proto::Limiter&) {
return not_implemented<proto::Limiter>("limiter.set");
proto::HandlerResult<proto::Limiter> VeloxDispatcher::on_limiter_set(const proto::Limiter& params) {
store::Settings settings(db_);
(void)settings.set_raw("downloads.speedLimitEnabled", params.enabled ? "true" : "false");
(void)settings.set_raw("downloads.speedLimitBps", std::to_string(params.globalBps));
// The bucket's own convention: 0 == unlimited. `enabled: false` means "no limit"
// regardless of what globalBps happens to hold (the GUI is expected to keep it around
// for when the user re-enables it, not to reset it to 0).
if (actions_) actions_->set_global_speed_limit(params.enabled ? static_cast<std::uint64_t>(params.globalBps) : 0);
if (on_mutation_) on_mutation_();
return params;
}
proto::HandlerResult<proto::MediaAddVariantResult>
VeloxDispatcher::on_media_addVariant(const proto::MediaAddVariantParams&) {
@@ -860,8 +986,33 @@ VeloxDispatcher::on_queue_list(const proto::QueueListParams&) {
return r;
}
proto::HandlerResult<proto::QueueReorderResult>
VeloxDispatcher::on_queue_reorder(const proto::QueueReorderParams&) {
return not_implemented<proto::QueueReorderResult>("queue.reorder");
VeloxDispatcher::on_queue_reorder(const proto::QueueReorderParams& params) {
store::Queues queues(db_);
auto exists = queues.get(params.queueId);
if (!exists)
return std::unexpected(proto::HandlerError{proto::ErrorCode::InternalError,
"queue.reorder: " + exists.error().message});
if (!exists->has_value())
return std::unexpected(proto::HandlerError{
proto::ErrorCode::InvalidParams, "no such queue",
nlohmann::json{{"queueId", params.queueId}}});
auto applied = queues.reorder(params.queueId, params.taskIds);
if (!applied)
return std::unexpected(proto::HandlerError{proto::ErrorCode::InternalError,
"queue.reorder: " + applied.error().message});
if (!*applied) {
return std::unexpected(proto::HandlerError{
proto::ErrorCode::InvalidParams,
"taskIds must be an exact permutation of the queue's current membership",
nlohmann::json{{"queueId", params.queueId}}});
}
auto after = queues.get(params.queueId);
proto::QueueReorderResult r;
if (after && after->has_value()) r.queue = **after;
if (on_mutation_) on_mutation_();
return r;
}
proto::HandlerResult<proto::QueueStartResult>
VeloxDispatcher::on_queue_start(const proto::QueueStartParams& params) {
@@ -926,19 +1077,87 @@ VeloxDispatcher::on_queue_upsert(const proto::QueueUpsertParams& params) {
}
proto::HandlerResult<proto::RulesListResult>
VeloxDispatcher::on_rules_list(const proto::RulesListParams&) {
return not_implemented<proto::RulesListResult>("rules.list");
store::Rules rules(db_);
auto items = rules.list();
if (!items)
return std::unexpected(proto::HandlerError{proto::ErrorCode::InternalError,
"rules.list: " + items.error().message});
proto::RulesListResult r;
r.items = std::move(*items);
return r;
}
proto::HandlerResult<proto::RulesUpsertResult>
VeloxDispatcher::on_rules_upsert(const proto::RulesUpsertParams&) {
return not_implemented<proto::RulesUpsertResult>("rules.upsert");
VeloxDispatcher::on_rules_upsert(const proto::RulesUpsertParams& params) {
store::Rules rules(db_);
auto items = rules.apply(params.upsert, params.remove.value_or(std::vector<std::string>{}));
if (!items)
return std::unexpected(proto::HandlerError{proto::ErrorCode::InternalError,
"rules.upsert: " + items.error().message});
proto::RulesUpsertResult r;
r.items = std::move(*items);
return r;
}
proto::HandlerResult<proto::ScheduleGetResult>
VeloxDispatcher::on_schedule_get(const proto::ScheduleGetParams&) {
return not_implemented<proto::ScheduleGetResult>("schedule.get");
VeloxDispatcher::on_schedule_get(const proto::ScheduleGetParams& params) {
store::Queues queues(db_);
proto::ScheduleGetResult r;
if (params.queueId) {
auto q = queues.get(*params.queueId);
if (!q)
return std::unexpected(proto::HandlerError{proto::ErrorCode::InternalError,
"schedule.get: " + q.error().message});
if (q->has_value()) {
proto::ScheduleGetResultItemsItem item;
item.queueId = (*q)->queueId;
item.schedule = (*q)->schedule;
r.items.push_back(std::move(item));
}
// No such queue: an empty items list rather than an error — same "the caller asked
// about something specific and there's nothing to say about it" shape download.get
// uses -32010 for, but schedule.get's own x-errors lists only -32003, so an empty
// result (not "every schedule", just this one queue's, which doesn't exist) is the
// faithful answer within what the schema actually allows.
return r;
}
auto all = queues.list();
if (!all)
return std::unexpected(proto::HandlerError{proto::ErrorCode::InternalError,
"schedule.get: " + all.error().message});
for (const auto& q : *all) {
proto::ScheduleGetResultItemsItem item;
item.queueId = q.queueId;
item.schedule = q.schedule;
r.items.push_back(std::move(item));
}
return r;
}
proto::HandlerResult<proto::ScheduleSetResult>
VeloxDispatcher::on_schedule_set(const proto::ScheduleSetParams&) {
return not_implemented<proto::ScheduleSetResult>("schedule.set");
VeloxDispatcher::on_schedule_set(const proto::ScheduleSetParams& params) {
store::Queues queues(db_);
auto exists = queues.get(params.queueId);
if (!exists)
return std::unexpected(proto::HandlerError{proto::ErrorCode::InternalError,
"schedule.set: " + exists.error().message});
if (!exists->has_value())
return std::unexpected(proto::HandlerError{
proto::ErrorCode::InvalidParams, "no such queue",
nlohmann::json{{"queueId", params.queueId}}});
auto applied = queues.set_schedule(params.queueId, params.schedule);
if (!applied)
return std::unexpected(proto::HandlerError{proto::ErrorCode::InternalError,
"schedule.set: " + applied.error().message});
proto::ScheduleSetResult r;
r.queueId = params.queueId;
r.schedule = params.schedule;
// nextRunAt is left unset: computing it correctly needs the same local-time,
// DST-aware window logic sched/schedule_window.hpp's window_open() already has half
// of (open-right-now, not next-transition) — worth building as its own pass rather
// than an approximate guess here. The field is optional; nullopt is a legal answer.
if (on_mutation_) on_mutation_();
return r;
}
proto::HandlerResult<proto::SettingsGetResult>
VeloxDispatcher::on_settings_get(const proto::SettingsGetParams& params) {
+17
View File
@@ -67,6 +67,19 @@ public:
std::function<void(velox::proto::HandlerResult<velox::proto::DownloadProbeResult>)>
done) = 0;
// download.refreshUrl ("IDM's 'Refresh Download Address'"): same async reasoning and
// the same server-layer special-case as probe_now — a real network round trip, up to
// the schema's own 30s x-deadlineMs. Re-probes the new URL, compares size/validator
// against what the task already has on record (contentChanged), persists the new URL
// and probe result, and — if the task holds a live engine handle — swaps its URL
// in-flight without losing progress.
virtual void refresh_url(
const std::string& wire_id, const std::string& url,
const std::optional<velox::proto::Headers>& headers,
const std::optional<std::vector<velox::proto::Cookie>>& cookies,
std::function<void(velox::proto::HandlerResult<velox::proto::DownloadRefreshUrlResult>)>
done) = 0;
// settings.set of a connection.* key: re-read connection.maxConcurrentDownloads /
// maxActiveSegments and the per-host cap table, push the new caps to the engine and
// the governor (Scheduler::reload_config's own doc comment names this exact trigger).
@@ -74,6 +87,10 @@ public:
// sched::Scheduler's own already-public reload_config() (returns DbResult<void>,
// consumed by main.cpp and a unit test — kept as-is rather than reshaped to fit here).
virtual void apply_settings_reload() = 0;
// limiter.set: pushed straight to the engine's shared global token bucket. 0 means
// unlimited (the bucket's own convention).
virtual void set_global_speed_limit(std::uint64_t bps) = 0;
};
} // namespace velox::daemon::rpc
+35
View File
@@ -207,6 +207,10 @@ void UdsServer::handle_line(Conn& c, const std::string& line) {
handle_download_probe(c, req);
return;
}
if (method == "download.refreshUrl") {
handle_download_refreshUrl(c, req);
return;
}
// Everything else: the generated router. It returns a null json for a notification
// that needs no reply.
@@ -244,6 +248,37 @@ void UdsServer::handle_download_probe(Conn& c, const json& request) {
});
}
void UdsServer::handle_download_refreshUrl(Conn& c, const json& request) {
const json id = request.contains("id") ? request.at("id") : json(nullptr);
const json params_json = request.contains("params") ? request.at("params") : json::object();
auto parsed = proto::parse<proto::DownloadRefreshUrlParams>(params_json, "params");
if (!parsed) {
queue_reply(c, rpc_error(id, proto::ErrorCode::InvalidParams, parsed.error().message,
json{{"path", parsed.error().path}}));
return;
}
if (!actions_) {
queue_reply(c, rpc_error(id, proto::ErrorCode::InternalError,
"not implemented in this build: download.refreshUrl"));
return;
}
const int fd = c.fd;
actions_->refresh_url(
parsed->taskId, parsed->url, parsed->headers, parsed->cookies,
[this, fd, id](proto::HandlerResult<proto::DownloadRefreshUrlResult> r) {
auto it = conns_.find(fd);
if (it == conns_.end()) return; // client gone while the probe was outstanding
if (r) {
queue_reply(*it->second, proto::make_result(id, *r));
} else {
queue_reply(*it->second,
rpc_error(id, r.error().code, r.error().message, r.error().data));
}
});
}
bool UdsServer::handle_session_method(Conn& c, const std::string& method, const json& request,
json& reply) {
const json id = request.contains("id") ? request.at("id") : json(nullptr);
+3
View File
@@ -76,6 +76,9 @@ private:
// Queues the reply itself, later, when actions_->probe_now()'s callback fires; does
// nothing if the connection is gone by then (client disconnected mid-probe).
void handle_download_probe(Conn& c, const nlohmann::json& request);
// Same reasoning as handle_download_probe — a real network round trip, same 30s
// x-deadlineMs.
void handle_download_refreshUrl(Conn& c, const nlohmann::json& request);
void queue_reply(Conn& c, const nlohmann::json& reply);
void flush(Conn& c);
+35
View File
@@ -267,6 +267,10 @@ void WsServer::handle_rpc(Conn& c, const std::string& text) {
handle_download_probe(c, req);
return;
}
if (method == "download.refreshUrl") {
handle_download_refreshUrl(c, req);
return;
}
json reply = proto::dispatch(dispatcher_, proto::Transport::Ws, req);
if (!reply.is_null()) send_text(c, reply);
@@ -302,6 +306,37 @@ void WsServer::handle_download_probe(Conn& c, const json& request) {
});
}
void WsServer::handle_download_refreshUrl(Conn& c, const json& request) {
const json id = request.contains("id") ? request.at("id") : json(nullptr);
const json params_json = request.contains("params") ? request.at("params") : json::object();
auto parsed = proto::parse<proto::DownloadRefreshUrlParams>(params_json, "params");
if (!parsed) {
send_text(c, rpc_error(id, proto::ErrorCode::InvalidParams, parsed.error().message,
json{{"path", parsed.error().path}}));
return;
}
if (!actions_) {
send_text(c, rpc_error(id, proto::ErrorCode::InternalError,
"not implemented in this build: download.refreshUrl"));
return;
}
const int fd = c.fd;
actions_->refresh_url(
parsed->taskId, parsed->url, parsed->headers, parsed->cookies,
[this, fd, id](proto::HandlerResult<proto::DownloadRefreshUrlResult> r) {
auto it = conns_.find(fd);
if (it == conns_.end()) return; // client gone while the probe was outstanding
if (r) {
send_text(*it->second, proto::make_result(id, *r));
} else {
send_text(*it->second,
rpc_error(id, r.error().code, r.error().message, r.error().data));
}
});
}
bool WsServer::handle_session_ws(Conn& c, const std::string& method, const json& request,
json& reply) {
const json id = request.contains("id") ? request.at("id") : json(nullptr);
+1
View File
@@ -81,6 +81,7 @@ private:
// See UdsServer::handle_download_probe — same reasoning, same pattern, duplicated per
// transport because each owns its own Conn/send mechanics.
void handle_download_probe(Conn& c, const nlohmann::json& request);
void handle_download_refreshUrl(Conn& c, const nlohmann::json& request);
void send_text(Conn& c, const nlohmann::json& value);
void send_frame(Conn& c, WsOpcode op, std::string_view payload);
+8 -1
View File
@@ -46,7 +46,8 @@ public:
virtual void provide_auth(vdm::TaskId, const std::string& username,
const std::string& password, bool remember) = 0;
virtual void decide(vdm::TaskId, vdm::task::Decision) = 0;
virtual void refresh_url(vdm::TaskId, const std::string& url) = 0;
virtual void refresh_url(vdm::TaskId, const std::string& url,
const std::vector<vdm::net::HeaderField>& headers = {}) = 0;
// The daemon is done with this task (it went terminal). Drop the handle. Idempotent.
virtual void release(vdm::TaskId) = 0;
@@ -59,6 +60,12 @@ public:
virtual void set_task_order(const std::vector<vdm::TaskId>& order) = 0;
virtual void set_max_active_segments(std::uint32_t n) = 0;
virtual void set_host_segment_cap(const std::string& host, std::uint32_t cap) = 0;
// limiter.set: 0 means unlimited (vdm::rate::TokenBucket's own convention), applied
// across every active transfer immediately — there is no "next task only" variant for
// a single shared global bucket, so `applyToRunning` on the wire has nothing to select
// between; it is accepted for schema compliance and always behaves as if true.
virtual void set_global_speed_limit(std::uint64_t bps) = 0;
};
} // namespace velox::daemon::sched
+6 -2
View File
@@ -45,8 +45,9 @@ public:
void decide(vdm::TaskId id, vdm::task::Decision d) override {
if (auto* h = find(id)) h->decide(d);
}
void refresh_url(vdm::TaskId id, const std::string& url) override {
if (auto* h = find(id)) h->refresh_url(url);
void refresh_url(vdm::TaskId id, const std::string& url,
const std::vector<vdm::net::HeaderField>& headers) override {
if (auto* h = find(id)) h->refresh_url(url, headers);
}
void release(vdm::TaskId id) override { handles_.erase(id); }
std::optional<vdm::task::Progress> progress(vdm::TaskId id) const override {
@@ -64,6 +65,9 @@ public:
void set_host_segment_cap(const std::string& host, std::uint32_t cap) override {
engine_.segment_budget().set_host_segment_cap(host, cap);
}
void set_global_speed_limit(std::uint64_t bps) override {
engine_.rate_limiter().set_global_limit(bps);
}
private:
vdm::task::DownloadHandle* find(vdm::TaskId id) {
+13 -1
View File
@@ -61,7 +61,16 @@ public:
void cancel(vdm::TaskId id, bool discard) override { cancelled.emplace_back(id, discard); }
void provide_auth(vdm::TaskId, const std::string&, const std::string&, bool) override {}
void decide(vdm::TaskId, vdm::task::Decision) override {}
void refresh_url(vdm::TaskId, const std::string&) override {}
void refresh_url(vdm::TaskId id, const std::string& url,
const std::vector<vdm::net::HeaderField>& headers) override {
refreshed_urls.emplace_back(id, url, headers);
}
struct RefreshCall {
vdm::TaskId id;
std::string url;
std::vector<vdm::net::HeaderField> headers;
};
std::vector<RefreshCall> refreshed_urls;
void release(vdm::TaskId id) override { released.push_back(id); }
std::optional<vdm::task::Progress> progress(vdm::TaskId id) const override {
auto it = fake_progress.find(id.value);
@@ -75,6 +84,9 @@ public:
void set_host_segment_cap(const std::string& h, std::uint32_t c) override {
host_caps.emplace_back(h, c);
}
void set_global_speed_limit(std::uint64_t bps) override { global_speed_limits.push_back(bps); }
std::vector<std::uint64_t> global_speed_limits;
const std::vector<vdm::TaskId>& last_order() const { return orders.back(); }
+83
View File
@@ -183,6 +183,16 @@ store::DbResult<void> Scheduler::reload_config() {
governor_.set_config(cfg);
engine_.set_max_active_segments(
static_cast<std::uint32_t>(std::max<std::int64_t>(cfg.max_active_segments, 1)));
// The global speed limit persists across a restart the same as any other setting, but
// (unlike connection.* above) nothing re-derives it into engine state on its own —
// limiter.set is the only other place that calls set_global_speed_limit, and that only
// fires on an explicit RPC in a running daemon. Push it here too so a limit set in a
// previous run is not silently unlimited again after a restart.
const bool limit_enabled = settings.get_bool("downloads.speedLimitEnabled");
const std::int64_t limit_bps = settings.get_int("downloads.speedLimitBps");
engine_.set_global_speed_limit(limit_enabled ? static_cast<std::uint64_t>(std::max<std::int64_t>(limit_bps, 0))
: 0);
return {};
}
@@ -634,4 +644,77 @@ void Scheduler::probe_now(
});
}
void Scheduler::refresh_url(
const std::string& wire_id, const std::string& url,
const std::optional<proto::Headers>& headers,
const std::optional<std::vector<proto::Cookie>>& cookies,
std::function<void(proto::HandlerResult<proto::DownloadRefreshUrlResult>)> done) {
auto got = store::Tasks(db_).get(wire_id);
if (!got || !got->has_value()) {
done(std::unexpected(proto::HandlerError{proto::ErrorCode::TaskNotFound, "no such task",
nlohmann::json{{"taskId", wire_id}}}));
return;
}
const store::TaskRow row = **got; // copied: read again by the async callback below
vdm::net::ProbeRequest req;
req.url = url;
if (headers)
for (const auto& [k, v] : *headers) req.headers.push_back({k, v});
if (cookies)
for (const auto& c : *cookies) req.cookies.push_back({c.name, c.value});
engine_.probe(req, [this, wire_id, row, url, headers, done](vdm::Result<vdm::net::ProbeResult> pr) {
deps_.post_to_loop([this, wire_id, row, url, headers, done, pr]() {
if (!pr) {
nlohmann::json data;
if (pr.error().http_status != 0) data["httpStatus"] = pr.error().http_status;
done(std::unexpected(proto::HandlerError{proto::ErrorCode::ProbeFailed,
pr.error().context, data}));
return;
}
// "if they do not [match], it says so rather than silently restarting" (the
// schema's own words) — comparison only fires when both sides actually have a
// value; an unknown size/validator on either end is not itself a mismatch.
bool content_changed = false;
if (row.size_bytes && pr->total_size &&
*row.size_bytes != static_cast<std::int64_t>(*pr->total_size))
content_changed = true;
if (row.etag && !row.etag->empty() && !pr->etag.empty() && *row.etag != pr->etag)
content_changed = true;
if (row.last_modified && !row.last_modified->empty() && !pr->last_modified.empty() &&
*row.last_modified != pr->last_modified)
content_changed = true;
store::Tasks tasks(db_);
store::Tasks::ProbeFields fields;
if (pr->total_size) fields.size_bytes = static_cast<std::int64_t>(*pr->total_size);
fields.resumable = pr->resumable;
if (!pr->etag.empty()) fields.etag = pr->etag;
if (!pr->last_modified.empty()) fields.last_modified = pr->last_modified;
if (!pr->mime.empty()) fields.content_type = pr->mime;
const std::string effective = pr->effective_url.empty() ? url : pr->effective_url;
fields.effective_url = effective;
(void)tasks.set_probe_result(wire_id, fields);
(void)tasks.set_url(wire_id, url);
if (auto eid = engine_id_of(wire_id)) {
std::vector<vdm::net::HeaderField> hdrs;
if (headers)
for (const auto& [k, v] : *headers) hdrs.push_back({k, v});
engine_.refresh_url(*eid, url, hdrs);
}
proto::DownloadRefreshUrlResult r;
r.ok = true;
r.resumable = pr->resumable;
r.contentChanged = content_changed;
if (pr->total_size) r.sizeBytes = static_cast<std::int64_t>(*pr->total_size);
r.effectiveUrl = effective;
done(r);
});
});
}
} // namespace velox::daemon::sched
+9
View File
@@ -141,6 +141,8 @@ public:
// port, which has no caller that wants the DbError).
void apply_settings_reload() override { (void)reload_config(); }
void set_global_speed_limit(std::uint64_t bps) override { engine_.set_global_speed_limit(bps); }
// rpc::TaskActionPort. Builds a vdm::net::ProbeRequest from `params`, runs it on the
// engine's probe pool (outside the segment budget, ADR 0011 §5), and converts the
// result back to proto terms — including the suggestedCategoryId/-SaveDir guess (a
@@ -153,6 +155,13 @@ public:
std::function<void(velox::proto::HandlerResult<velox::proto::DownloadProbeResult>)>
done) override;
void refresh_url(
const std::string& wire_id, const std::string& url,
const std::optional<velox::proto::Headers>& headers,
const std::optional<std::vector<velox::proto::Cookie>>& cookies,
std::function<void(velox::proto::HandlerResult<velox::proto::DownloadRefreshUrlResult>)>
done) override;
// Diagnostics / tests.
std::optional<std::string> wire_id_of(vdm::TaskId id) const;
std::optional<vdm::TaskId> engine_id_of(const std::string& wire_id) const;
@@ -0,0 +1,10 @@
-- Migration 0004 — rules gets a name column.
--
-- 0001's rules table had no column for Rule.name (Rule.schema.json's own optional,
-- maxLength-64 label field) — store/rules.cpp discovered this the hard way building
-- rules.list/rules.upsert: every rules.list call failed outright ("no such column:
-- name") because the SELECT it needs to project onto proto::Rule names a column that was
-- never there. A plain ALTER TABLE ADD COLUMN suffices here (no CHECK constraint to
-- rebuild around, unlike 0002/0003).
ALTER TABLE rules ADD COLUMN name TEXT;
+51
View File
@@ -2,6 +2,7 @@
#include <sqlite3.h>
#include <algorithm>
#include <cstdio>
#include <random>
#include <string>
@@ -85,6 +86,56 @@ DbResult<bool> Queues::set_state(std::string_view queue_id, std::string_view sta
return sqlite3_changes(db_.raw()) > 0;
}
DbResult<bool> Queues::set_schedule(std::string_view queue_id,
const std::optional<proto::Schedule>& schedule) {
auto st = db_.prepare("UPDATE queues SET schedule = ?2 WHERE queue_id = ?1");
if (!st) return std::unexpected(st.error());
if (auto b = st->bind(1, queue_id); !b) return std::unexpected(b.error());
if (auto r = schedule ? st->bind(2, std::string_view(nlohmann::json(*schedule).dump()))
: st->bind_null(2);
!r)
return std::unexpected(r.error());
if (auto r = st->step(); !r) return std::unexpected(r.error());
return sqlite3_changes(db_.raw()) > 0;
}
DbResult<bool> Queues::reorder(std::string_view queue_id, const std::vector<std::string>& task_ids) {
std::vector<std::string> current;
{
auto st = db_.prepare(
"SELECT task_id FROM tasks WHERE queue_id = ?1 ORDER BY queue_position");
if (!st) return std::unexpected(st.error());
if (auto b = st->bind(1, queue_id); !b) return std::unexpected(b.error());
for (;;) {
auto row = st->step();
if (!row) return std::unexpected(row.error());
if (!*row) break;
current.push_back(st->column_text(0));
}
}
// Exact permutation: same size, same members, order aside.
std::vector<std::string> a = current, b = task_ids;
std::sort(a.begin(), a.end());
std::sort(b.begin(), b.end());
if (a != b) return false;
auto txn = db_.transaction([&]() -> DbResult<void> {
for (std::size_t i = 0; i < task_ids.size(); ++i) {
auto st = db_.prepare("UPDATE tasks SET queue_position = ?2 WHERE task_id = ?1");
if (!st) return std::unexpected(st.error());
if (auto bd = st->bind(1, std::string_view(task_ids[i])); !bd)
return std::unexpected(bd.error());
if (auto bd = st->bind(2, static_cast<std::int64_t>(i)); !bd)
return std::unexpected(bd.error());
if (auto r = st->step(); !r) return std::unexpected(r.error());
}
return {};
});
if (!txn) return std::unexpected(txn.error());
return true;
}
DbResult<proto::Queue> Queues::upsert(proto::Queue queue) {
if (queue.queueId.empty()) {
std::random_device rd;
+12
View File
@@ -27,6 +27,18 @@ public:
// id doesn't exist.
DbResult<bool> set_state(std::string_view queue_id, std::string_view state);
// schedule.set: nullopt clears it (NULL = manual control, per the schema's own
// words). false if the queue doesn't exist.
DbResult<bool> set_schedule(std::string_view queue_id,
const std::optional<velox::proto::Schedule>& schedule);
// queue.reorder: `task_ids` must be an exact permutation of the queue's current
// membership (the schema's own words — "anything else is -32602 rather than a
// partial reorder, so a stale drag from an out-of-date view cannot quietly reshuffle
// the queue"). false (no write at all) if it isn't; true and queue_position rewritten
// to match `task_ids`'s order if it is.
DbResult<bool> reorder(std::string_view queue_id, const std::vector<std::string>& task_ids);
// "Omit queueId to create" (queue.upsert's own words) — an empty id generates one.
// taskIds is ignored (membership changes only through download.update / queue.reorder,
// per the schema's own note); a create defaults to 'stopped', a replace keeps the
+82
View File
@@ -318,6 +318,88 @@ DbResult<std::int64_t> Tasks::count() {
return (*row) ? st->column_int(0) : 0;
}
DbResult<bool> Tasks::apply_update(std::string_view task_id, const UpdatePatch& patch) {
// One UPDATE per present field: simplest thing that's obviously correct for a
// single-row edit with ~9 independent optional fields, and it means a field the
// caller didn't touch is never rewritten with its own unchanged value (matters for
// no-op-detection callers, though download.update doesn't currently need that).
bool touched_any = false;
auto run = [&](const char* sql, auto&& binder) -> DbResult<void> {
auto st = db_.prepare(sql);
if (!st) return std::unexpected(st.error());
if (auto r = st->bind(1, task_id); !r) return std::unexpected(r.error());
if (auto r = binder(*st); !r) return std::unexpected(r.error());
if (auto r = st->step(); !r) return std::unexpected(r.error());
touched_any = true;
return {};
};
if (patch.save_dir && patch.filename) {
if (auto r = run("UPDATE tasks SET save_dir = ?2, filename = ?3 WHERE task_id = ?1",
[&](Stmt& s) {
if (auto b = s.bind(2, std::string_view(*patch.save_dir)); !b) return b;
return s.bind(3, std::string_view(*patch.filename));
});
!r)
return std::unexpected(r.error());
}
if (patch.category_id) {
if (auto r = run("UPDATE tasks SET category_id = ?2 WHERE task_id = ?1",
[&](Stmt& s) { return s.bind(2, std::string_view(*patch.category_id)); });
!r)
return std::unexpected(r.error());
}
if (patch.queue_id) {
if (auto r = run("UPDATE tasks SET queue_id = ?2, queue_position = ?3 WHERE task_id = ?1",
[&](Stmt& s) {
if (auto b = s.bind(2, std::string_view(*patch.queue_id)); !b) return b;
return patch.queue_position ? s.bind(3, *patch.queue_position)
: s.bind_null(3);
});
!r)
return std::unexpected(r.error());
}
if (patch.description) {
if (auto r = run("UPDATE tasks SET description = ?2 WHERE task_id = ?1",
[&](Stmt& s) { return s.bind(2, std::string_view(*patch.description)); });
!r)
return std::unexpected(r.error());
}
if (patch.req_segments) {
if (auto r = run("UPDATE tasks SET req_segments = ?2 WHERE task_id = ?1",
[&](Stmt& s) { return s.bind(2, *patch.req_segments); });
!r)
return std::unexpected(r.error());
}
if (patch.req_buffer_bytes) {
if (auto r = run("UPDATE tasks SET req_buffer_bytes = ?2 WHERE task_id = ?1",
[&](Stmt& s) { return s.bind(2, *patch.req_buffer_bytes); });
!r)
return std::unexpected(r.error());
}
if (patch.checksum_algo && patch.checksum_value) {
if (auto r = run(
"UPDATE tasks SET checksum_algo = ?2, checksum_value = ?3 WHERE task_id = ?1",
[&](Stmt& s) {
if (auto b = s.bind(2, std::string_view(*patch.checksum_algo)); !b) return b;
return s.bind(3, std::string_view(*patch.checksum_value));
});
!r)
return std::unexpected(r.error());
}
return touched_any;
}
DbResult<bool> Tasks::set_url(std::string_view task_id, std::string_view url) {
auto st = db_.prepare("UPDATE tasks SET url = ?2 WHERE task_id = ?1");
if (!st) return std::unexpected(st.error());
if (auto r = st->bind(1, task_id); !r) return std::unexpected(r.error());
if (auto r = st->bind(2, url); !r) return std::unexpected(r.error());
if (auto r = st->step(); !r) return std::unexpected(r.error());
return sqlite3_changes(db_.raw()) > 0;
}
DbResult<bool> Tasks::has_active_duplicate(std::string_view url) {
auto st = db_.prepare(
"SELECT 1 FROM tasks WHERE url = ?1 "
+28
View File
@@ -85,6 +85,34 @@ public:
// is almost always a mistake, e.g. two tabs triggering the same link).
DbResult<bool> has_active_duplicate(std::string_view url);
// download.update's patch, already resolved by the caller (new save_dir/filename
// canonicalized and root-checked, any file already moved on disk — this only writes
// the row). Every field is applied when present; queue_position is written alongside
// queue_id (nullopt leaves the existing position alone — the caller decides what
// "moved into a queue" should set it to). Note: the generated parser collapses "field
// absent" and "field explicitly null" to the same nullopt (DownloadUpdateParamsPatch
// has no way to tell them apart on the wire as generated), so this — like the RPC
// layer above it — can only ever set category_id/queue_id/description/checksum, never
// clear them back to NULL through this call.
struct UpdatePatch {
std::optional<std::string> save_dir;
std::optional<std::string> filename;
std::optional<std::string> category_id;
std::optional<std::string> queue_id;
std::optional<std::int64_t> queue_position;
std::optional<std::string> description;
std::optional<std::int64_t> req_segments;
std::optional<std::int64_t> req_buffer_bytes;
std::optional<std::string> checksum_algo;
std::optional<std::string> checksum_value;
};
DbResult<bool> apply_update(std::string_view task_id, const UpdatePatch& patch);
// download.refreshUrl: point the task at a freshly-issued URL. Separate from
// apply_update/set_probe_result since neither owns the base `url` column — refreshUrl
// is the one caller that changes it after creation.
DbResult<bool> set_url(std::string_view task_id, std::string_view url);
// Byte-counter update from an engine progress tick — cheaper than a full row rewrite,
// and keeps download.list / download.get current between state transitions.
DbResult<bool> update_progress(std::string_view task_id, std::int64_t downloaded_bytes,