core: net/probe + Content-Disposition parser + URL splitter (stage 3)

net/content_disposition — total parser for the mojibake-prone header:
RFC 6266 filename (quoted/token), RFC 5987 filename* ext-values
(charset'lang'pct-encoded, incl. RFC 2231 continuations), legacy RFC 2047
encoded-words (=?UTF-8?B?..?= / ?Q?), and raw Latin-1 bytes; prefers
filename* over filename; strips path components AFTER decoding (a base64
payload can hold '/'). 22-case test table.

net/text_codec (internal) — percent-decode, UTF-8 validation, Latin-1->
UTF-8, base64, RFC 2047 — shared by the CD parser and the URL splitter.

net/url — a small total URL splitter (scheme/userinfo/host/port/path/
query/fragment, http(s) validity) and url_filename() for the last path
segment; used for the filename fallback.

net/probe — HEAD then a ranged GET bytes=0-0 that PROVES resumability
(206 + matching Content-Range + a validator), rather than trusting
Accept-Ranges which servers lie about; the ranged GET is also the HEAD-
refused (403/405/501) fallback. 401/407 -> success result with
requires_auth, not an error. Runs on its own pool (max_concurrent,
default 4) outside the segment budget per ADR 0011 §5. suggest_filename()
does the resolution order (explicit -> disposition -> URL -> download.bin)
with a light strip; rules/ (stage 9) owns the authoritative sanitize.

tools/fuzz — libFuzzer targets for the CD parser and the URL splitter,
compiling the parser sources directly so they're fully instrumented;
self-guards on VELOX_BUILD_FUZZ + Clang (the top-level CMake adds every
tools/* unconditionally). Seed corpora included.

Fixed on the way: a p -> Transfer -> State -> cbs -> p reference cycle in
Prober that leaked every probe (drop the stored Transfer; the worker
keeps State alive). Tests green under ASan/UBSan and TSan.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HPPSGhiArbvQgwC2DNiURS
This commit is contained in:
2026-09-09 23:53:55 +04:00
co-authored by Claude Sonnet 5
parent fdacf732fa
commit 201ebc55d4
25 changed files with 1731 additions and 19 deletions
+8 -10
View File
@@ -13,6 +13,10 @@ add_library(veloxcore STATIC
src/util/thread_pool.cpp src/util/thread_pool.cpp
src/net/curl_error.cpp src/net/curl_error.cpp
src/net/http_client.cpp src/net/http_client.cpp
src/net/text_codec.cpp
src/net/content_disposition.cpp
src/net/url.cpp
src/net/probe.cpp
) )
add_library(velox::core ALIAS veloxcore) add_library(velox::core ALIAS veloxcore)
@@ -37,13 +41,7 @@ if(VELOX_BUILD_TESTS)
add_subdirectory(tests) add_subdirectory(tests)
endif() endif()
# libFuzzer is clang-only; a GCC configure with -DVELOX_BUILD_FUZZ=ON (the `ci` preset) # Fuzz targets live in ${CMAKE_SOURCE_DIR}/tools/fuzz (lane CORE) and are wired in by the
# must not hard-fail. No fuzz targets exist yet — they arrive with net/probe (stage 3) # top-level CMakeLists.txt, which add_subdirectory()s every tools/* with a CMakeLists.
# and meta/veloxpart (stage 5). # tools/fuzz/CMakeLists.txt self-guards on VELOX_BUILD_FUZZ + a Clang compiler.
if(VELOX_BUILD_FUZZ AND NOT CMAKE_CXX_COMPILER_ID MATCHES "Clang") # Present: Content-Disposition, URL (stage 3). Coming: .veloxpart.meta (stage 5).
message(STATUS "veloxcore: VELOX_BUILD_FUZZ set but compiler is "
"${CMAKE_CXX_COMPILER_ID}; fuzz targets need Clang and will be skipped.")
endif()
# When fuzz targets land (stage 3: Content-Disposition, URL; stage 5: .veloxpart.meta),
# they are added here under `if(VELOX_BUILD_FUZZ AND CMAKE_CXX_COMPILER_ID MATCHES "Clang")`
# and live in ${CMAKE_SOURCE_DIR}/tools/fuzz (owned by lane CORE).
@@ -0,0 +1,42 @@
// vdm/net/content_disposition.hpp — parse a Content-Disposition header into a filename.
//
// This is a classic mojibake source (AGENT-CORE §3): RFC 6266 `filename`, RFC 5987
// `filename*` ext-values, RFC 2047 encoded-words in the legacy quoted form, and raw
// Latin-1 bytes all show up in the wild. The parser is total — hostile input yields a
// best-effort or empty result, never a throw or a crash — and has its own test table
// (content_disposition_test.cpp) and a fuzz target (tools/fuzz).
//
// This header compiles standalone.
#ifndef VDM_NET_CONTENT_DISPOSITION_HPP
#define VDM_NET_CONTENT_DISPOSITION_HPP
#include <string>
#include <string_view>
namespace vdm::net {
struct ContentDisposition {
enum class Type { none, inline_, attachment, form_data, other };
Type type = Type::none;
// Best-effort UTF-8 filename, path components stripped, or empty when the header
// carries none. NOT sanitized for the filesystem — that is rules/ (stage 9); this
// only decodes and de-mojibakes. `..` and control characters may still be present.
std::string filename;
// The filename came from an RFC 5987 `filename*` ext-value (preferred over a plain
// `filename` per RFC 6266 §4.3 when both are present).
bool filename_from_ext = false;
[[nodiscard]] bool is_attachment() const noexcept { return type == Type::attachment; }
[[nodiscard]] bool has_filename() const noexcept { return !filename.empty(); }
};
// Parse the value of a Content-Disposition header (everything after the colon).
[[nodiscard]] ContentDisposition parse_content_disposition(std::string_view header_value);
} // namespace vdm::net
#endif // VDM_NET_CONTENT_DISPOSITION_HPP
+90
View File
@@ -0,0 +1,90 @@
// vdm/net/probe.hpp — "what is at this URL?" without downloading it.
//
// Feeds the File Info dialog (docs/04 §2). HEAD first; a ranged GET `bytes=0-0` follows to
// PROVE resumability (a 206 with a matching Content-Range) rather than trust
// `Accept-Ranges`, which servers lie about (docs/06 R4). The ranged GET is also the
// fallback when HEAD is refused (403/405/501).
//
// Runs on its own small worker pool, sized outside the segment budget (ADR 0011 §5) so a
// burst of probes can't starve transfers and capture.offer's 750 ms path never waits on
// one.
//
// This header compiles standalone.
#ifndef VDM_NET_PROBE_HPP
#define VDM_NET_PROBE_HPP
#include <cstdint>
#include <functional>
#include <memory>
#include <optional>
#include <string>
#include <string_view>
#include <vector>
#include "vdm/net/content_disposition.hpp"
#include "vdm/net/http_types.hpp"
#include "vdm/util/result.hpp"
namespace vdm::net {
struct ProbeRequest {
std::string url;
std::vector<HeaderField> headers; // browser headers, verbatim
std::vector<Cookie> cookies;
std::string user_agent;
std::string referrer;
ProxyConfig proxy;
long connect_timeout_ms = 15000;
long overall_timeout_ms = 25000; // download.probe deadline is 30 s
};
struct ProbeResult {
std::string effective_url;
std::vector<std::string> redirect_chain; // requested URL first, effective_url last
long http_status = 0;
std::optional<std::uint64_t> total_size; // full-resource size, if known
std::string mime; // Content-Type value (params kept)
std::string etag;
std::string last_modified;
bool accept_ranges = false; // server advertised Accept-Ranges: bytes
bool resumable = false; // PROVEN: ranged GET -> 206 + matching Content-Range,
// and a validator (ETag or Last-Modified) is present
bool requires_auth = false; // a 401/407 was seen
// Decoded, path-stripped; NOT filesystem-sanitized (rules/ owns that, stage 9).
std::string filename_from_disposition;
std::string filename_from_url;
ContentDisposition::Type disposition_type = ContentDisposition::Type::none;
};
// Resolution order (docs/04 §2.5): explicit user name -> Content-Disposition -> URL path
// segment -> "download.bin". Only a light path/control strip here; rules/ does the
// authoritative sanitize, byte cap, and collision handling.
[[nodiscard]] std::string suggest_filename(const ProbeResult &r,
std::string_view explicit_name = {});
class Prober {
public:
// max_concurrent bounds outstanding probe transfers; the rest queue.
explicit Prober(unsigned max_concurrent = 4);
~Prober();
Prober(const Prober &) = delete;
Prober &operator=(const Prober &) = delete;
// Async. `done` runs on an internal worker thread, exactly once. A 401/407 is a
// SUCCESS result with requires_auth = true (the GUI collects credentials), not an
// error; transport failures and hard HTTP errors (404/410/5xx) are errors.
void probe(ProbeRequest req, std::function<void(Result<ProbeResult>)> done);
private:
struct Impl;
std::unique_ptr<Impl> impl_;
};
} // namespace vdm::net
#endif // VDM_NET_PROBE_HPP
+40
View File
@@ -0,0 +1,40 @@
// vdm/net/url.hpp — a small, total URL splitter.
//
// Not a full RFC 3986 parser (libcurl does the real fetching); just enough to pull a
// filename out of a path and to sanity-check a scheme. Total on hostile input — it has a
// fuzz target (tools/fuzz) — never throws, never asserts.
//
// This header compiles standalone.
#ifndef VDM_NET_URL_HPP
#define VDM_NET_URL_HPP
#include <cstdint>
#include <optional>
#include <string>
#include <string_view>
namespace vdm::net {
struct SplitUrl {
std::string scheme; // lowercased, without "://"
std::string userinfo; // before '@', if any
std::string host; // lowercased; bracketed IPv6 keeps its brackets stripped
std::optional<std::uint16_t> port;
std::string path; // includes the leading '/', or empty
std::string query; // without the '?'
std::string fragment; // without the '#'
bool valid = false; // scheme + host present and scheme is http/https
[[nodiscard]] bool is_http() const noexcept { return scheme == "http" || scheme == "https"; }
};
[[nodiscard]] SplitUrl split_url(std::string_view url);
// The last non-empty path segment, percent-decoded, path components stripped. Empty when
// the path has no usable segment (ends in '/', is empty, or is only "/").
[[nodiscard]] std::string url_filename(std::string_view url);
} // namespace vdm::net
#endif // VDM_NET_URL_HPP
+279
View File
@@ -0,0 +1,279 @@
// vdm/net/content_disposition.cpp
#include "vdm/net/content_disposition.hpp"
#include <algorithm>
#include <string>
#include <utility>
#include <vector>
#include "net/text_codec.hpp"
namespace vdm::net {
namespace {
using detail::decode_rfc2047;
using detail::latin1_to_utf8;
using detail::percent_decode;
using detail::to_utf8_best_effort;
std::string_view trim(std::string_view s) {
while (!s.empty() && (s.front() == ' ' || s.front() == '\t'))
s.remove_prefix(1);
while (!s.empty() &&
(s.back() == ' ' || s.back() == '\t' || s.back() == '\r' || s.back() == '\n'))
s.remove_suffix(1);
return s;
}
std::string ascii_lower(std::string_view s) {
std::string r(s);
for (char &c : r)
if (c >= 'A' && c <= 'Z')
c = static_cast<char>(c - 'A' + 'a');
return r;
}
// Split "type; a=b; c*=d; e=\"f;g\"" into the type and a param list, honouring quoted
// strings (a ';' inside quotes is not a separator). For quoted values the value stored is
// the *raw* inner text (quotes removed, escapes NOT yet resolved) with `quoted = true`;
// callers resolve escapes and strip path components together (order matters — see
// unquote_strip). Keys are lowercased.
struct Param {
std::string key;
std::string value;
bool quoted = false;
};
struct Params {
std::string type;
std::vector<Param> kv;
[[nodiscard]] const Param *find(std::string_view key) const {
for (const auto &p : kv)
if (p.key == key)
return &p;
return nullptr;
}
};
Params tokenize(std::string_view h) {
Params out;
std::size_t i = 0;
const std::size_t n = h.size();
auto read_segment = [&]() -> std::string_view {
std::size_t start = i;
bool in_q = false;
for (; i < n; ++i) {
char c = h[i];
if (c == '"') {
in_q = !in_q;
} else if (c == '\\' && in_q && i + 1 < n) {
++i; // skip escaped char
} else if (c == ';' && !in_q) {
break;
}
}
std::string_view seg = h.substr(start, i - start);
if (i < n)
++i; // consume ';'
return seg;
};
out.type = ascii_lower(trim(read_segment()));
while (i < n) {
std::string_view seg = trim(read_segment());
if (seg.empty())
continue;
auto eq = seg.find('=');
if (eq == std::string_view::npos) {
out.kv.emplace_back(ascii_lower(seg), std::string{});
continue;
}
std::string key = ascii_lower(trim(seg.substr(0, eq)));
std::string_view rawval = trim(seg.substr(eq + 1));
Param param;
param.key = std::move(key);
if (rawval.size() >= 2 && rawval.front() == '"') {
std::string_view inner = rawval.substr(1);
auto close = inner.rfind('"');
if (close != std::string_view::npos)
inner = inner.substr(0, close);
param.value.assign(inner); // raw, escapes unresolved
param.quoted = true;
} else {
param.value.assign(rawval);
}
out.kv.push_back(std::move(param));
}
return out;
}
std::string strip_path(std::string s) {
auto slash = s.find_last_of("/\\");
if (slash != std::string::npos)
s.erase(0, slash + 1);
return s;
}
// Resolve ONLY the `\"` escape (needed so a quote can appear mid-name). Every other
// backslash is kept literal and later treated as a path separator by strip_path — real
// Windows paths in the wild use `\` unescaped, and path-traversal defence matters more
// than supporting the vanishingly rare filename with a literal backslash.
std::string unescape_dquote(std::string_view raw) {
std::string out;
out.reserve(raw.size());
for (std::size_t i = 0; i < raw.size(); ++i) {
if (raw[i] == '\\' && i + 1 < raw.size() && raw[i + 1] == '"') {
out.push_back('"');
++i;
} else {
out.push_back(raw[i]);
}
}
return out;
}
// Decode an RFC 5987 ext-value: charset'lang'pct-encoded-octets
std::string decode_ext_value(std::string_view v) {
auto q1 = v.find('\'');
if (q1 == std::string_view::npos)
return percent_decode(v); // malformed: best effort
auto q2 = v.find('\'', q1 + 1);
if (q2 == std::string_view::npos)
return percent_decode(v.substr(q1 + 1));
std::string_view charset = v.substr(0, q1);
std::string_view enc = v.substr(q2 + 1);
std::string bytes = percent_decode(enc);
std::string cs = ascii_lower(charset);
if (cs == "iso-8859-1" || cs == "latin1")
return latin1_to_utf8(bytes);
return to_utf8_best_effort(bytes); // utf-8 or unknown -> best effort
}
// Reassemble RFC 2231 continuations: name*0*, name*1, name*2* ... in order.
std::string join_continuations(const Params &p, std::string_view base, bool &is_ext) {
std::vector<std::pair<int, std::string>> parts;
is_ext = false;
for (const auto &param : p.kv) {
const std::string &k = param.key;
const std::string &val = param.value;
if (k.size() <= base.size() + 1 || k.compare(0, base.size(), base) != 0)
continue;
if (k[base.size()] != '*')
continue;
std::string_view rest(k);
rest.remove_prefix(base.size() + 1); // after "base*"
bool star = false;
if (!rest.empty() && rest.back() == '*') {
star = true;
rest.remove_suffix(1);
}
int idx = 0;
for (char c : rest) {
if (c < '0' || c > '9') {
idx = -1;
break;
}
idx = idx * 10 + (c - '0');
}
if (idx < 0)
continue;
if (star)
is_ext = true;
parts.emplace_back(idx, val);
}
if (parts.empty())
return {};
std::sort(parts.begin(), parts.end(),
[](const auto &a, const auto &b) { return a.first < b.first; });
// Piece 0 (if star-form) carries charset'lang' prefix; later pieces are raw
// percent-encoded. Concatenate the percent-encoded text then decode once.
std::string charset_prefix;
std::string enc;
bool first = true;
for (auto &[idx, val] : parts) {
if (first && is_ext) {
auto q1 = val.find('\'');
auto q2 = (q1 == std::string::npos) ? std::string::npos : val.find('\'', q1 + 1);
if (q2 != std::string::npos) {
charset_prefix = val.substr(0, q2 + 1);
enc += val.substr(q2 + 1);
} else {
enc += val;
}
} else {
enc += val;
}
first = false;
}
if (is_ext)
return decode_ext_value(charset_prefix + enc);
return to_utf8_best_effort(percent_decode(enc));
}
ContentDisposition::Type classify(std::string_view t) {
if (t == "inline")
return ContentDisposition::Type::inline_;
if (t == "attachment")
return ContentDisposition::Type::attachment;
if (t == "form-data")
return ContentDisposition::Type::form_data;
if (t.empty())
return ContentDisposition::Type::none;
return ContentDisposition::Type::other;
}
} // namespace
ContentDisposition parse_content_disposition(std::string_view header_value) {
ContentDisposition cd;
header_value = trim(header_value);
if (header_value.empty())
return cd;
Params p = tokenize(header_value);
cd.type = classify(p.type);
// RFC 6266 §4.3: prefer filename* over filename.
std::string ext_name;
bool ext_is_ext = false;
if (const Param *fstar = p.find("filename*")) {
ext_name = decode_ext_value(fstar->value);
ext_is_ext = true;
} else {
std::string joined = join_continuations(p, "filename", ext_is_ext);
if (!joined.empty())
ext_name = std::move(joined);
}
std::string plain_name;
if (const Param *f = p.find("filename")) {
// Resolve `\"`, then decode legacy encoded-words if present — BEFORE stripping
// path components, since a base64 payload can legitimately contain '/'.
std::string raw = f->quoted ? unescape_dquote(f->value) : f->value;
bool had_ew = false;
std::string decoded = decode_rfc2047(raw, &had_ew);
plain_name = had_ew ? std::move(decoded) : to_utf8_best_effort(raw);
}
if (!ext_name.empty()) {
cd.filename = strip_path(std::move(ext_name));
cd.filename_from_ext = ext_is_ext;
} else if (!plain_name.empty()) {
cd.filename = strip_path(std::move(plain_name));
cd.filename_from_ext = false;
}
// Trim ASCII whitespace the decoders may have left at the edges.
std::string_view fv = trim(cd.filename);
cd.filename.assign(fv);
return cd;
}
} // namespace vdm::net
+348
View File
@@ -0,0 +1,348 @@
// vdm/net/probe.cpp
#include "vdm/net/probe.hpp"
#include <charconv>
#include <deque>
#include <mutex>
#include "net/curl_error.hpp"
#include "vdm/net/http_client.hpp"
#include "vdm/net/url.hpp"
namespace vdm::net {
namespace {
std::string_view trim(std::string_view s) {
while (!s.empty() && (s.front() == ' ' || s.front() == '\t'))
s.remove_prefix(1);
while (!s.empty() && (s.back() == ' ' || s.back() == '\t'))
s.remove_suffix(1);
return s;
}
bool iequals(std::string_view a, std::string_view b) {
return HeaderList::iequals(a, b);
}
// "bytes 0-0/12345" -> 12345 ; "bytes 0-0/*" or malformed -> nullopt
std::optional<std::uint64_t> total_from_content_range(std::string_view v) {
auto slash = v.find('/');
if (slash == std::string_view::npos)
return std::nullopt;
std::string_view tail = trim(v.substr(slash + 1));
if (tail.empty() || tail == "*")
return std::nullopt;
std::uint64_t n = 0;
auto [p, ec] = std::from_chars(tail.data(), tail.data() + tail.size(), n);
(void)p;
if (ec != std::errc{})
return std::nullopt;
return n;
}
std::string light_sanitize(std::string_view in) {
std::string out;
out.reserve(in.size());
for (unsigned char c : in) {
if (c == '/' || c == '\\' || c == 0) {
out.push_back('_');
} else if (c >= 0x20 || (c & 0x80)) { // keep printable ASCII + all UTF-8 bytes
out.push_back(static_cast<char>(c));
}
}
while (!out.empty() && (out.back() == '.' || out.back() == ' '))
out.pop_back();
if (out == "." || out == "..")
out.clear();
return out;
}
} // namespace
std::string suggest_filename(const ProbeResult &r, std::string_view explicit_name) {
std::string cand;
if (!explicit_name.empty())
cand = light_sanitize(explicit_name);
if (cand.empty() && !r.filename_from_disposition.empty())
cand = light_sanitize(r.filename_from_disposition);
if (cand.empty() && !r.filename_from_url.empty())
cand = light_sanitize(r.filename_from_url);
if (cand.empty())
cand = "download.bin";
return cand;
}
// --- Prober::Impl ---------------------------------------------------------------------
struct Prober::Impl {
struct Job {
ProbeRequest req;
std::function<void(Result<ProbeResult>)> done;
};
struct P {
Impl *self = nullptr;
Job job;
ProbeResult result;
bool had_head_ok = false;
bool delivered = false;
};
explicit Impl(unsigned max_concurrent)
: max_(max_concurrent ? max_concurrent : 1), client_(HttpClient::Options{.workers = 2}) {}
unsigned max_;
HttpClient client_;
std::mutex mu_;
unsigned inflight_ = 0;
std::deque<Job> pending_;
void submit(Job j) {
{
std::lock_guard lk(mu_);
if (inflight_ >= max_) {
pending_.push_back(std::move(j));
return;
}
++inflight_;
}
start(std::move(j));
}
void finish_one() {
Job next;
bool have_next = false;
{
std::lock_guard lk(mu_);
--inflight_;
if (!pending_.empty()) {
next = std::move(pending_.front());
pending_.pop_front();
++inflight_;
have_next = true;
}
}
if (have_next)
start(std::move(next));
}
Request base_request(const ProbeRequest &pr) {
Request r;
r.url = pr.url;
r.headers = pr.headers;
r.cookies = pr.cookies;
r.user_agent = pr.user_agent;
r.referrer = pr.referrer;
r.proxy = pr.proxy;
r.follow_redirects = true;
r.accept_encoding = false;
r.connect_timeout_ms = pr.connect_timeout_ms;
r.overall_timeout_ms = pr.overall_timeout_ms;
r.low_speed_bytes_per_sec = 0; // probes are tiny; no stall detector
r.low_speed_secs = 0;
return r;
}
void start(Job j) {
auto p = std::make_shared<P>();
p->self = this;
p->job = std::move(j);
p->result.effective_url = p->job.req.url;
Request req = base_request(p->job.req);
req.method = Method::head;
TransferCallbacks cbs;
cbs.on_head = [p](const ResponseHead &h) {
absorb_head(p->result, h);
return DataAction::abort;
};
cbs.on_data = [](ConstByteSpan) { return DataAction::abort; };
cbs.on_finished = [p](Result<TransferStats> r) { on_head_done(p, std::move(r)); };
// The worker keeps Transfer::State alive; the callbacks keep `p` alive. Storing
// the Transfer in `p` would make a p -> Transfer -> State -> cbs -> p cycle.
client_.start(std::move(req), std::move(cbs));
}
static void absorb_head(ProbeResult &res, const ResponseHead &h) {
if (h.status)
res.http_status = h.status;
if (!h.effective_url.empty())
res.effective_url = h.effective_url;
refresh_common(res, h);
if (h.content_length && !res.total_size)
res.total_size = h.content_length;
}
static void refresh_common(ProbeResult &res, const ResponseHead &h) {
if (auto v = h.headers.get("Content-Type"); v && res.mime.empty()) {
std::string_view mv = *v;
mv = mv.substr(0, mv.find(';'));
res.mime.assign(trim(mv));
}
if (auto v = h.headers.get("ETag"); v && res.etag.empty())
res.etag.assign(*v);
if (auto v = h.headers.get("Last-Modified"); v && res.last_modified.empty())
res.last_modified.assign(*v);
if (auto v = h.headers.get("Accept-Ranges")) {
if (iequals(trim(*v), "bytes"))
res.accept_ranges = true;
}
if (auto v = h.headers.get("Content-Disposition");
v && res.filename_from_disposition.empty()) {
ContentDisposition cd = parse_content_disposition(*v);
res.disposition_type = cd.type;
if (cd.has_filename())
res.filename_from_disposition = cd.filename;
}
}
static bool has_validator(const ProbeResult &r) {
return !r.etag.empty() || !r.last_modified.empty();
}
static void on_head_done(std::shared_ptr<P> p, Result<TransferStats> r) {
const long status = p->result.http_status;
if (status == 0) { // never got headers -> transport failure
deliver(p, r.has_value() ? Result<ProbeResult>(ErrorInfo(Error::probe_failed))
: Result<ProbeResult>(std::move(r).error()));
return;
}
p->had_head_ok = (status >= 200 && status < 300);
if (status == 401 || status == 407) {
p->result.requires_auth = true;
finalize_and_deliver(p);
return;
}
if (status == 403 || status == 405 || status == 501) {
start_range_get(p); // HEAD refused; the ranged GET is now the primary probe
return;
}
if (status >= 400) {
deliver(p, ErrorInfo(detail::error_from_curl(CURLE_OK, status), "probe HEAD",
static_cast<int>(status)));
return;
}
// 2xx. Prove resumability with a ranged GET unless the server said "none".
if (auto ar = p->result.accept_ranges; !ar) {
// Accept-Ranges absent or not "bytes": still try one ranged GET — servers
// that support ranges without advertising are common (docs/06 R4).
}
start_range_get(p);
}
static void start_range_get(const std::shared_ptr<P> &p) {
Request req = p->self->base_request(p->job.req);
req.method = Method::get;
req.range = ByteRange{0, 0};
TransferCallbacks cbs;
cbs.on_head = [p](const ResponseHead &h) {
absorb_range_head(p->result, h);
return DataAction::abort; // we don't need the one body byte
};
cbs.on_data = [](ConstByteSpan) { return DataAction::abort; };
cbs.on_finished = [p](Result<TransferStats> r) { on_range_done(p, std::move(r)); };
p->self->client_.start(std::move(req), std::move(cbs));
}
static void absorb_range_head(ProbeResult &res, const ResponseHead &h) {
if (h.status)
res.http_status = h.status;
if (!h.effective_url.empty())
res.effective_url = h.effective_url;
refresh_common(res, h);
if (h.status == 206) {
res.accept_ranges = true; // proven, not just advertised
if (auto cr = h.headers.get("Content-Range")) {
if (auto total = total_from_content_range(*cr))
res.total_size = total;
}
} else if (h.status == 200) {
if (h.content_length)
res.total_size = h.content_length;
}
}
static void on_range_done(std::shared_ptr<P> p, Result<TransferStats> r) {
const long status = p->result.http_status;
if (status == 0) { // range GET died at transport level
if (p->had_head_ok) {
p->result.resumable = false;
finalize_and_deliver(p);
} else {
deliver(p, r.has_value() ? Result<ProbeResult>(ErrorInfo(Error::probe_failed))
: Result<ProbeResult>(std::move(r).error()));
}
return;
}
if (status == 401 || status == 407) {
p->result.requires_auth = true;
finalize_and_deliver(p);
return;
}
if (status == 206) {
p->result.resumable = has_validator(p->result);
finalize_and_deliver(p);
return;
}
if (status == 200 || status == 416) {
p->result.resumable = false;
finalize_and_deliver(p);
return;
}
if (status >= 400) {
if (p->had_head_ok) {
p->result.resumable = false;
finalize_and_deliver(p);
} else {
deliver(p, ErrorInfo(detail::error_from_curl(CURLE_OK, status), "probe ranged GET",
static_cast<int>(status)));
}
return;
}
p->result.resumable = false;
finalize_and_deliver(p);
}
static void finalize_and_deliver(const std::shared_ptr<P> &p) {
ProbeResult &res = p->result;
if (res.effective_url.empty())
res.effective_url = p->job.req.url;
res.filename_from_url = url_filename(res.effective_url);
if (res.filename_from_url.empty() && res.effective_url != p->job.req.url)
res.filename_from_url = url_filename(p->job.req.url);
res.redirect_chain.clear();
res.redirect_chain.push_back(p->job.req.url);
if (res.effective_url != p->job.req.url)
res.redirect_chain.push_back(res.effective_url);
deliver(p, ProbeResult(res));
}
static void deliver(const std::shared_ptr<P> &p, Result<ProbeResult> out) {
if (p->delivered)
return;
p->delivered = true;
Impl *self = p->self;
p->job.done(std::move(out));
self->finish_one();
}
};
// --- Prober -----------------------------------------------------------------------
Prober::Prober(unsigned max_concurrent) : impl_(std::make_unique<Impl>(max_concurrent)) {}
Prober::~Prober() = default;
void Prober::probe(ProbeRequest req, std::function<void(Result<ProbeResult>)> done) {
impl_->submit(Impl::Job{std::move(req), std::move(done)});
}
} // namespace vdm::net
+262
View File
@@ -0,0 +1,262 @@
// vdm/net/text_codec.cpp
#include "net/text_codec.hpp"
#include <array>
#include <cctype>
namespace vdm::net::detail {
namespace {
int hex_val(char c) noexcept {
if (c >= '0' && c <= '9')
return c - '0';
if (c >= 'a' && c <= 'f')
return c - 'a' + 10;
if (c >= 'A' && c <= 'F')
return c - 'A' + 10;
return -1;
}
void append_utf8(std::string &out, std::uint32_t cp) {
if (cp <= 0x7F) {
out.push_back(static_cast<char>(cp));
} else if (cp <= 0x7FF) {
out.push_back(static_cast<char>(0xC0 | (cp >> 6)));
out.push_back(static_cast<char>(0x80 | (cp & 0x3F)));
} else if (cp <= 0xFFFF) {
out.push_back(static_cast<char>(0xE0 | (cp >> 12)));
out.push_back(static_cast<char>(0x80 | ((cp >> 6) & 0x3F)));
out.push_back(static_cast<char>(0x80 | (cp & 0x3F)));
} else {
out.push_back(static_cast<char>(0xF0 | (cp >> 18)));
out.push_back(static_cast<char>(0x80 | ((cp >> 12) & 0x3F)));
out.push_back(static_cast<char>(0x80 | ((cp >> 6) & 0x3F)));
out.push_back(static_cast<char>(0x80 | (cp & 0x3F)));
}
}
bool charset_is(std::string_view cs, std::string_view want) {
if (cs.size() != want.size())
return false;
for (std::size_t i = 0; i < cs.size(); ++i) {
char a = cs[i], b = want[i];
if (a >= 'A' && a <= 'Z')
a = static_cast<char>(a - 'A' + 'a');
if (b >= 'A' && b <= 'Z')
b = static_cast<char>(b - 'A' + 'a');
if (a != b)
return false;
}
return true;
}
bool is_utf8(std::string_view cs) {
return charset_is(cs, "utf-8") || charset_is(cs, "utf8");
}
bool is_latin1(std::string_view cs) {
return charset_is(cs, "iso-8859-1") || charset_is(cs, "latin1") ||
charset_is(cs, "iso8859-1") || charset_is(cs, "windows-1252");
}
} // namespace
std::string percent_decode(std::string_view in, bool plus_as_space) {
std::string out;
out.reserve(in.size());
for (std::size_t i = 0; i < in.size(); ++i) {
char c = in[i];
if (c == '%' && i + 2 < in.size()) {
int hi = hex_val(in[i + 1]);
int lo = hex_val(in[i + 2]);
if (hi >= 0 && lo >= 0) {
out.push_back(static_cast<char>((hi << 4) | lo));
i += 2;
continue;
}
}
if (c == '+' && plus_as_space) {
out.push_back(' ');
continue;
}
out.push_back(c);
}
return out;
}
bool is_valid_utf8(std::string_view s) noexcept {
std::size_t i = 0;
const std::size_t n = s.size();
auto cont = [&](std::size_t k) {
return k < n && (static_cast<unsigned char>(s[k]) & 0xC0) == 0x80;
};
while (i < n) {
unsigned char c = static_cast<unsigned char>(s[i]);
if (c < 0x80) {
++i;
} else if ((c & 0xE0) == 0xC0) {
if (!cont(i + 1))
return false;
std::uint32_t cp = (c & 0x1F) << 6 | (static_cast<unsigned char>(s[i + 1]) & 0x3F);
if (cp < 0x80)
return false; // overlong
i += 2;
} else if ((c & 0xF0) == 0xE0) {
if (!cont(i + 1) || !cont(i + 2))
return false;
std::uint32_t cp = (c & 0x0F) << 12 |
(static_cast<unsigned char>(s[i + 1]) & 0x3F) << 6 |
(static_cast<unsigned char>(s[i + 2]) & 0x3F);
if (cp < 0x800 || (cp >= 0xD800 && cp <= 0xDFFF))
return false;
i += 3;
} else if ((c & 0xF8) == 0xF0) {
if (!cont(i + 1) || !cont(i + 2) || !cont(i + 3))
return false;
std::uint32_t cp = (c & 0x07) << 18 |
(static_cast<unsigned char>(s[i + 1]) & 0x3F) << 12 |
(static_cast<unsigned char>(s[i + 2]) & 0x3F) << 6 |
(static_cast<unsigned char>(s[i + 3]) & 0x3F);
if (cp < 0x10000 || cp > 0x10FFFF)
return false;
i += 4;
} else {
return false;
}
}
return true;
}
std::string latin1_to_utf8(std::string_view s) {
std::string out;
out.reserve(s.size() + s.size() / 2);
for (char ch : s)
append_utf8(out, static_cast<unsigned char>(ch));
return out;
}
std::string to_utf8_best_effort(std::string_view s) {
return is_valid_utf8(s) ? std::string(s) : latin1_to_utf8(s);
}
std::string base64_decode(std::string_view in) {
auto val = [](char c) -> int {
if (c >= 'A' && c <= 'Z')
return c - 'A';
if (c >= 'a' && c <= 'z')
return c - 'a' + 26;
if (c >= '0' && c <= '9')
return c - '0' + 52;
if (c == '+')
return 62;
if (c == '/')
return 63;
return -1;
};
std::string out;
out.reserve(in.size() / 4 * 3 + 3);
std::uint32_t acc = 0;
int bits = 0;
for (char c : in) {
if (c == '=' || c == '\r' || c == '\n' || c == ' ' || c == '\t')
continue;
int v = val(c);
if (v < 0)
continue; // skip stray bytes
acc = (acc << 6) | static_cast<std::uint32_t>(v);
bits += 6;
if (bits >= 8) {
bits -= 8;
out.push_back(static_cast<char>((acc >> bits) & 0xFF));
}
}
return out;
}
std::string decode_rfc2047(std::string_view in, bool *had_encoded_word) {
if (had_encoded_word)
*had_encoded_word = false;
std::string out;
out.reserve(in.size());
std::size_t i = 0;
const std::size_t n = in.size();
while (i < n) {
auto start = in.find("=?", i);
if (start == std::string_view::npos) {
out.append(in.substr(i));
break;
}
out.append(in.substr(i, start - i));
// =?charset?enc?text?=
auto q1 = in.find('?', start + 2);
if (q1 == std::string_view::npos) {
out.append(in.substr(start));
break;
}
auto q2 = in.find('?', q1 + 1);
if (q2 == std::string_view::npos || q2 != q1 + 2) {
out.append("=?");
i = start + 2;
continue;
}
auto end = in.find("?=", q2 + 1);
if (end == std::string_view::npos) {
out.append(in.substr(start));
break;
}
std::string_view charset = in.substr(start + 2, q1 - (start + 2));
char enc = in[q1 + 1];
std::string_view text = in.substr(q2 + 1, end - (q2 + 1));
std::string bytes;
if (enc == 'B' || enc == 'b') {
bytes = base64_decode(text);
} else if (enc == 'Q' || enc == 'q') {
for (std::size_t k = 0; k < text.size(); ++k) {
char c = text[k];
if (c == '_') {
bytes.push_back(' ');
} else if (c == '=' && k + 2 < text.size()) {
int hi = hex_val(text[k + 1]);
int lo = hex_val(text[k + 2]);
if (hi >= 0 && lo >= 0) {
bytes.push_back(static_cast<char>((hi << 4) | lo));
k += 2;
} else {
bytes.push_back(c);
}
} else {
bytes.push_back(c);
}
}
} else {
out.append(in.substr(start, end + 2 - start)); // unknown encoding: verbatim
i = end + 2;
continue;
}
if (is_utf8(charset))
out.append(to_utf8_best_effort(bytes));
else if (is_latin1(charset))
out.append(latin1_to_utf8(bytes));
else
out.append(to_utf8_best_effort(bytes));
if (had_encoded_word)
*had_encoded_word = true;
i = end + 2;
// RFC 2047: whitespace between adjacent encoded words is elided.
std::size_t j = i;
while (j < n && (in[j] == ' ' || in[j] == '\t'))
++j;
if (j < n && in.compare(j, 2, "=?") == 0)
i = j;
}
return out;
}
} // namespace vdm::net::detail
+40
View File
@@ -0,0 +1,40 @@
// vdm/net/text_codec.hpp — internal: small byte/text codecs for header parsing.
//
// Not a public header. Everything here is pure, allocation-bounded, and total (no throw,
// no assert on input): the inputs come off the wire from untrusted servers.
#ifndef VDM_NET_TEXT_CODEC_HPP
#define VDM_NET_TEXT_CODEC_HPP
#include <cstdint>
#include <string>
#include <string_view>
namespace vdm::net::detail {
// Percent-decode ("%XX"). A stray '%' or a non-hex digit after it is emitted literally.
// `plus_as_space` handles application/x-www-form-urlencoded style; off for URL paths.
[[nodiscard]] std::string percent_decode(std::string_view in, bool plus_as_space = false);
// True if `s` is well-formed UTF-8 (no overlong forms, no surrogates, no > U+10FFFF).
[[nodiscard]] bool is_valid_utf8(std::string_view s) noexcept;
// Reinterpret each byte as a Latin-1 (ISO-8859-1) code point and re-encode as UTF-8.
[[nodiscard]] std::string latin1_to_utf8(std::string_view s);
// Decode standard base64 (RFC 4648, '+' '/', optional '=' padding). Whitespace is
// skipped. Invalid trailing bits are dropped. Returns the decoded bytes.
[[nodiscard]] std::string base64_decode(std::string_view in);
// Decode RFC 2047 "encoded-word" runs: =?charset?B?..?= / =?charset?Q?..?=. Text outside
// encoded words is passed through. Only UTF-8 and ISO-8859-1/Latin-1 charsets are
// transcoded; anything else is passed through as-is (best effort). `had_encoded_word`
// reports whether at least one well-formed word was found.
[[nodiscard]] std::string decode_rfc2047(std::string_view in, bool *had_encoded_word = nullptr);
// If `s` is valid UTF-8, return it unchanged; otherwise treat it as Latin-1 and transcode.
[[nodiscard]] std::string to_utf8_best_effort(std::string_view s);
} // namespace vdm::net::detail
#endif // VDM_NET_TEXT_CODEC_HPP
+126
View File
@@ -0,0 +1,126 @@
// vdm/net/url.cpp
#include "vdm/net/url.hpp"
#include <charconv>
#include "net/text_codec.hpp"
namespace vdm::net {
namespace {
std::string ascii_lower(std::string_view s) {
std::string r(s);
for (char &c : r)
if (c >= 'A' && c <= 'Z')
c = static_cast<char>(c - 'A' + 'a');
return r;
}
bool valid_scheme(std::string_view s) {
if (s.empty())
return false;
if (!((s[0] >= 'a' && s[0] <= 'z') || (s[0] >= 'A' && s[0] <= 'Z')))
return false;
for (char c : s) {
bool ok = (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') ||
c == '+' || c == '-' || c == '.';
if (!ok)
return false;
}
return true;
}
} // namespace
SplitUrl split_url(std::string_view url) {
SplitUrl out;
auto scheme_end = url.find("://");
if (scheme_end == std::string_view::npos)
return out;
std::string_view scheme = url.substr(0, scheme_end);
if (!valid_scheme(scheme))
return out;
out.scheme = ascii_lower(scheme);
std::string_view rest = url.substr(scheme_end + 3);
// authority ends at the first '/', '?' or '#'
std::size_t auth_end = rest.size();
for (std::size_t i = 0; i < rest.size(); ++i) {
char c = rest[i];
if (c == '/' || c == '?' || c == '#') {
auth_end = i;
break;
}
}
std::string_view authority = rest.substr(0, auth_end);
std::string_view tail = rest.substr(auth_end);
if (auto at = authority.rfind('@'); at != std::string_view::npos) {
out.userinfo.assign(authority.substr(0, at));
authority = authority.substr(at + 1);
}
std::string_view host = authority;
std::string_view port;
if (!authority.empty() && authority.front() == '[') {
auto close = authority.find(']');
if (close != std::string_view::npos) {
host = authority.substr(1, close - 1); // strip brackets
if (close + 1 < authority.size() && authority[close + 1] == ':')
port = authority.substr(close + 2);
}
} else if (auto colon = authority.rfind(':'); colon != std::string_view::npos) {
host = authority.substr(0, colon);
port = authority.substr(colon + 1);
}
out.host = ascii_lower(host);
if (!port.empty()) {
unsigned v = 0;
auto [p, ec] = std::from_chars(port.data(), port.data() + port.size(), v);
(void)p;
if (ec == std::errc{} && v > 0 && v <= 65535)
out.port = static_cast<std::uint16_t>(v);
}
// tail = path [ '?' query ] [ '#' fragment ]
std::string_view path_and_rest = tail;
if (auto hash = path_and_rest.find('#'); hash != std::string_view::npos) {
out.fragment.assign(path_and_rest.substr(hash + 1));
path_and_rest = path_and_rest.substr(0, hash);
}
if (auto q = path_and_rest.find('?'); q != std::string_view::npos) {
out.query.assign(path_and_rest.substr(q + 1));
path_and_rest = path_and_rest.substr(0, q);
}
out.path.assign(path_and_rest);
out.valid = !out.host.empty() && out.is_http();
return out;
}
std::string url_filename(std::string_view url) {
SplitUrl u = split_url(url);
std::string_view path = u.path;
if (path.empty())
return {};
auto slash = path.find_last_of('/');
std::string_view seg = (slash == std::string_view::npos) ? path : path.substr(slash + 1);
if (seg.empty())
return {};
std::string name = detail::percent_decode(seg);
// Guard against a decoded segment that reintroduces a separator or NUL.
for (char &c : name)
if (c == '/' || c == '\\' || c == '\0')
c = '_';
if (name == "." || name == "..")
return {};
return name;
}
} // namespace vdm::net
+14 -7
View File
@@ -23,14 +23,21 @@ vdm_add_test(veloxcore_log_test util/log_test.cpp)
# net/ integration tests drive tools/testserver (lane PKG/QA). Skip cleanly if it isn't # net/ integration tests drive tools/testserver (lane PKG/QA). Skip cleanly if it isn't
# in the tree yet (lanes merge independently). # in the tree yet (lanes merge independently).
vdm_add_test(veloxcore_content_disposition_test net/content_disposition_test.cpp)
vdm_add_test(veloxcore_url_test net/url_test.cpp)
set(_testserver ${CMAKE_SOURCE_DIR}/tools/testserver/testserver.py) set(_testserver ${CMAKE_SOURCE_DIR}/tools/testserver/testserver.py)
vdm_add_test(veloxcore_http_client_test net/http_client_test.cpp) foreach(net_it http_client probe)
target_include_directories(veloxcore_http_client_test PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/net) vdm_add_test(veloxcore_${net_it}_test net/${net_it}_test.cpp)
target_include_directories(veloxcore_${net_it}_test
PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/net)
if(EXISTS ${_testserver}) if(EXISTS ${_testserver})
target_compile_definitions(veloxcore_http_client_test target_compile_definitions(veloxcore_${net_it}_test
PRIVATE VDM_TESTSERVER_PY="${_testserver}") PRIVATE VDM_TESTSERVER_PY="${_testserver}")
set_tests_properties(veloxcore_http_client_test PROPERTIES TIMEOUT 120) set_tests_properties(veloxcore_${net_it}_test PROPERTIES TIMEOUT 120)
else() endif()
message(STATUS "veloxcore: tools/testserver not present; http_client_test will skip " endforeach()
"its server-backed cases.") if(NOT EXISTS ${_testserver})
message(STATUS "veloxcore: tools/testserver not present; net integration tests will "
"skip their server-backed cases.")
endif() endif()
+148
View File
@@ -0,0 +1,148 @@
#include "vdm/net/content_disposition.hpp"
#include <string>
#include <string_view>
#include "vtest.hpp"
using vdm::net::ContentDisposition;
using vdm::net::parse_content_disposition;
using Type = vdm::net::ContentDisposition::Type;
namespace {
// "€" is U+20AC -> UTF-8 E2 82 AC. "£" is U+00A3 -> UTF-8 C2 A3.
const std::string kEuro = "\xE2\x82\xAC";
const std::string kPound = "\xC2\xA3";
const std::string kEAcute = "\xC3\xA9"; // é U+00E9
} // namespace
VT_TEST(cd_plain_quoted) {
auto cd = parse_content_disposition(R"(attachment; filename="report.pdf")");
VT_CHECK(cd.type == Type::attachment);
VT_CHECK_EQ(cd.filename, std::string("report.pdf"));
VT_CHECK(!cd.filename_from_ext);
}
VT_TEST(cd_plain_token_unquoted) {
auto cd = parse_content_disposition("attachment; filename=report.pdf");
VT_CHECK_EQ(cd.filename, std::string("report.pdf"));
}
VT_TEST(cd_inline_no_filename) {
auto cd = parse_content_disposition("inline");
VT_CHECK(cd.type == Type::inline_);
VT_CHECK(!cd.has_filename());
}
VT_TEST(cd_rfc5987_utf8_ext_value) {
auto cd = parse_content_disposition("attachment; filename*=UTF-8''%e2%82%ac%20rates.pdf");
VT_CHECK_EQ(cd.filename, kEuro + " rates.pdf");
VT_CHECK(cd.filename_from_ext);
}
VT_TEST(cd_prefers_ext_over_plain) {
auto cd = parse_content_disposition(
R"(attachment; filename="EURO rates.pdf"; filename*=UTF-8''%e2%82%ac%20rates.pdf)");
VT_CHECK_EQ(cd.filename, kEuro + " rates.pdf");
VT_CHECK(cd.filename_from_ext);
}
VT_TEST(cd_rfc5987_latin1_ext_value) {
// %A3 = £ in ISO-8859-1
auto cd = parse_content_disposition("attachment; filename*=ISO-8859-1''%A3rates.pdf");
VT_CHECK_EQ(cd.filename, kPound + "rates.pdf");
}
VT_TEST(cd_legacy_rfc2047_base64) {
// base64("<euro> rates.pdf") with euro as UTF-8
// "€ rates.pdf" -> bytes E2 82 AC 20 72 61 74 65 73 2E 70 64 66 -> base64:
auto cd =
parse_content_disposition(R"(attachment; filename="=?UTF-8?B?4oKsIHJhdGVzLnBkZg==?=")");
VT_CHECK_EQ(cd.filename, kEuro + " rates.pdf");
}
VT_TEST(cd_legacy_rfc2047_qencoded_latin1) {
// =?ISO-8859-1?Q?=A3rates.pdf?= -> £rates.pdf
auto cd = parse_content_disposition(R"(attachment; filename="=?ISO-8859-1?Q?=A3rates.pdf?=")");
VT_CHECK_EQ(cd.filename, kPound + "rates.pdf");
}
VT_TEST(cd_raw_utf8_bytes_in_quotes) {
std::string h = "attachment; filename=\"caf" + kEAcute + ".txt\"";
auto cd = parse_content_disposition(h);
VT_CHECK_EQ(cd.filename, "caf" + kEAcute + ".txt");
}
VT_TEST(cd_raw_latin1_byte_in_quotes) {
// 0xE9 is 'é' in Latin-1; not valid UTF-8 alone -> transcoded
std::string h = "attachment; filename=\"caf\xE9.txt\"";
auto cd = parse_content_disposition(h);
VT_CHECK_EQ(cd.filename, "caf" + kEAcute + ".txt");
}
VT_TEST(cd_strips_path_components) {
VT_CHECK_EQ(parse_content_disposition(R"(attachment; filename="../../etc/passwd")").filename,
std::string("passwd"));
// real Windows paths in the wild use unescaped backslashes as separators
VT_CHECK_EQ(parse_content_disposition(R"(attachment; filename="C:\Windows\evil.exe")").filename,
std::string("evil.exe"));
// a base64 payload can contain '/', so decode must happen before path stripping
VT_CHECK_EQ(parse_content_disposition(R"(attachment; filename="=?UTF-8?B?Li4vLi4vc2VjcmV0?=")")
.filename,
std::string("secret")); // decodes to "../../secret", then stripped
}
VT_TEST(cd_quoted_dquote_escape) {
// \" is the one escape we resolve, so a quote can appear mid-name
auto cd = parse_content_disposition(R"(attachment; filename="quote\"here.txt")");
VT_CHECK_EQ(cd.filename, std::string("quote\"here.txt"));
}
VT_TEST(cd_semicolon_inside_quotes_is_not_a_separator) {
auto cd = parse_content_disposition(R"(attachment; filename="a;b;c.txt")");
VT_CHECK_EQ(cd.filename, std::string("a;b;c.txt"));
}
VT_TEST(cd_form_data) {
auto cd = parse_content_disposition(R"(form-data; name="file"; filename="upload.bin")");
VT_CHECK(cd.type == Type::form_data);
VT_CHECK_EQ(cd.filename, std::string("upload.bin"));
}
VT_TEST(cd_rfc2231_continuations) {
auto cd = parse_content_disposition(
"attachment; filename*0*=UTF-8''%e2%82%ac; filename*1*=%20rates; filename*2=.pdf");
VT_CHECK_EQ(cd.filename, kEuro + " rates.pdf");
}
VT_TEST(cd_empty_and_garbage_do_not_crash) {
VT_CHECK(!parse_content_disposition("").has_filename());
VT_CHECK(!parse_content_disposition(";;;;").has_filename());
VT_CHECK(!parse_content_disposition("attachment;").has_filename());
VT_CHECK(!parse_content_disposition(R"(attachment; filename=)").has_filename());
VT_CHECK(!parse_content_disposition(R"(attachment; filename="")").has_filename());
// truncated ext-value
auto cd = parse_content_disposition("attachment; filename*=UTF-8''%e2%82");
VT_CHECK(cd.type == Type::attachment); // no crash; filename is whatever fell out
// truncated encoded-word
auto trunc = parse_content_disposition(R"(attachment; filename="=?UTF-8?B?4oKs")");
(void)trunc;
}
VT_TEST(cd_bad_percent_escapes_in_ext_value) {
// stray % and non-hex digits are emitted literally, no crash
auto cd = parse_content_disposition("attachment; filename*=UTF-8''%ZZ%%file%2");
VT_CHECK(cd.type == Type::attachment);
}
VT_TEST(cd_case_insensitive_keys_and_type) {
auto cd = parse_content_disposition(R"(ATTACHMENT; FileName="x.txt")");
VT_CHECK(cd.type == Type::attachment);
VT_CHECK_EQ(cd.filename, std::string("x.txt"));
}
VT_TEST(cd_unknown_type_is_other) {
auto cd = parse_content_disposition(R"(signal; filename="x.txt")");
VT_CHECK(cd.type == Type::other);
VT_CHECK_EQ(cd.filename, std::string("x.txt"));
}
+163
View File
@@ -0,0 +1,163 @@
#include "vdm/net/probe.hpp"
#include <chrono>
#include <future>
#include <string>
#include "testserver_fixture.hpp"
#include "vtest.hpp"
using namespace vdm;
using namespace vdm::net;
using vdm::testing::TestServer;
namespace {
Result<ProbeResult> run_probe(Prober &p, const std::string &url) {
std::promise<Result<ProbeResult>> prom;
auto fut = prom.get_future();
ProbeRequest req;
req.url = url;
req.overall_timeout_ms = 15000;
p.probe(std::move(req), [&](Result<ProbeResult> r) { prom.set_value(std::move(r)); });
if (fut.wait_for(std::chrono::seconds(25)) != std::future_status::ready)
return Err{Error::timeout, "probe test wait"};
return fut.get();
}
const std::string kEuro = "\xE2\x82\xAC";
} // namespace
VT_TEST(probe_plain_resumable_file) {
TestServer srv;
VT_REQUIRE(srv.available());
Prober p(4);
auto r = run_probe(p, srv.url("/plain/file/2M"));
VT_REQUIRE(r.has_value());
const auto &pr = r.value();
VT_CHECK_EQ(pr.http_status, 206); // proven via the ranged GET
VT_CHECK(pr.accept_ranges);
VT_CHECK(pr.resumable); // 206 + ETag validator
VT_REQUIRE(pr.total_size.has_value());
VT_CHECK_EQ(*pr.total_size, 2u * 1024 * 1024);
VT_CHECK(!pr.etag.empty());
VT_CHECK_EQ(pr.filename_from_url, std::string("2M"));
VT_CHECK(!pr.requires_auth);
}
VT_TEST(probe_no_range_server_is_not_resumable) {
TestServer srv;
VT_REQUIRE(srv.available());
Prober p(4);
auto r = run_probe(p, srv.url("/no-range/file/1M"));
VT_REQUIRE(r.has_value());
const auto &pr = r.value();
VT_CHECK(!pr.resumable); // no 206 ever
VT_CHECK(!pr.accept_ranges);
VT_REQUIRE(pr.total_size.has_value());
VT_CHECK_EQ(*pr.total_size, 1u * 1024 * 1024);
}
VT_TEST(probe_lying_accept_ranges_still_not_resumable) {
TestServer srv;
VT_REQUIRE(srv.available());
Prober p(4);
// advertises Accept-Ranges: bytes but never returns 206
auto r = run_probe(p, srv.url("/lies-about-accept-ranges/file/1M"));
VT_REQUIRE(r.has_value());
const auto &pr = r.value();
VT_CHECK(pr.accept_ranges); // it advertised
VT_CHECK(!pr.resumable); // ...but never proved it -> resume is off
}
VT_TEST(probe_reads_utf8_content_disposition) {
TestServer srv;
VT_REQUIRE(srv.available());
Prober p(4);
auto r = run_probe(p, srv.url("/utf8-content-disposition/file/8K"));
VT_REQUIRE(r.has_value());
VT_CHECK_EQ(r.value().filename_from_disposition, kEuro + " rates.pdf");
VT_CHECK_EQ(suggest_filename(r.value()), kEuro + " rates.pdf");
}
VT_TEST(probe_reads_legacy_content_disposition) {
TestServer srv;
VT_REQUIRE(srv.available());
Prober p(4);
auto r = run_probe(p, srv.url("/legacy-content-disposition/file/8K"));
VT_REQUIRE(r.has_value());
VT_CHECK(!r.value().filename_from_disposition.empty()); // decoded, not the raw =?...?=
VT_CHECK(r.value().filename_from_disposition.find("=?") == std::string::npos);
}
VT_TEST(probe_follows_redirect_and_reports_effective_url) {
TestServer srv;
VT_REQUIRE(srv.available());
Prober p(4);
auto r = run_probe(p, srv.url("/redirect-chain/file/64K"));
VT_REQUIRE(r.has_value());
const auto &pr = r.value();
VT_CHECK(pr.effective_url != srv.url("/redirect-chain/file/64K"));
VT_REQUIRE(pr.redirect_chain.size() >= 2);
VT_CHECK_EQ(pr.redirect_chain.front(), srv.url("/redirect-chain/file/64K"));
VT_CHECK_EQ(pr.redirect_chain.back(), pr.effective_url);
}
VT_TEST(probe_401_is_requires_auth_not_error) {
TestServer srv;
VT_REQUIRE(srv.available());
Prober p(4);
auto r = run_probe(p, srv.url("/401-basic/file/64K"));
VT_REQUIRE(r.has_value()); // success result...
VT_CHECK(r.value().requires_auth); // ...flagged for the credential dialog
}
VT_TEST(probe_404_is_an_error) {
TestServer srv;
VT_REQUIRE(srv.available());
Prober p(4);
auto r = run_probe(p, srv.url("/plain/nope"));
VT_REQUIRE(!r.has_value());
VT_CHECK_EQ(r.error().code, Error::not_found);
}
VT_TEST(probe_dead_host_is_connect_failed) {
Prober p(4);
std::promise<Result<ProbeResult>> prom;
auto fut = prom.get_future();
ProbeRequest req;
req.url = "http://127.0.0.1:1/x";
req.connect_timeout_ms = 2000;
p.probe(std::move(req), [&](Result<ProbeResult> r) { prom.set_value(std::move(r)); });
VT_REQUIRE(fut.wait_for(std::chrono::seconds(10)) == std::future_status::ready);
auto r = fut.get();
VT_REQUIRE(!r.has_value());
VT_CHECK_EQ(r.error().code, Error::connect_failed);
}
VT_TEST(probe_pool_serialises_excess_requests) {
TestServer srv;
VT_REQUIRE(srv.available());
Prober p(2); // only 2 at a time
constexpr int kN = 8;
std::vector<std::future<Result<ProbeResult>>> futs;
std::vector<std::promise<Result<ProbeResult>>> proms(kN);
for (int i = 0; i < kN; ++i) {
futs.push_back(proms[i].get_future());
ProbeRequest req;
req.url = srv.url("/plain/file/4K");
p.probe(std::move(req),
[pr = &proms[i]](Result<ProbeResult> r) { pr->set_value(std::move(r)); });
}
for (auto &f : futs) {
VT_REQUIRE(f.wait_for(std::chrono::seconds(30)) == std::future_status::ready);
VT_CHECK(f.get().has_value());
}
}
+71
View File
@@ -0,0 +1,71 @@
#include "vdm/net/url.hpp"
#include <string>
#include "vtest.hpp"
using vdm::net::split_url;
using vdm::net::SplitUrl;
using vdm::net::url_filename;
VT_TEST(url_basic_https) {
auto u = split_url("https://example.com/path/to/file.iso?x=1#frag");
VT_CHECK(u.valid);
VT_CHECK_EQ(u.scheme, std::string("https"));
VT_CHECK_EQ(u.host, std::string("example.com"));
VT_CHECK(!u.port.has_value());
VT_CHECK_EQ(u.path, std::string("/path/to/file.iso"));
VT_CHECK_EQ(u.query, std::string("x=1"));
VT_CHECK_EQ(u.fragment, std::string("frag"));
}
VT_TEST(url_port_userinfo_lowercasing) {
auto u = split_url("HTTP://User:[email protected]:8080/a");
VT_CHECK_EQ(u.scheme, std::string("http"));
VT_CHECK_EQ(u.userinfo, std::string("User:pw"));
VT_CHECK_EQ(u.host, std::string("host.example.com"));
VT_REQUIRE(u.port.has_value());
VT_CHECK_EQ(*u.port, 8080);
}
VT_TEST(url_ipv6_host) {
auto u = split_url("http://[2001:db8::1]:9000/file");
VT_CHECK_EQ(u.host, std::string("2001:db8::1"));
VT_REQUIRE(u.port.has_value());
VT_CHECK_EQ(*u.port, 9000);
}
VT_TEST(url_no_path) {
auto u = split_url("https://example.com");
VT_CHECK(u.valid);
VT_CHECK_EQ(u.path, std::string(""));
}
VT_TEST(url_non_http_is_invalid_but_parsed) {
auto u = split_url("ftp://host/file");
VT_CHECK(!u.valid); // not http/https
VT_CHECK_EQ(u.scheme, std::string("ftp"));
VT_CHECK(!u.is_http());
}
VT_TEST(url_garbage_does_not_crash) {
VT_CHECK(!split_url("").valid);
VT_CHECK(!split_url("not a url").valid);
VT_CHECK(!split_url("://noscheme/x").valid);
VT_CHECK(!split_url("http://").valid);
auto a = split_url("http://////");
auto b = split_url("https://h/%%%/%");
(void)a;
(void)b;
}
VT_TEST(url_filename_extraction) {
VT_CHECK_EQ(url_filename("https://x.com/a/b/report%20final.pdf"),
std::string("report final.pdf"));
VT_CHECK_EQ(url_filename("https://x.com/a/b/file.iso?sig=abc"), std::string("file.iso"));
VT_CHECK_EQ(url_filename("https://x.com/dir/"), std::string(""));
VT_CHECK_EQ(url_filename("https://x.com"), std::string(""));
VT_CHECK_EQ(url_filename("https://x.com/%2e%2e"), std::string("")); // ".." rejected
VT_CHECK_EQ(url_filename("https://x.com/a%2Fb"),
std::string("a_b")); // decoded '/' neutralised
}
+42
View File
@@ -0,0 +1,42 @@
# libFuzzer targets for CORE parsers. Lane CORE owns tools/fuzz.
#
# Self-guarding: the top-level CMakeLists.txt add_subdirectory()s every tools/* that has a
# CMakeLists, unconditionally, so this file must opt out on its own when fuzzing isn't
# wanted or the compiler can't do libFuzzer.
#
# Each target compiles the parser sources directly (not the whole libveloxcore) so the
# code under test is fully fuzzer-instrumented without a second build of the library.
if(NOT VELOX_BUILD_FUZZ)
return()
endif()
if(NOT CMAKE_CXX_COMPILER_ID MATCHES "Clang")
message(STATUS "tools/fuzz: libFuzzer needs Clang (have ${CMAKE_CXX_COMPILER_ID}); "
"skipping fuzz targets.")
return()
endif()
set(_core ${CMAKE_SOURCE_DIR}/core)
set(_fuzz_flags -g -O1 -fsanitize=fuzzer,address,undefined -fno-omit-frame-pointer)
function(vdm_add_fuzzer name)
add_executable(${name} ${ARGN})
target_include_directories(${name} PRIVATE ${_core}/include ${_core}/src)
target_compile_features(${name} PRIVATE cxx_std_23)
target_compile_options(${name} PRIVATE ${_fuzz_flags})
target_link_options(${name} PRIVATE ${_fuzz_flags})
endfunction()
vdm_add_fuzzer(fuzz_content_disposition
content_disposition_fuzz.cpp
${_core}/src/net/content_disposition.cpp
${_core}/src/net/text_codec.cpp)
vdm_add_fuzzer(fuzz_url
url_fuzz.cpp
${_core}/src/net/url.cpp
${_core}/src/net/text_codec.cpp)
# Seed corpora live next to the harnesses.
file(GLOB _cd_seeds ${CMAKE_CURRENT_SOURCE_DIR}/corpus/content_disposition/*)
file(GLOB _url_seeds ${CMAKE_CURRENT_SOURCE_DIR}/corpus/url/*)
+23
View File
@@ -0,0 +1,23 @@
// Fuzz target for the Content-Disposition parser (AGENT-CORE §3: mojibake source, needs a
// fuzz target). The parser must be total on any input — no crash, no UB, bounded work.
//
// clang++ -std=c++23 -fsanitize=fuzzer,address,undefined ... (see CMakeLists.txt)
// ./fuzz_content_disposition -max_len=4096 corpus/content_disposition/
#include <cstddef>
#include <cstdint>
#include <string_view>
#include "vdm/net/content_disposition.hpp"
extern "C" int LLVMFuzzerTestOneInput(const std::uint8_t *data, std::size_t size) {
std::string_view header(reinterpret_cast<const char *>(data), size);
auto cd = vdm::net::parse_content_disposition(header);
// Light invariants: a returned filename never contains a path separator or NUL.
for (char c : cd.filename)
if (c == '/' || c == '\\' || c == '\0')
__builtin_trap();
return 0;
}
@@ -0,0 +1 @@
attachment; filename*=UTF-8''%e2%82%ac%20rates.pdf
@@ -0,0 +1 @@
inline
@@ -0,0 +1 @@
attachment; filename="report.pdf"
@@ -0,0 +1 @@
attachment; filename="=?UTF-8?B?4oKsIHJhdGVzLnBkZg==?="
@@ -0,0 +1 @@
attachment; filename="../../etc/passwd"
+1
View File
@@ -0,0 +1 @@
https://example.com/path/to/file.iso?sig=abc#f
+1
View File
@@ -0,0 +1 @@
https://x.com/%2e%2e/%2f
+1
View File
@@ -0,0 +1 @@
ftp://host/x
+1
View File
@@ -0,0 +1 @@
http://[2001:db8::1]:9000/a%20b.bin
+24
View File
@@ -0,0 +1,24 @@
// Fuzz target for the URL splitter (AGENT-CORE stage 3: "a fuzz target ... URL parsing").
// split_url / url_filename must be total on any input.
#include <cstddef>
#include <cstdint>
#include <string_view>
#include "vdm/net/url.hpp"
extern "C" int LLVMFuzzerTestOneInput(const std::uint8_t *data, std::size_t size) {
std::string_view url(reinterpret_cast<const char *>(data), size);
auto u = vdm::net::split_url(url);
(void)u;
std::string name = vdm::net::url_filename(url);
for (char c : name)
if (c == '/' || c == '\\' || c == '\0')
__builtin_trap();
if (name == "." || name == "..")
__builtin_trap();
return 0;
}