core: net/probe + Content-Disposition parser + URL splitter (stage 3)

net/content_disposition — total parser for the mojibake-prone header:
RFC 6266 filename (quoted/token), RFC 5987 filename* ext-values
(charset'lang'pct-encoded, incl. RFC 2231 continuations), legacy RFC 2047
encoded-words (=?UTF-8?B?..?= / ?Q?), and raw Latin-1 bytes; prefers
filename* over filename; strips path components AFTER decoding (a base64
payload can hold '/'). 22-case test table.

net/text_codec (internal) — percent-decode, UTF-8 validation, Latin-1->
UTF-8, base64, RFC 2047 — shared by the CD parser and the URL splitter.

net/url — a small total URL splitter (scheme/userinfo/host/port/path/
query/fragment, http(s) validity) and url_filename() for the last path
segment; used for the filename fallback.

net/probe — HEAD then a ranged GET bytes=0-0 that PROVES resumability
(206 + matching Content-Range + a validator), rather than trusting
Accept-Ranges which servers lie about; the ranged GET is also the HEAD-
refused (403/405/501) fallback. 401/407 -> success result with
requires_auth, not an error. Runs on its own pool (max_concurrent,
default 4) outside the segment budget per ADR 0011 §5. suggest_filename()
does the resolution order (explicit -> disposition -> URL -> download.bin)
with a light strip; rules/ (stage 9) owns the authoritative sanitize.

tools/fuzz — libFuzzer targets for the CD parser and the URL splitter,
compiling the parser sources directly so they're fully instrumented;
self-guards on VELOX_BUILD_FUZZ + Clang (the top-level CMake adds every
tools/* unconditionally). Seed corpora included.

Fixed on the way: a p -> Transfer -> State -> cbs -> p reference cycle in
Prober that leaked every probe (drop the stored Transfer; the worker
keeps State alive). Tests green under ASan/UBSan and TSan.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HPPSGhiArbvQgwC2DNiURS
This commit is contained in:
2026-09-09 23:53:55 +04:00
co-authored by Claude Sonnet 5
parent fdacf732fa
commit 201ebc55d4
25 changed files with 1731 additions and 19 deletions
@@ -0,0 +1,42 @@
// vdm/net/content_disposition.hpp — parse a Content-Disposition header into a filename.
//
// This is a classic mojibake source (AGENT-CORE §3): RFC 6266 `filename`, RFC 5987
// `filename*` ext-values, RFC 2047 encoded-words in the legacy quoted form, and raw
// Latin-1 bytes all show up in the wild. The parser is total — hostile input yields a
// best-effort or empty result, never a throw or a crash — and has its own test table
// (content_disposition_test.cpp) and a fuzz target (tools/fuzz).
//
// This header compiles standalone.
#ifndef VDM_NET_CONTENT_DISPOSITION_HPP
#define VDM_NET_CONTENT_DISPOSITION_HPP
#include <string>
#include <string_view>
namespace vdm::net {
struct ContentDisposition {
enum class Type { none, inline_, attachment, form_data, other };
Type type = Type::none;
// Best-effort UTF-8 filename, path components stripped, or empty when the header
// carries none. NOT sanitized for the filesystem — that is rules/ (stage 9); this
// only decodes and de-mojibakes. `..` and control characters may still be present.
std::string filename;
// The filename came from an RFC 5987 `filename*` ext-value (preferred over a plain
// `filename` per RFC 6266 §4.3 when both are present).
bool filename_from_ext = false;
[[nodiscard]] bool is_attachment() const noexcept { return type == Type::attachment; }
[[nodiscard]] bool has_filename() const noexcept { return !filename.empty(); }
};
// Parse the value of a Content-Disposition header (everything after the colon).
[[nodiscard]] ContentDisposition parse_content_disposition(std::string_view header_value);
} // namespace vdm::net
#endif // VDM_NET_CONTENT_DISPOSITION_HPP
+90
View File
@@ -0,0 +1,90 @@
// vdm/net/probe.hpp — "what is at this URL?" without downloading it.
//
// Feeds the File Info dialog (docs/04 §2). HEAD first; a ranged GET `bytes=0-0` follows to
// PROVE resumability (a 206 with a matching Content-Range) rather than trust
// `Accept-Ranges`, which servers lie about (docs/06 R4). The ranged GET is also the
// fallback when HEAD is refused (403/405/501).
//
// Runs on its own small worker pool, sized outside the segment budget (ADR 0011 §5) so a
// burst of probes can't starve transfers and capture.offer's 750 ms path never waits on
// one.
//
// This header compiles standalone.
#ifndef VDM_NET_PROBE_HPP
#define VDM_NET_PROBE_HPP
#include <cstdint>
#include <functional>
#include <memory>
#include <optional>
#include <string>
#include <string_view>
#include <vector>
#include "vdm/net/content_disposition.hpp"
#include "vdm/net/http_types.hpp"
#include "vdm/util/result.hpp"
namespace vdm::net {
struct ProbeRequest {
std::string url;
std::vector<HeaderField> headers; // browser headers, verbatim
std::vector<Cookie> cookies;
std::string user_agent;
std::string referrer;
ProxyConfig proxy;
long connect_timeout_ms = 15000;
long overall_timeout_ms = 25000; // download.probe deadline is 30 s
};
struct ProbeResult {
std::string effective_url;
std::vector<std::string> redirect_chain; // requested URL first, effective_url last
long http_status = 0;
std::optional<std::uint64_t> total_size; // full-resource size, if known
std::string mime; // Content-Type value (params kept)
std::string etag;
std::string last_modified;
bool accept_ranges = false; // server advertised Accept-Ranges: bytes
bool resumable = false; // PROVEN: ranged GET -> 206 + matching Content-Range,
// and a validator (ETag or Last-Modified) is present
bool requires_auth = false; // a 401/407 was seen
// Decoded, path-stripped; NOT filesystem-sanitized (rules/ owns that, stage 9).
std::string filename_from_disposition;
std::string filename_from_url;
ContentDisposition::Type disposition_type = ContentDisposition::Type::none;
};
// Resolution order (docs/04 §2.5): explicit user name -> Content-Disposition -> URL path
// segment -> "download.bin". Only a light path/control strip here; rules/ does the
// authoritative sanitize, byte cap, and collision handling.
[[nodiscard]] std::string suggest_filename(const ProbeResult &r,
std::string_view explicit_name = {});
class Prober {
public:
// max_concurrent bounds outstanding probe transfers; the rest queue.
explicit Prober(unsigned max_concurrent = 4);
~Prober();
Prober(const Prober &) = delete;
Prober &operator=(const Prober &) = delete;
// Async. `done` runs on an internal worker thread, exactly once. A 401/407 is a
// SUCCESS result with requires_auth = true (the GUI collects credentials), not an
// error; transport failures and hard HTTP errors (404/410/5xx) are errors.
void probe(ProbeRequest req, std::function<void(Result<ProbeResult>)> done);
private:
struct Impl;
std::unique_ptr<Impl> impl_;
};
} // namespace vdm::net
#endif // VDM_NET_PROBE_HPP
+40
View File
@@ -0,0 +1,40 @@
// vdm/net/url.hpp — a small, total URL splitter.
//
// Not a full RFC 3986 parser (libcurl does the real fetching); just enough to pull a
// filename out of a path and to sanity-check a scheme. Total on hostile input — it has a
// fuzz target (tools/fuzz) — never throws, never asserts.
//
// This header compiles standalone.
#ifndef VDM_NET_URL_HPP
#define VDM_NET_URL_HPP
#include <cstdint>
#include <optional>
#include <string>
#include <string_view>
namespace vdm::net {
struct SplitUrl {
std::string scheme; // lowercased, without "://"
std::string userinfo; // before '@', if any
std::string host; // lowercased; bracketed IPv6 keeps its brackets stripped
std::optional<std::uint16_t> port;
std::string path; // includes the leading '/', or empty
std::string query; // without the '?'
std::string fragment; // without the '#'
bool valid = false; // scheme + host present and scheme is http/https
[[nodiscard]] bool is_http() const noexcept { return scheme == "http" || scheme == "https"; }
};
[[nodiscard]] SplitUrl split_url(std::string_view url);
// The last non-empty path segment, percent-decoded, path components stripped. Empty when
// the path has no usable segment (ends in '/', is empty, or is only "/").
[[nodiscard]] std::string url_filename(std::string_view url);
} // namespace vdm::net
#endif // VDM_NET_URL_HPP