sami/vdm
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

merge: lane/pkg-qa

Browse Source
This commit is contained in:
sami
2026-09-12 11:05:07 +04:00
parent 765c1701d4 efe76c319d
commit 963a76b6be
3 changed files with 524 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/ci.yml
+23
View File
@@ -4,6 +4,9 @@ on:
push:
branches: [main]
pull_request:
schedule:
- cron: '17 3 * * *' # nightly-integration only; every other job stays PR/push-triggered
workflow_dispatch: # lets a human fire nightly-integration on demand
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
@@ -225,3 +228,23 @@ jobs:
run: cmake --build --preset dev --target velox_conformance_cpp
- name: Run conformance (ctest -L conformance)
run: ctest --preset dev -L conformance --output-on-failure
nightly-integration:
# Real veloxd + tools/testserver, 50 concurrent downloads mixing hostile modes,
# every completed file's SHA-256 checked against testserver's own /sha256/ route,
# veloxd's open-FD count checked flat across the run. Nightly, not per-PR: it's
# ~2 minutes of real network I/O against a local server, not a schema check.
# See tests/integration/README.md#nightly-integration-run for what each assertion
# catches and the forced-failure transcript proving it isn't vacuous.
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Bootstrap toolchain
run: sudo ./tools/bootstrap.sh
- name: Configure
run: cmake --preset dev
- name: Build veloxd
run: cmake --build --preset dev --target veloxd
- name: Nightly integration run
run: python3 tests/integration/nightly_run.py --veloxd build/dev/bin/veloxd --tasks 50 --timeout 180
tests/integration/README.md
+83 -3
View File
@@ -4,6 +4,84 @@ Owned by PKG/QA. Real binaries against `tools/mockd` / `tools/testserver`, headl
Unit tests live in each lane; this tree is for behaviour that only shows up when the
pieces run together (throughput, memory over time, reconnect).
## Nightly integration run
`tests/integration/nightly_run.py`, wired as the `nightly-integration` job in `ci.yml`
(`schedule: '17 3 * * *'`, plus `workflow_dispatch` for an on-demand run). Real `veloxd`
+ `tools/testserver`, 50 concurrent downloads mixing hostile testserver modes
(`flaky-reset`, `throttled`, `no-range`, mostly `plain`), asserting:
1. every completed file's SHA-256 matches testserver's own `/<mode>/sha256/<size>` route
(computed by this script re-hashing the file on disk — never trusting veloxd's own
claim of success);
2. every task reaches a terminal state inside the timeout (a task stuck retrying forever
is a failure, not a hang for CI's `timeout(1)` to paper over);
3. `veloxd`'s own open-FD count (`/proc/<pid>/fd`) returns to within a small fixed
tolerance of its pre-run baseline, checked after a settle window.
`veloxd` runs isolated: `XDG_RUNTIME_DIR`, `XDG_DATA_HOME` and `XDG_CONFIG_HOME` all
point into a fresh `mkdtemp()` (not this session's scratch dir — its path is long enough
to overflow `AF_UNIX`'s ~108-byte `sun_path`; verified live via "File name too long"
before switching to `tempfile.mkdtemp()`). `saveTo.allowedRoots` is seeded straight into
`velox.db` after a migrations-only warm-up start: `settings.set` returns `-32603 "not
implemented in this build"` on the `veloxd` this job builds — verified live, not assumed
— so direct DB seeding is the only entry point that currently exists, not a workaround
for a wrong contract. Every task also gets its own `filename` override on `download.add`:
tasks share `(mode, size)` pairs by design (testserver's content is a pure function of
path, not of who's asking), and without distinct filenames they raced each other for the
same destination path — this was caught live on the first real run of this script (48/50
"passed" with io_errors and checksum mismatches on the collided tasks) before the
`filename` override was added.
### What makes each check go red, proven once
Per the standing note in this repo's history (four green checks that didn't look where
the bug was — an always-false guard, a `ctest` label matching zero tests, `--check`
validating pkg-config instead of apt names, conformance validating only fixtures): every
assertion here was forced red once, on purpose, before being trusted.
| Assertion | Forced via | Observed |
|---|---|---|
| Checksum match | `VDM_NIGHTLY_FORCE_BAD_HASH=1` (substitutes a wrong hash for task 0's comparison only, after the real download and hash succeed) | `FAIL: task 0 (plain, 256K) checksum mismatch: got 6f4c254c…, testserver says 0000…0000`, exit 1 |
| FD-leak tolerance | `VDM_NIGHTLY_FORCE_FD_LEAK=1` (adds 25 to the post-run FD count) | `FAIL: veloxd leaked file descriptors: 19 -> 49 (tolerance 10)`, exit 1 |
| A task that can't succeed still fails the run | `VDM_NIGHTLY_FORCE_HOSTILE_STALL=1` (task 0's URL points at 192.0.2.1, TEST-NET-1 — unroutable, so the connection just hangs) | `FAIL: task 0 (plain, 256K) ended in state 'failed', error={'code': 'timeout', ...}`, exit 1 |
The first version of the `HOSTILE_STALL` hook used testserver's `416-always` mode,
expecting it to never yield a 2xx to a Range probe — but `veloxd` correctly falls back
to a plain full GET on a 416 and the task completed fine, so that attempt proved
nothing (a real finding in itself: worth knowing the daemon handles this correctly).
Re-run any of the three whenever the corresponding assertion changes, to re-prove it
still catches what it claims to — that's the point of the hooks living in the script
rather than being one-off manual edits.
### Proven: the harness cannot leak its daemon
A run of this script was itself killed hard (its own harness process, not a graceful
stop) mid-download and left `veloxd` running under `systemd --user` for 4h40m — the
`finally:` teardown never got to run, because nothing runs after `SIGKILL`. Fixed by
giving every child (`veloxd` and `testserver.py`) `PR_SET_PDEATHSIG` (via a `preexec_fn`
calling `prctl` through `ctypes`) plus its own process group (`start_new_session=True`):
the kernel now delivers `SIGKILL` to a child the instant its parent dies, by any means,
without the harness needing to run any code at all. `finally:` still does the graceful
SIGTERM-then-SIGKILL `killpg` for the normal-exit path; `PR_SET_PDEATHSIG` is what
covers the path `finally:` cannot reach.
Proven live: started the harness, waited for its "baseline FDs" log line (proof the
real `veloxd` and `testserver.py` were both already up as separate process groups
under it), then `kill -9`'d the harness itself and confirmed both children were gone
within 1.5 s — nothing left running, nothing to clean up by hand.
### Known limitation: single-instance lock is per-euid, not per-XDG-tree
`veloxd`'s single-instance guard binds an abstract-namespace socket keyed only by
`geteuid()` (`daemon/src/main.cpp`), so `XDG_RUNTIME_DIR` isolation does not let two
`veloxd` processes for the same Unix user run side by side — confirmed live: a second
instance exits with "another instance is already running for this user" even with fully
distinct `XDG_*` dirs. Harmless on a real CI runner (one job, one user, one `veloxd`) but
means this script cannot run concurrently with another `veloxd` on the same machine —
worth knowing before parallelizing this job or running it by hand next to another lane's
manual testing.
## GUI M1 definition-of-done gates (R3)
`gui/docs/pkg-qa-requests-m1.md` R3: three GUI DoD items are not unit tests and have
@@ -85,11 +163,13 @@ Contract:
path: rss.json
```
`gui-dod-nightly` needs a `schedule:` trigger added to the top of `ci.yml` (there is none
today), or its own workflow file. Decide that when the harness lands.
`gui-dod-nightly` needs `if: github.event_name == 'schedule'` (the `nightly-integration`
job below already added that trigger to `ci.yml` — `cron: '17 3 * * *'` — so this no
longer needs its own).
### Status
Blocked on GUI's harness. Not urgent (GUI M1 DoD, not M0). When GUI files the follow-up
with the real `run.sh` path and the `rss-flat` slack number, PKG/QA drops the `TODO(GUI)`
markers, adds the `schedule:` trigger, and marks `gui-dod` required.
markers and marks `gui-dod` required. The `schedule:` trigger `gui-dod-nightly` needs is
already in `ci.yml`.
tests/integration/nightly_run.py
+418
View File
@@ -0,0 +1,418 @@
#!/usr/bin/env python3
"""tests/integration/nightly_run.py — the nightly integration gate.
Real veloxd + tools/testserver, 50 concurrent downloads mixing hostile testserver
modes, every completed file's SHA-256 checked against testserver's own /sha256/
route, and veloxd's open-FD count checked flat across the run. See
tests/integration/README.md#nightly-integration-run for what this proves and how
each failure mode was forced once to prove the check is not vacuous.
Standard library only (matches tools/testserver's own constraint) — this runs on
whatever CI image happens to have Python and does not need its own dependency
install step.
Usage:
python3 tests/integration/nightly_run.py --veloxd /path/to/build/dev/bin/veloxd
Exit codes: 0 all good, 1 a task failed or a checksum/FD assertion tripped,
2 usage/environment error (daemon or testserver would not start).
"""
from __future__ import annotations
import argparse
import ctypes
import hashlib
import json
import os
import re
import shutil
import signal
import socket
import sqlite3
import subprocess
import sys
import tempfile
import time
import urllib.request
from dataclasses import dataclass
from typing import NoReturn
REPO_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
TESTSERVER = os.path.join(REPO_ROOT, "tools", "testserver", "testserver.py")
_PR_SET_PDEATHSIG = 1
def _die_with_parent() -> None:
"""preexec_fn for every child we spawn: ask the kernel to SIGKILL this child the
moment its parent (this harness) goes away, by any means — including this
process's own SIGKILL, which no `finally:` block ever runs for. Verified live: a
prior run left a stranded veloxd running under systemd --user for 4h40m after the
harness was killed -9'd, because cleanup lived only in `finally:`. Proven fixed by
SIGKILL-ing a running harness mid-download and confirming both children exit with
it — see tests/integration/README.md.
ctypes, not python-prctl: this is stdlib-only by the same rule as testserver.py,
and it's one syscall.
"""
libc = ctypes.CDLL("libc.so.6", use_errno=True)
libc.prctl(_PR_SET_PDEATHSIG, signal.SIGKILL, 0, 0, 0)
# --- self-test hooks --------------------------------------------------------------
# Each of these deliberately breaks one assertion so the gate can be proven red on
# demand, without hand-editing the script. `tests/integration/README.md` records one
# transcript per flag from the session that added this file; re-run any of them
# whenever the corresponding assertion is touched, to re-prove it still catches the
# failure it claims to.
FORCE_BAD_HASH = os.environ.get("VDM_NIGHTLY_FORCE_BAD_HASH") == "1"
FORCE_FD_LEAK = os.environ.get("VDM_NIGHTLY_FORCE_FD_LEAK") == "1"
FORCE_HOSTILE_STALL = os.environ.get("VDM_NIGHTLY_FORCE_HOSTILE_STALL") == "1"
@dataclass
class TaskPlan:
idx: int
mode: str
size: str
task_id: str | None = None
url_override: str | None = None # self-test hook only; see FORCE_HOSTILE_STALL
@dataclass
class RpcClient:
sock: socket.socket
_next_id: int = 1
_buf: bytes = b""
def call(self, method: str, params: dict, timeout: float = 10.0) -> dict:
req = {"jsonrpc": "2.0", "id": self._next_id, "method": method, "params": params}
self._next_id += 1
self.sock.sendall((json.dumps(req) + "\n").encode())
self.sock.settimeout(timeout)
while b"\n" not in self._buf:
chunk = self.sock.recv(65536)
if not chunk:
raise ConnectionError(f"veloxd closed the connection mid-{method}")
self._buf += chunk
line, _, self._buf = self._buf.partition(b"\n")
return json.loads(line.decode())
def die(msg: str, code: int = 2) -> NoReturn:
print(f"nightly_run: {msg}", file=sys.stderr)
sys.exit(code)
def wait_for(pred, timeout: float, interval: float = 0.1, what: str = "condition"):
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if pred():
return
time.sleep(interval)
raise TimeoutError(f"timed out waiting for {what}")
def start_testserver(log_path: str) -> tuple[subprocess.Popen, int]:
proc = subprocess.Popen(
[sys.executable, TESTSERVER, "--port", "0"],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True,
bufsize=1,
preexec_fn=_die_with_parent,
start_new_session=True,
)
log = open(log_path, "w")
port = None
deadline = time.monotonic() + 10
while time.monotonic() < deadline:
line = proc.stdout.readline()
if not line:
break
log.write(line)
log.flush()
m = re.search(r"\b(\d{2,5})\b", line)
if m:
port = int(m.group(1))
break
if port is None:
proc.terminate()
die("testserver never printed its ephemeral port")
# Drain the rest of stdout into the log in the background so the pipe never fills.
def _drain():
for line in proc.stdout:
log.write(line)
log.flush()
import threading
threading.Thread(target=_drain, daemon=True).start()
return proc, port
def start_veloxd(veloxd_bin: str, env: dict, sock_path: str, log_path: str) -> subprocess.Popen:
proc = subprocess.Popen(
[veloxd_bin],
env=env,
stdout=open(log_path, "a"),
stderr=subprocess.STDOUT,
preexec_fn=_die_with_parent,
start_new_session=True,
)
try:
wait_for(lambda: os.path.exists(sock_path), timeout=10, what="veloxd's UDS socket")
except TimeoutError:
proc.terminate()
die(
"veloxd never created its socket — check the log at " + log_path,
)
return proc
def seed_settings(db_path: str, allowed_root: str, max_concurrent: int) -> None:
"""Pre-seed saveTo.allowedRoots (and raise the concurrency cap) directly in the
DB. settings.set is not implemented in this build yet (verified live: it returns
-32603 "not implemented in this build"), so this is the only way in — not a
workaround for a wrong contract, just the one entry point that currently exists.
"""
conn = sqlite3.connect(db_path)
try:
for key, value in (
("saveTo.allowedRoots", json.dumps([allowed_root])),
("saveTo.defaultDir", json.dumps(allowed_root)),
("connection.maxConcurrentDownloads", json.dumps(max_concurrent)),
("connection.maxActiveSegments", json.dumps(max_concurrent * 2)),
):
conn.execute(
"INSERT INTO settings(key, value) VALUES(?, ?) "
"ON CONFLICT(key) DO UPDATE SET value = excluded.value",
(key, value),
)
conn.commit()
finally:
conn.close()
def open_fd_count(pid: int) -> int:
return len(os.listdir(f"/proc/{pid}/fd"))
def build_plan(n: int, hostile_stall: bool) -> list[TaskPlan]:
# 60% plain, mixed in with the three named hostile modes from the brief.
modes = (
["plain"] * 30
+ ["flaky-reset"] * 8
+ ["throttled"] * 6
+ ["no-range"] * 6
)
assert len(modes) == n
plan = []
for i, mode in enumerate(modes):
# Small enough that even `throttled` (1 MiB/s default) finishes well inside
# the budget. Sizes repeat across tasks (only 8 distinct values) — that's
# fine for testserver, whose content is a function of (seed, path, size),
# but every task must still get its own `filename` override below: two
# tasks that share both mode and size share a URL, and without an override
# they'd share a destination filename too and race each other's rename —
# verified live, see tests/integration/README.md.
size = f"{256 + (i % 8) * 16}K"
t = TaskPlan(idx=i, mode=mode, size=size)
if hostile_stall and i == 0:
# Self-test hook, verified live: the first attempt at this used
# testserver's 416-always mode, expecting it to never yield a 2xx to a
# Range probe — but veloxd correctly falls back to a plain full GET on a
# 416, so that task completed fine and the hook proved nothing.
# TEST-NET-1 (RFC 5737) is unassigned and unroutable, so a connection
# attempt to it hangs until veloxd's own connection.timeoutSec — also
# verified live: it lands the task in 'failed' (retryable timeout) well
# inside this script's own --timeout, so this hook actually exercises the
# "reached a terminal state that isn't complete" branch below, not the
# "never reached one at all" branch — both are real ways a hostile source
# must not be allowed to read as a silent pass. See
# VDM_NIGHTLY_FORCE_HOSTILE_STALL in the README.
t.url_override = "http://192.0.2.1:1/black-hole"
plan.append(t)
return plan
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--veloxd", default=os.path.join(REPO_ROOT, "build", "dev", "bin", "veloxd"))
ap.add_argument("--tasks", type=int, default=50)
ap.add_argument("--timeout", type=float, default=120.0, help="seconds to wait for all tasks")
args = ap.parse_args()
if not os.path.isfile(args.veloxd):
die(f"veloxd binary not found at {args.veloxd} — build it first (cmake --build ...)")
# $TMPDIR, not the caller's scratch dir: AF_UNIX paths are capped at ~108 bytes
# (sun_path) and a deeply-nested scratch/session path blows that budget — verified
# live ("File name too long") before switching to mkdtemp() here.
base = tempfile.mkdtemp(prefix="vdm-nightly-")
print(f"nightly_run: workdir {base}")
ok = True
daemon = None
ts = None
try:
for d in ("runtime", "data", "config", "downloads"):
os.makedirs(os.path.join(base, d), exist_ok=True)
os.chmod(os.path.join(base, "runtime"), 0o700)
downloads_dir = os.path.join(base, "downloads")
env = os.environ.copy()
env["XDG_RUNTIME_DIR"] = os.path.join(base, "runtime")
env["XDG_DATA_HOME"] = os.path.join(base, "data")
env["XDG_CONFIG_HOME"] = os.path.join(base, "config")
sock_path = os.path.join(base, "runtime", "velox", "velox.sock")
veloxd_log = os.path.join(base, "veloxd.log")
# Warm-up start: apply migrations and create velox.db, nothing else.
warm = start_veloxd(args.veloxd, env, sock_path, veloxd_log)
warm.terminate()
warm.wait(timeout=5)
db_path = os.path.join(base, "data", "velox", "velox.db")
if not os.path.isfile(db_path):
die(f"veloxd's warm-up start never created {db_path}")
seed_settings(db_path, downloads_dir, max_concurrent=args.tasks)
ts, port = start_testserver(os.path.join(base, "testserver.log"))
daemon = start_veloxd(args.veloxd, env, sock_path, veloxd_log)
s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
s.connect(sock_path)
rpc = RpcClient(sock=s)
hello = rpc.call(
"session.hello",
{"clientType": "cli", "clientName": "nightly-run", "protocolVersion": "1.0.0"},
)
if "error" in hello:
die(f"session.hello failed: {hello['error']}")
baseline_fds = open_fd_count(daemon.pid)
print(f"nightly_run: veloxd baseline FDs = {baseline_fds}")
plan = build_plan(args.tasks, hostile_stall=FORCE_HOSTILE_STALL)
for t in plan:
url = t.url_override or f"http://127.0.0.1:{port}/{t.mode}/file/{t.size}"
# Every task gets its own filename: several tasks share a (mode, size) and
# therefore a URL, and without this override they'd race each other for the
# same destination path — verified live (see README).
filename = f"task-{t.idx:02d}-{t.mode}-{t.size}"
add = rpc.call(
"download.add",
{"url": url, "saveDir": downloads_dir, "segments": 4, "filename": filename},
)
if "error" in add:
ok = False
print(f"task {t.idx} ({t.mode}, {t.size}): download.add failed: {add['error']}")
continue
t.task_id = add["result"]["taskId"]
# --- poll every task to a terminal state -------------------------------------
terminal = {"complete", "failed", "cancelled"}
by_id = {t.task_id: t for t in plan if t.task_id}
results: dict[str, dict] = {}
deadline = time.monotonic() + args.timeout
while time.monotonic() < deadline and len(results) < len(by_id):
lst = rpc.call("download.list", {"limit": 5000})
if "error" in lst:
die(f"download.list failed: {lst['error']}")
for item in lst["result"]["items"]:
if item["taskId"] in by_id and item["state"] in terminal:
results[item["taskId"]] = item
time.sleep(0.5)
missing = set(by_id) - set(results)
if missing:
ok = False
print(f"FAIL: {len(missing)}/{len(by_id)} task(s) never reached a terminal state "
f"within {args.timeout}s: {sorted(missing)}")
# --- checksum: every completed file against testserver's own /sha256/ route -
for task_id, item in results.items():
t = by_id[task_id]
if item["state"] != "complete":
ok = False
err = item.get("error")
print(f"FAIL: task {t.idx} ({t.mode}, {t.size}) ended in state "
f"'{item['state']}', error={err}")
continue
path = os.path.join(item["saveDir"], item["filename"])
if not os.path.isfile(path):
ok = False
print(f"FAIL: task {t.idx} ({t.mode}, {t.size}) reported complete but "
f"{path} does not exist")
continue
h = hashlib.sha256()
with open(path, "rb") as f:
for chunk in iter(lambda: f.read(1 << 20), b""):
h.update(chunk)
actual = h.hexdigest()
expected_url = f"http://127.0.0.1:{port}/{t.mode}/sha256/{t.size}"
with urllib.request.urlopen(expected_url, timeout=5) as resp:
expected = json.loads(resp.read())["sha256"]
if FORCE_BAD_HASH and t.idx == 0:
# Self-test hook: corrupt the comparison itself, not the file, so a
# clean re-run without the env var proves nothing was left broken.
expected = "0" * 64
if actual != expected:
ok = False
print(f"FAIL: task {t.idx} ({t.mode}, {t.size}) checksum mismatch: "
f"got {actual}, testserver says {expected}")
# --- FD leak: veloxd's own /proc/<pid>/fd count returns to baseline ----------
# One settle window for the last connections/segment threads to actually
# close, then compare. A real leak grows roughly with the number of tasks
# that touched a socket, so a fixed small tolerance (not "not more than
# double") is the point of this check.
time.sleep(2.0)
final_fds = open_fd_count(daemon.pid)
if FORCE_FD_LEAK:
final_fds += 25 # self-test hook — see VDM_NIGHTLY_FORCE_FD_LEAK in the README
# Verified live across several 50-task runs: a clean run settles at baseline+3
# to baseline+5 (thread-pool and libcurl connection-cache bookkeeping that
# doesn't unwind instantly), so 4 alone was noisy enough to fail a clean run by
# chance. 10 still catches a real leak by a wide margin — a run with tasks stuck
# retrying showed baseline+7 before those tasks even reached their timeout.
tolerance = 10
print(f"nightly_run: veloxd FDs after run = {final_fds} (baseline {baseline_fds}, "
f"tolerance {tolerance})")
if final_fds > baseline_fds + tolerance:
ok = False
print(f"FAIL: veloxd leaked file descriptors: {baseline_fds} -> {final_fds} "
f"(tolerance {tolerance})")
print(f"nightly_run: {len(results)}/{len(by_id)} tasks reached a terminal state, "
f"{'OK' if ok else 'FAILURES ABOVE'}")
return 0 if ok else 1
finally:
# Belt-and-suspenders on top of PR_SET_PDEATHSIG: a graceful path here still
# prefers SIGTERM before SIGKILL. killpg (not kill) because start_new_session
# made each child its own process group leader, so this also reaps anything
# it spawned (testserver's --drain thread aside, which is ours, not a child).
for proc in (daemon, ts):
if proc is not None and proc.poll() is None:
try:
os.killpg(proc.pid, signal.SIGTERM)
except ProcessLookupError:
continue
try:
proc.wait(timeout=5)
except subprocess.TimeoutExpired:
try:
os.killpg(proc.pid, signal.SIGKILL)
except ProcessLookupError:
pass
shutil.rmtree(base, ignore_errors=True)
if __name__ == "__main__":
sys.exit(main())