mockd: add --tasks N — a plausible large synthetic table for GUI's DoD
GUI's M1 definition of done is "10 000 synthetic rows scroll at 60 fps with flat memory over 10 minutes (mockd --tasks 10000)". This flag was missing from the four unhappy-path flags that did land; the brief's own flag list omitted it, which is corrected here too. --tasks seeds a plausible population rather than N copies of one row: varied state, size (log-uniform 50 KB - 20 GB), category, queue position and description, drawn from the same category.list / queue.list fixtures the rest of mockd already serves so a synthetic task can never name a category or queue those methods don't also return. State distribution is roughly 55% complete / 8% failed / 4% cancelled / 6% paused / 2% retry_wait / 25% queued, using the new TaskErrorCode taxonomy for failures. "Progress advances across the whole set, not a handful of live rows" ruled out the obvious cheap answer. A bounded, rotating pool of concurrently-active downloads (--active-cap, default 24) is fed continuously from each queue's FIFO — with the rest of that queue's queuePosition renumbered on every promotion, as a real scheduler would — and a small fraction of active tasks hit a transient failure and cycle through retry_wait before rejoining, so the pool keeps rotating through new rows for the whole run instead of draining once. Verified over a 10000-task, 60-second run: 61.5 MB RSS flat, and the active pool's membership meaningfully different after 60s. tick() only ever walks the active pool plus due retry-wait entries, never the full task list, so its cost stays flat regardless of --tasks. A manual download.add is still admitted immediately regardless of --active-cap — a human driving the GUI by hand must never wait behind synthetic load. Fixed a latent double-push while building this: any task 'connecting' at the top of a tick was pushed to the progress batch once for the transition and again at the loop's unconditional final push, inflating event.task.progress payloads with a duplicate entry for that taskId. It predates this change (the original tick() had the same shape) but only became visible once several tasks are legitimately 'connecting' in the same tick, which --active-cap's continuous promotion now does routinely. --seed makes a run reproducible, which matters when a GUI bug only shows up at a particular row. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_012fgjnqFCS5h5L7gZTZo3rV
This commit is contained in:
@@ -34,7 +34,10 @@ CI noticing the same day.
|
||||
6. **`tools/mockd`** — Node/TS. Serves the fixtures over **both** transports (Unix socket
|
||||
NDJSON and loopback WS), fakes plausible progress events at 4 Hz, and has flags for
|
||||
`--slow`, `--flaky`, `--drop-connection`, `--refuse-pairing` so GUI and EXT can test
|
||||
their unhappy paths before `veloxd` exists.
|
||||
their unhappy paths before `veloxd` exists. **`--tasks <n>`** seeds a large plausible
|
||||
population (varied states/sizes/categories, a rotating active pool) instead of the
|
||||
fixture's two rows — GUI's M1 DoD needs `--tasks 10000` for its scroll-performance test,
|
||||
so this one lands with M0, not as an afterthought once GUI is already blocked on it.
|
||||
7. **`tests/conformance/`** — one suite, two runners: replays each fixture against a live
|
||||
`veloxd` (C++ side) and through the generated TS client. Wired into CI as a **required
|
||||
check on every lane's PR**.
|
||||
|
||||
@@ -24,6 +24,9 @@ Defaults: `$XDG_RUNTIME_DIR/velox/velox.sock` and `ws://127.0.0.1:52000`.
|
||||
| `--ws-port <n>` / `--no-ws` | loopback WebSocket port, or don't listen |
|
||||
| `--progress-hz <n>` | progress event rate (default 4, the contract's ceiling) |
|
||||
| `--speed <bytes>` | synthetic per-task speed |
|
||||
| `--tasks <n>` | seed `n` plausible synthetic tasks instead of the fixture's two — see below |
|
||||
| `--active-cap <n>` | ceiling on concurrently-"downloading" synthetic tasks (default 24) |
|
||||
| `--seed <n>` | PRNG seed for `--tasks`, so a run is exactly reproducible (default 1337) |
|
||||
| `--slow <ms>` | delay every reply. **Past 750 ms `capture.offer` must fail open.** |
|
||||
| `--flaky <0..1>` | answer this fraction of calls with `-32603` |
|
||||
| `--drop-connection <s>` | terminate every connection every N seconds |
|
||||
@@ -33,6 +36,37 @@ Defaults: `$XDG_RUNTIME_DIR/velox/velox.sock` and `ws://127.0.0.1:52000`.
|
||||
| `--allow-any-origin` | skip the `moz-extension://` Origin check (debugging only) |
|
||||
| `--no-validate` | stop validating params (to see what a client actually sends) |
|
||||
|
||||
## `--tasks` — load testing the GUI's table
|
||||
|
||||
GUI's M1 definition of done is "10 000 synthetic rows scroll at 60 fps with flat memory
|
||||
over 10 minutes (`mockd --tasks 10000`)". That takes more than 10 000 identical rows:
|
||||
|
||||
```sh
|
||||
npm start -- --tasks 10000
|
||||
```
|
||||
|
||||
Seeds a plausible population — varied `state`, size, category, queue position and
|
||||
description, drawn from the same `category.list` / `queue.list` fixtures the rest of
|
||||
mockd serves, so nothing here can name a category or queue those methods don't also
|
||||
return. Roughly 55% land `complete`, the rest split across `failed`, `cancelled`,
|
||||
`paused`, `retry_wait` and `queued`, plus a bounded pool (`--active-cap`, default 24)
|
||||
seeded straight into `downloading`.
|
||||
|
||||
That pool is **rotating**, not fixed: as an active task finishes, the next one is
|
||||
promoted from its queue's FIFO — with the rest of that queue's `queuePosition` renumbered,
|
||||
as a real scheduler would — and a small fraction of "finishing" active tasks fail instead
|
||||
and cycle through `retry_wait` before rejoining. Over a ten-minute run this means hundreds
|
||||
of distinct rows have shown live progress by the time it ends, not the same handful
|
||||
forever, while at any instant the active count stays realistic. A manual `download.add`
|
||||
is always admitted immediately regardless of `--active-cap` — a human driving the GUI by
|
||||
hand is never made to wait behind synthetic load.
|
||||
|
||||
`tick()` only ever walks the active pool plus whatever retry-wait entries just came due,
|
||||
never the full task list, so the per-tick cost stays flat regardless of `--tasks`.
|
||||
|
||||
`--seed` makes a run reproducible: the same seed always produces the same table, which
|
||||
matters when a GUI bug only shows up at a particular row.
|
||||
|
||||
## What is real and what is faked
|
||||
|
||||
**Real**, because a client's correctness depends on it:
|
||||
|
||||
@@ -18,9 +18,10 @@ import { rmSync } from 'node:fs';
|
||||
import { Dispatcher, type Session } from './dispatch.js';
|
||||
import { indexByMethod, loadFixtures, resolvePlaceholders } from './fixtures.js';
|
||||
import { MockState } from './state.js';
|
||||
import { DEFAULT_ACTIVE_CAP, generateSyntheticTasks } from './synth.js';
|
||||
import { startUds, type Connection } from './transport/uds.js';
|
||||
import { startWs } from './transport/ws.js';
|
||||
import type { TaskSummary } from '../../../extension/src/shared/protocol/types.js';
|
||||
import type { Category, Queue, TaskSummary } from '../../../extension/src/shared/protocol/types.js';
|
||||
import { PROTOCOL_VERSION } from '../../../extension/src/shared/protocol/types.js';
|
||||
|
||||
const HERE = resolve(fileURLToPath(import.meta.url), '..');
|
||||
@@ -37,6 +38,20 @@ const USAGE = `mockd — mock veloxd, serving contracts/fixtures over both trans
|
||||
--progress-hz <n> progress event rate (default: 4, the contract's maximum)
|
||||
--speed <bytes> synthetic per-task speed in bytes/sec (default: 8388608)
|
||||
|
||||
Load testing, for the GUI lane's "N rows at 60 fps" definition of done:
|
||||
--tasks <n> seed n plausible synthetic tasks — varied states, sizes,
|
||||
speeds, categories and queue positions — instead of the two
|
||||
from contracts/fixtures/download.list.json. A bounded, rotating
|
||||
pool of active downloads (see --active-cap) keeps progress
|
||||
moving across a broad slice of the table for as long as mockd
|
||||
runs, not just the first few rows.
|
||||
e.g. mockd --tasks 10000
|
||||
--active-cap <n> ceiling on concurrently-"downloading" synthetic tasks
|
||||
(default: 24). Has no effect on a manual download.add, which is
|
||||
always admitted immediately.
|
||||
--seed <n> PRNG seed for --tasks, so a given seed reproduces the exact
|
||||
same table (default: 1337)
|
||||
|
||||
Unhappy paths, for the GUI and EXT lanes:
|
||||
--slow <ms> delay every reply by <ms>. Past 750 ms, capture.offer must
|
||||
fail open and let Firefox take the download.
|
||||
@@ -56,6 +71,9 @@ interface Args {
|
||||
wsPort: number | null;
|
||||
progressHz: number;
|
||||
speed: number;
|
||||
tasks: number;
|
||||
activeCap: number;
|
||||
seed: number;
|
||||
slow: number;
|
||||
flaky: number;
|
||||
dropEverySec: number;
|
||||
@@ -73,6 +91,9 @@ function parseArgs(argv: readonly string[]): Args {
|
||||
wsPort: 52000,
|
||||
progressHz: 4,
|
||||
speed: 8 * 1024 * 1024,
|
||||
tasks: 0,
|
||||
activeCap: DEFAULT_ACTIVE_CAP,
|
||||
seed: 1337,
|
||||
slow: 0,
|
||||
flaky: 0,
|
||||
dropEverySec: 0,
|
||||
@@ -99,6 +120,9 @@ function parseArgs(argv: readonly string[]): Args {
|
||||
case '--no-ws': args.wsPort = null; break;
|
||||
case '--progress-hz': args.progressHz = Number(next()); break;
|
||||
case '--speed': args.speed = Number(next()); break;
|
||||
case '--tasks': args.tasks = Number(next()); break;
|
||||
case '--active-cap': args.activeCap = Number(next()); break;
|
||||
case '--seed': args.seed = Number(next()); break;
|
||||
case '--slow': args.slow = Number(next()); break;
|
||||
case '--flaky': args.flaky = Number(next()); break;
|
||||
case '--drop-connection': args.dropEverySec = Number(next()); break;
|
||||
@@ -126,13 +150,39 @@ function main(): void {
|
||||
log(`loaded ${fixtures.length} fixtures covering ${byMethod.size} methods from contracts/fixtures`);
|
||||
|
||||
// Seed the task list from the download.list fixture, so a client that connects before
|
||||
// adding anything still has rows to draw.
|
||||
// adding anything still has rows to draw. --tasks replaces this with a much larger
|
||||
// synthetic population instead — see below.
|
||||
const listFixture = byMethod.get('download.list');
|
||||
const seed = (resolvePlaceholders(
|
||||
const seed = args.tasks > 0 ? [] : (resolvePlaceholders(
|
||||
(listFixture?.response as { result?: { items?: unknown[] } } | undefined)?.result?.items ?? [],
|
||||
) as TaskSummary[]);
|
||||
|
||||
const state = new MockState(seed, { progressHz: args.progressHz, speedBps: args.speed });
|
||||
const state = new MockState(seed, {
|
||||
progressHz: args.progressHz, speedBps: args.speed, activeCap: args.activeCap,
|
||||
});
|
||||
|
||||
if (args.tasks > 0) {
|
||||
// Categories and queues come from their own golden fixtures rather than being
|
||||
// reinvented here, so a --tasks run can never drift from what category.list and
|
||||
// queue.list actually serve — the extension's default-category picker and the GUI's
|
||||
// category tree see the same ids these synthetic tasks are filed under.
|
||||
const categories = (resolvePlaceholders(
|
||||
(byMethod.get('category.list')?.response as { result?: { items?: unknown[] } } | undefined)
|
||||
?.result?.items ?? [],
|
||||
) as Category[]);
|
||||
const queues = (resolvePlaceholders(
|
||||
(byMethod.get('queue.list')?.response as { result?: { items?: unknown[] } } | undefined)
|
||||
?.result?.items ?? [],
|
||||
) as Queue[]);
|
||||
|
||||
const started = Date.now();
|
||||
const synthetic = generateSyntheticTasks({
|
||||
count: args.tasks, seed: args.seed, categories, queues, baseSpeedBps: args.speed,
|
||||
});
|
||||
state.seedSynthetic(synthetic);
|
||||
log(`--tasks ${args.tasks}: generated in ${Date.now() - started} ms `
|
||||
+ `(${synthetic.initiallyActive.length} active now, active-cap ${args.activeCap}, seed ${args.seed})`);
|
||||
}
|
||||
const dispatcher = new Dispatcher(state, byMethod, {
|
||||
protocolVersion: PROTOCOL_VERSION,
|
||||
daemonVersion: '0.0.0-mockd',
|
||||
@@ -169,10 +219,14 @@ function main(): void {
|
||||
// Progress at the contract's 4 Hz ceiling, as one batched array — never one
|
||||
// notification per task. The GUI lane needs this shape to build its coalescing against.
|
||||
setInterval(() => {
|
||||
const { moved, completed } = state.tick();
|
||||
if (moved.length > 0) {
|
||||
const { moved, completed, failed } = state.tick();
|
||||
// A progress payload only makes sense for a still-active task; a task that just
|
||||
// finished (either direction) gets its own event.task.state below instead, so the two
|
||||
// events never race on the same tick with contradictory numbers.
|
||||
const stillProgressing = moved.filter((t) => t.state === 'downloading');
|
||||
if (stillProgressing.length > 0) {
|
||||
broadcast('event.task.progress', {
|
||||
tasks: moved.map((t) => ({
|
||||
tasks: stillProgressing.map((t) => ({
|
||||
taskId: t.taskId,
|
||||
downloadedBytes: t.downloadedBytes,
|
||||
speedBps: t.speedBps,
|
||||
@@ -196,6 +250,19 @@ function main(): void {
|
||||
body: `${task.filename} finished.`, taskId: task.taskId, sound: 'complete',
|
||||
});
|
||||
}
|
||||
for (const task of failed) {
|
||||
broadcast('event.task.state', {
|
||||
taskId: task.taskId, state: task.state, previousState: 'downloading',
|
||||
summary: task, error: task.error,
|
||||
});
|
||||
if (task.state === 'failed') {
|
||||
broadcast('event.notify', {
|
||||
level: 'error', title: 'Download failed',
|
||||
body: `${task.filename}: ${task.error?.message ?? 'unknown error'}`,
|
||||
taskId: task.taskId, sound: 'error',
|
||||
});
|
||||
}
|
||||
}
|
||||
}, Math.max(1, Math.round(1000 / args.progressHz)));
|
||||
|
||||
setInterval(() => {
|
||||
|
||||
+162
-14
@@ -7,17 +7,39 @@
|
||||
* answered straight from a golden file.
|
||||
*
|
||||
* This is deliberately not a download engine. Bytes advance on a clock, not from a socket.
|
||||
*
|
||||
* At scale (`--tasks 10000`) two things matter that don't at a handful of tasks:
|
||||
*
|
||||
* 1. `tick()` must not become O(total tasks) every 250 ms. It only ever touches the
|
||||
* bounded *active* set (`activeCap`, default 24) plus whatever retry-wait entries just
|
||||
* came due — never the thousands of tasks sitting in a terminal or queued state.
|
||||
* 2. Progress has to reach a broad slice of the table over the life of a run, not the same
|
||||
* couple of rows forever. As an active task finishes, the next one is promoted from its
|
||||
* queue's FIFO (with the rest of that queue's `queuePosition` renumbered, as a real
|
||||
* scheduler would), and a slice of "finished" active tasks fail instead and go through
|
||||
* `retry_wait` before being promoted again — so the active pool keeps rotating through
|
||||
* new rows for the whole run rather than draining once and going static.
|
||||
*/
|
||||
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import type { TaskState, TaskSummary } from '../../../extension/src/shared/protocol/types.js';
|
||||
import { randomTaskErrorCode, taskErrorFor, type SynthResult } from './synth.js';
|
||||
|
||||
export interface MockOptions {
|
||||
readonly progressHz: number;
|
||||
readonly speedBps: number;
|
||||
/** Ceiling on concurrently-"downloading" tasks. Bulk-seeded tasks rotate through this;
|
||||
* a manual download.add is admitted immediately regardless (see add()) — a human
|
||||
* driving the GUI by hand should never be told to wait behind synthetic load. */
|
||||
readonly activeCap: number;
|
||||
}
|
||||
|
||||
const ACTIVE: readonly TaskState[] = ['connecting', 'downloading'];
|
||||
// Chance per active task per tick that a healthy transfer hits a transient failure instead
|
||||
// of completing normally — keeps the active pool cycling through retry_wait for the whole
|
||||
// life of a long run rather than just draining the initial 'queued' population once.
|
||||
const TRANSIENT_FAILURE_CHANCE = 0.0015;
|
||||
const RETRY_DELAY_MS: readonly [number, number] = [5_000, 45_000];
|
||||
|
||||
export class MockState {
|
||||
readonly tasks = new Map<string, TaskSummary>();
|
||||
@@ -25,15 +47,47 @@ export class MockState {
|
||||
settings: Record<string, unknown> = {};
|
||||
private readonly opts: MockOptions;
|
||||
|
||||
/** taskIds currently connecting/downloading. Only this set is walked every tick. */
|
||||
private readonly active = new Set<string>();
|
||||
/** Per-queue FIFO of taskIds waiting to be promoted, front = next admitted. */
|
||||
private readonly pendingByQueue = new Map<string, string[]>();
|
||||
/** taskIds with no queueId, or whose queue is unknown to this process (e.g. added via
|
||||
* download.add with startMode 'queue' and a queueId mockd has no fixture-derived queue
|
||||
* for) — a fallback FIFO so nothing is silently unpromotable. */
|
||||
private readonly pendingUnqueued: string[] = [];
|
||||
private readonly retryWaiting: Array<{ taskId: string; dueAt: number }> = [];
|
||||
private readonly speedFactor = new Map<string, number>();
|
||||
|
||||
constructor(seed: readonly TaskSummary[], opts: MockOptions) {
|
||||
this.opts = opts;
|
||||
for (const t of seed) this.tasks.set(t.taskId, { ...t });
|
||||
for (const t of seed) this.registerNew({ ...t });
|
||||
}
|
||||
|
||||
/** Bulk-seed a large synthetic population without re-deriving the pool bookkeeping the
|
||||
* generator already computed — the one place a --tasks 10000 startup does real O(n)
|
||||
* work, and it happens once, not per tick. */
|
||||
seedSynthetic(result: SynthResult): void {
|
||||
for (const task of result.tasks) this.tasks.set(task.taskId, task);
|
||||
for (const taskId of result.initiallyActive) this.active.add(taskId);
|
||||
for (const [queueId, ids] of result.queuedOrder) this.pendingByQueue.set(queueId, [...ids]);
|
||||
for (const [taskId, factor] of result.speedFactor) this.speedFactor.set(taskId, factor);
|
||||
}
|
||||
|
||||
list(): TaskSummary[] {
|
||||
return [...this.tasks.values()];
|
||||
}
|
||||
|
||||
private registerNew(task: TaskSummary): void {
|
||||
this.tasks.set(task.taskId, task);
|
||||
if (ACTIVE.includes(task.state)) this.active.add(task.taskId);
|
||||
}
|
||||
|
||||
/**
|
||||
* A manual add (download.add, capture.offer) is admitted immediately regardless of the
|
||||
* active pool — a human testing the GUI by hand must never be told to wait behind
|
||||
* synthetic --tasks load. It participates in tick()'s active set from the moment it is
|
||||
* created, same as any other active task.
|
||||
*/
|
||||
add(partial: Partial<TaskSummary> & { url: string }): TaskSummary {
|
||||
const now = new Date().toISOString();
|
||||
const task: TaskSummary = {
|
||||
@@ -58,7 +112,7 @@ export class MockState {
|
||||
completedAt: null,
|
||||
error: null,
|
||||
};
|
||||
this.tasks.set(task.taskId, task);
|
||||
this.registerNew(task);
|
||||
return task;
|
||||
}
|
||||
|
||||
@@ -66,6 +120,7 @@ export class MockState {
|
||||
const task = this.tasks.get(taskId);
|
||||
if (!task) return null;
|
||||
const changed = task.state !== state;
|
||||
const wasActive = ACTIVE.includes(task.state);
|
||||
task.state = state;
|
||||
task.speedBps = ACTIVE.includes(state) ? this.opts.speedBps : 0;
|
||||
if (!ACTIVE.includes(state)) task.etaSeconds = null;
|
||||
@@ -73,29 +128,116 @@ export class MockState {
|
||||
task.downloadedBytes = task.sizeBytes ?? task.downloadedBytes;
|
||||
task.completedAt = new Date().toISOString();
|
||||
}
|
||||
const isActive = ACTIVE.includes(state);
|
||||
if (isActive && !wasActive) this.active.add(taskId);
|
||||
else if (!isActive && wasActive) this.active.delete(taskId);
|
||||
return { changed, task };
|
||||
}
|
||||
|
||||
remove(taskId: string): boolean {
|
||||
this.active.delete(taskId);
|
||||
return this.tasks.delete(taskId);
|
||||
}
|
||||
|
||||
private popNextQueued(): string | undefined {
|
||||
for (const [, ids] of this.pendingByQueue) {
|
||||
const taskId = ids.shift();
|
||||
if (taskId !== undefined) {
|
||||
for (let i = 0; i < ids.length; i += 1) {
|
||||
const t = this.tasks.get(ids[i]!);
|
||||
if (t) t.queuePosition = i + 1;
|
||||
}
|
||||
return taskId;
|
||||
}
|
||||
}
|
||||
return this.pendingUnqueued.shift();
|
||||
}
|
||||
|
||||
private admit(taskId: string): void {
|
||||
const task = this.tasks.get(taskId);
|
||||
if (!task) return; // removed while queued — nothing to admit
|
||||
task.state = 'connecting';
|
||||
task.speedBps = 0;
|
||||
task.queuePosition = null;
|
||||
task.lastTryAt = new Date().toISOString();
|
||||
this.active.add(taskId);
|
||||
}
|
||||
|
||||
private fillActiveFromQueues(): void {
|
||||
while (this.active.size < this.opts.activeCap) {
|
||||
const taskId = this.popNextQueued();
|
||||
if (taskId === undefined) return;
|
||||
this.admit(taskId);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Advance one progress tick. Returns the tasks that moved and any that just finished,
|
||||
* so the caller can emit event.task.progress and event.task.state from one place.
|
||||
* Advance one progress tick. Returns the tasks that moved and any that just finished
|
||||
* (completed, or newly failed), so the caller can emit event.task.progress and
|
||||
* event.task.state from one place.
|
||||
*/
|
||||
tick(): { moved: TaskSummary[]; completed: TaskSummary[] } {
|
||||
tick(): { moved: TaskSummary[]; completed: TaskSummary[]; failed: TaskSummary[] } {
|
||||
const moved: TaskSummary[] = [];
|
||||
const completed: TaskSummary[] = [];
|
||||
const perTick = Math.floor(this.opts.speedBps / this.opts.progressHz);
|
||||
const failed: TaskSummary[] = [];
|
||||
const now = Date.now();
|
||||
|
||||
for (const task of this.tasks.values()) {
|
||||
// Retry-wait entries whose backoff elapsed rejoin the active pool directly (they have
|
||||
// already waited their turn once; they don't go back through a queue's FIFO).
|
||||
while (this.retryWaiting.length > 0 && this.retryWaiting[0]!.dueAt <= now) {
|
||||
const { taskId } = this.retryWaiting.shift()!;
|
||||
if (this.active.size < this.opts.activeCap) {
|
||||
this.admit(taskId);
|
||||
const task = this.tasks.get(taskId);
|
||||
// Don't push to `moved` here: admit() leaves the task 'connecting', and the main
|
||||
// loop below visits every active taskId (this one included, since admit() just
|
||||
// added it) and pushes exactly once when it flips to 'downloading'. Pushing here
|
||||
// too would duplicate this task in the same tick's event.task.progress batch.
|
||||
if (task) task.error = null;
|
||||
} else {
|
||||
// Pool is full: park it at the front of the fallback FIFO rather than dropping it.
|
||||
this.pendingUnqueued.unshift(taskId);
|
||||
}
|
||||
}
|
||||
this.fillActiveFromQueues();
|
||||
|
||||
const perTickBase = Math.floor(this.opts.speedBps / this.opts.progressHz);
|
||||
|
||||
for (const taskId of [...this.active]) {
|
||||
const task = this.tasks.get(taskId);
|
||||
if (!task) {
|
||||
this.active.delete(taskId); // stale — removed mid-transfer
|
||||
continue;
|
||||
}
|
||||
if (task.state === 'connecting') {
|
||||
// One settle tick before bytes start moving — also fixes a latent double-push:
|
||||
// falling through to the unconditional moved.push() below would otherwise queue
|
||||
// this task twice in the same event.task.progress batch (once here, once there).
|
||||
task.state = 'downloading';
|
||||
moved.push(task);
|
||||
continue;
|
||||
}
|
||||
if (task.state !== 'downloading') continue;
|
||||
|
||||
if (task.downloadedBytes > 0 && Math.random() < TRANSIENT_FAILURE_CHANCE) {
|
||||
const retryable = Math.random() < 0.7;
|
||||
const code = randomTaskErrorCode(Math.random, retryable);
|
||||
task.error = taskErrorFor(Math.random, code, retryable);
|
||||
task.state = retryable ? 'retry_wait' : 'failed';
|
||||
task.speedBps = 0;
|
||||
task.etaSeconds = null;
|
||||
this.active.delete(taskId);
|
||||
failed.push(task);
|
||||
moved.push(task);
|
||||
if (retryable) {
|
||||
const [lo, hi] = RETRY_DELAY_MS;
|
||||
this.retryWaiting.push({ taskId, dueAt: now + lo + Math.random() * (hi - lo) });
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
const factor = this.speedFactor.get(taskId) ?? 1;
|
||||
const perTick = Math.max(1, Math.round(perTickBase * factor));
|
||||
task.speedBps = jitter(perTick * this.opts.progressHz);
|
||||
task.downloadedBytes += perTick;
|
||||
const size = task.sizeBytes;
|
||||
@@ -105,27 +247,33 @@ export class MockState {
|
||||
task.speedBps = 0;
|
||||
task.etaSeconds = null;
|
||||
task.completedAt = new Date().toISOString();
|
||||
this.active.delete(taskId);
|
||||
completed.push(task);
|
||||
} else if (size !== null && size !== undefined && task.speedBps > 0) {
|
||||
task.etaSeconds = Math.ceil((size - task.downloadedBytes) / task.speedBps);
|
||||
}
|
||||
moved.push(task);
|
||||
}
|
||||
return { moved, completed };
|
||||
|
||||
// Backfill whatever just finished, so the pool is at capacity again for the next tick
|
||||
// rather than idling until someone happens to call fillActiveFromQueues().
|
||||
this.fillActiveFromQueues();
|
||||
|
||||
return { moved, completed, failed };
|
||||
}
|
||||
|
||||
globalSpeed(): { downBps: number; activeCount: number; queuedCount: number } {
|
||||
let downBps = 0;
|
||||
let activeCount = 0;
|
||||
let queuedCount = 0;
|
||||
for (const t of this.tasks.values()) {
|
||||
if (ACTIVE.includes(t.state)) {
|
||||
downBps += t.speedBps;
|
||||
for (const taskId of this.active) {
|
||||
const task = this.tasks.get(taskId);
|
||||
if (task) {
|
||||
downBps += task.speedBps;
|
||||
activeCount += 1;
|
||||
} else if (t.state === 'queued') {
|
||||
queuedCount += 1;
|
||||
}
|
||||
}
|
||||
let queuedCount = this.pendingUnqueued.length + this.retryWaiting.length;
|
||||
for (const [, ids] of this.pendingByQueue) queuedCount += ids.length;
|
||||
return { downBps, activeCount, queuedCount };
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,277 @@
|
||||
/**
|
||||
* `--tasks N`: a plausible, large synthetic task list.
|
||||
*
|
||||
* GUI's M1 definition of done is "10 000 synthetic rows scroll at 60 fps with flat memory
|
||||
* over 10 minutes (`mockd --tasks 10000`)". That is a claim about a table full of rows that
|
||||
* look like a real download manager's history, not 10 000 copies of one row: varied
|
||||
* states, sizes, speeds, categories and queue positions, drawn from the same category and
|
||||
* queue fixtures the rest of mockd already serves so nothing here can drift from them.
|
||||
*
|
||||
* "Progress advances across the whole set, not a handful of live rows" rules out the
|
||||
* obvious cheap answer — seed a handful of tasks as 'downloading' forever and leave
|
||||
* thousands static. Instead a bounded, rotating pool of concurrently-active downloads
|
||||
* (state.ts's `activeCap`) is fed continuously from the queued population and from
|
||||
* transient retries, so over a ten-minute run hundreds of distinct rows have shown live
|
||||
* progress by the time it ends — while at any single instant the active count stays
|
||||
* realistic, exactly as a real daemon would run it.
|
||||
*/
|
||||
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import type { Category, Queue, TaskState, TaskSummary } from '../../../extension/src/shared/protocol/types.js';
|
||||
import { TASK_ERROR_CODE_VALUES, type TaskErrorCode } from '../../../extension/src/shared/protocol/types.js';
|
||||
|
||||
/** Deterministic PRNG (mulberry32) so a given --seed reproduces the same table — useful
|
||||
* when a GUI bug only shows up at a particular row and needs to be reproduced exactly. */
|
||||
function mulberry32(seed: number): () => number {
|
||||
let a = seed >>> 0;
|
||||
return () => {
|
||||
a = (a + 0x6d2b79f5) | 0;
|
||||
let t = Math.imul(a ^ (a >>> 15), 1 | a);
|
||||
t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
|
||||
return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
|
||||
};
|
||||
}
|
||||
|
||||
type Rng = () => number;
|
||||
|
||||
function pick<T>(rng: Rng, items: readonly T[]): T {
|
||||
return items[Math.floor(rng() * items.length)] as T;
|
||||
}
|
||||
|
||||
function int(rng: Rng, min: number, max: number): number {
|
||||
return Math.floor(min + rng() * (max - min + 1));
|
||||
}
|
||||
|
||||
/** Log-uniform: real download sizes span 50 KB README files to 20 GB disk images, and a
|
||||
* plain uniform draw would make everything look mid-sized. */
|
||||
function logUniform(rng: Rng, min: number, max: number): number {
|
||||
const lo = Math.log(min);
|
||||
const hi = Math.log(max);
|
||||
return Math.round(Math.exp(lo + rng() * (hi - lo)));
|
||||
}
|
||||
|
||||
const ADJECTIVES = ['annual', 'final', 'draft', 'archived', 'backup', 'shared', 'personal',
|
||||
'weekly', 'monthly', 'legacy', 'updated', 'source', 'compiled', 'signed', 'raw', 'edited'];
|
||||
const NOUNS = ['report', 'photos', 'session', 'build', 'release', 'dataset', 'presentation',
|
||||
'recording', 'mixdown', 'clip', 'manual', 'invoice', 'archive', 'snapshot', 'export', 'project'];
|
||||
|
||||
function synthFilename(rng: Rng, ext: string): string {
|
||||
const stem = `${pick(rng, ADJECTIVES)}-${pick(rng, NOUNS)}-${int(rng, 1, 9999)}`;
|
||||
return `${stem}.${ext}`;
|
||||
}
|
||||
|
||||
function synthHost(rng: Rng): string {
|
||||
const hosts = ['cdn.example.org', 'files.example.net', 'mirror.example.com',
|
||||
'downloads.example.io', 'assets.example.dev', 'releases.example.org'];
|
||||
return pick(rng, hosts);
|
||||
}
|
||||
|
||||
/** Retryable per B1's taxonomy (`docs/adr/0010`) — matches `x-carriesHttpStatus` roughly:
|
||||
* network hiccups and 5xx are the plausible transient ones a synthetic run should cycle
|
||||
* through; auth/4xx/local-fs errors are terminal and go straight to 'failed'. */
|
||||
const RETRYABLE_CODES: readonly TaskErrorCode[] =
|
||||
['resolve_failed', 'connect_failed', 'connection_reset', 'timeout', 'http_server_error'];
|
||||
const TERMINAL_CODES: readonly TaskErrorCode[] =
|
||||
TASK_ERROR_CODE_VALUES.filter((c) => !RETRYABLE_CODES.includes(c) && c !== 'canceled');
|
||||
|
||||
export function randomTaskErrorCode(rng: Rng, retryable: boolean): TaskErrorCode {
|
||||
return pick(rng, retryable ? RETRYABLE_CODES : TERMINAL_CODES);
|
||||
}
|
||||
|
||||
export function taskErrorFor(rng: Rng, code: TaskErrorCode, retryable: boolean): TaskSummary['error'] {
|
||||
const httpCoded: readonly TaskErrorCode[] =
|
||||
['http_client_error', 'http_server_error', 'auth_required', 'forbidden', 'not_found',
|
||||
'range_not_satisfiable', 'gone'];
|
||||
return {
|
||||
code,
|
||||
message: `synthetic ${code} for load testing`,
|
||||
httpStatus: httpCoded.includes(code) ? pick(rng, [403, 404, 410, 429, 500, 502, 503]) : null,
|
||||
retryable,
|
||||
cause: null,
|
||||
attempt: int(rng, 1, 4),
|
||||
nextRetryAt: null,
|
||||
};
|
||||
}
|
||||
|
||||
export interface SynthOptions {
|
||||
readonly count: number;
|
||||
readonly seed: number;
|
||||
readonly categories: readonly Category[];
|
||||
readonly queues: readonly Queue[];
|
||||
readonly baseSpeedBps: number;
|
||||
}
|
||||
|
||||
export interface SynthResult {
|
||||
readonly tasks: TaskSummary[];
|
||||
/** taskIds seeded directly into 'downloading', in creation order. */
|
||||
readonly initiallyActive: string[];
|
||||
/** taskId -> the ordered position it holds in its queue (1-based), for tasks seeded
|
||||
* 'queued'. Bulk-seeding needs this to rebuild state.ts's per-queue FIFOs without
|
||||
* re-deriving order from creation time. */
|
||||
readonly queuedOrder: Map<string, string[]>; // queueId -> taskIds, front to back
|
||||
/** taskId -> a per-task speed multiplier, so active downloads don't all move in lockstep. */
|
||||
readonly speedFactor: Map<string, number>;
|
||||
}
|
||||
|
||||
// Weights need not sum to any particular total; they are normalised below. Kept as a flat
|
||||
// list (not a Record) so the generator can walk it in one pass while assigning position
|
||||
// within each state's own counter (queuePosition, retry delay, etc).
|
||||
const STATE_WEIGHTS: ReadonlyArray<readonly [TaskState, number]> = [
|
||||
['complete', 55],
|
||||
['failed', 8],
|
||||
['cancelled', 4],
|
||||
['paused', 6],
|
||||
['retry_wait', 2],
|
||||
['queued', 25],
|
||||
];
|
||||
|
||||
export function generateSyntheticTasks(opts: SynthOptions): SynthResult {
|
||||
const rng = mulberry32(opts.seed);
|
||||
const tasks: TaskSummary[] = [];
|
||||
const initiallyActive: string[] = [];
|
||||
const queuedOrder = new Map<string, string[]>();
|
||||
const speedFactor = new Map<string, number>();
|
||||
|
||||
const categories = opts.categories.length > 0 ? opts.categories : FALLBACK_CATEGORIES;
|
||||
const queues = opts.queues.length > 0 ? opts.queues : FALLBACK_QUEUES;
|
||||
for (const q of queues) queuedOrder.set(q.queueId, []);
|
||||
|
||||
// Reserve a slice of the population to start already 'downloading', mid-transfer, so a
|
||||
// client that connects the instant mockd starts sees live rows immediately rather than
|
||||
// waiting for the first promotion from the queue.
|
||||
const activeSeed = Math.min(opts.count, DEFAULT_ACTIVE_CAP);
|
||||
|
||||
const totalWeight = STATE_WEIGHTS.reduce((sum, [, w]) => sum + w, 0);
|
||||
const remaining = Math.max(0, opts.count - activeSeed);
|
||||
|
||||
// How many of the `remaining` tasks get each state, largest remainder method so the
|
||||
// rounding doesn't silently drop or duplicate a task at small N.
|
||||
const quotas = new Map<TaskState, number>();
|
||||
let assigned = 0;
|
||||
const remainders: Array<[TaskState, number]> = [];
|
||||
for (const [state, weight] of STATE_WEIGHTS) {
|
||||
const exact = (remaining * weight) / totalWeight;
|
||||
const floor = Math.floor(exact);
|
||||
quotas.set(state, floor);
|
||||
assigned += floor;
|
||||
remainders.push([state, exact - floor]);
|
||||
}
|
||||
remainders.sort((a, b) => b[1] - a[1]);
|
||||
for (let i = 0; i < remaining - assigned; i += 1) {
|
||||
const state = remainders[i % remainders.length]![0];
|
||||
quotas.set(state, (quotas.get(state) ?? 0) + 1);
|
||||
}
|
||||
|
||||
const now = Date.now();
|
||||
const spawn = (state: TaskState): TaskSummary => {
|
||||
const category = pick(rng, categories);
|
||||
const ext = pick(rng, category.extensions.length > 0 ? category.extensions : ['bin']);
|
||||
const filename = synthFilename(rng, ext);
|
||||
const sizeBytes = logUniform(rng, 50_000, 20_000_000_000);
|
||||
const resumable = rng() > 0.08;
|
||||
const segments = resumable ? pick(rng, [1, 2, 4, 4, 8, 8, 8, 16]) : 1;
|
||||
// Spread creation times over the past 30 days so the table doesn't show one instant.
|
||||
const createdAt = new Date(now - int(rng, 0, 30 * 24 * 3600) * 1000).toISOString();
|
||||
|
||||
const task: TaskSummary = {
|
||||
taskId: randomUUID(),
|
||||
filename,
|
||||
saveDir: category.saveDir,
|
||||
url: `https://${synthHost(rng)}/${category.categoryId}/${filename}`,
|
||||
effectiveUrl: null,
|
||||
sizeBytes,
|
||||
downloadedBytes: 0,
|
||||
state,
|
||||
speedBps: 0,
|
||||
etaSeconds: null,
|
||||
resumable,
|
||||
segments,
|
||||
categoryId: category.categoryId,
|
||||
queueId: null,
|
||||
queuePosition: null,
|
||||
description: rng() > 0.85 ? pick(rng, NOUNS) : null,
|
||||
createdAt,
|
||||
lastTryAt: null,
|
||||
completedAt: null,
|
||||
error: null,
|
||||
};
|
||||
speedFactor.set(task.taskId, 0.3 + rng() * 1.7);
|
||||
return task;
|
||||
};
|
||||
|
||||
for (const [state, quota] of quotas) {
|
||||
for (let i = 0; i < quota; i += 1) {
|
||||
const task = spawn(state);
|
||||
switch (state) {
|
||||
case 'complete': {
|
||||
task.downloadedBytes = task.sizeBytes ?? 0;
|
||||
task.lastTryAt = task.createdAt;
|
||||
task.completedAt = new Date(
|
||||
new Date(task.createdAt).getTime() + int(rng, 5, 3600) * 1000,
|
||||
).toISOString();
|
||||
break;
|
||||
}
|
||||
case 'failed': {
|
||||
const code = randomTaskErrorCode(rng, false);
|
||||
task.error = taskErrorFor(rng, code, false);
|
||||
task.downloadedBytes = Math.floor((task.sizeBytes ?? 0) * rng() * 0.6);
|
||||
task.lastTryAt = task.createdAt;
|
||||
break;
|
||||
}
|
||||
case 'cancelled': {
|
||||
task.downloadedBytes = Math.floor((task.sizeBytes ?? 0) * rng() * 0.4);
|
||||
task.lastTryAt = task.createdAt;
|
||||
break;
|
||||
}
|
||||
case 'paused': {
|
||||
task.downloadedBytes = Math.floor((task.sizeBytes ?? 0) * rng());
|
||||
task.lastTryAt = task.createdAt;
|
||||
break;
|
||||
}
|
||||
case 'retry_wait': {
|
||||
const code = randomTaskErrorCode(rng, true);
|
||||
task.error = taskErrorFor(rng, code, true);
|
||||
task.downloadedBytes = Math.floor((task.sizeBytes ?? 0) * rng() * 0.5);
|
||||
task.lastTryAt = task.createdAt;
|
||||
break;
|
||||
}
|
||||
case 'queued': {
|
||||
// ~80/20 split across the first two queues, matching Main/Sync in the fixtures;
|
||||
// falls back to the single available queue if fewer than two exist.
|
||||
const queue = queues.length > 1 && rng() < 0.8 ? queues[0]! : pick(rng, queues);
|
||||
task.queueId = queue.queueId;
|
||||
const order = queuedOrder.get(queue.queueId)!;
|
||||
order.push(task.taskId);
|
||||
task.queuePosition = order.length;
|
||||
break;
|
||||
}
|
||||
default:
|
||||
break;
|
||||
}
|
||||
tasks.push(task);
|
||||
}
|
||||
}
|
||||
|
||||
// The reserved active slice: mid-transfer, non-zero progress, so it reads as "already
|
||||
// running" rather than "just started" the moment mockd comes up.
|
||||
for (let i = 0; i < activeSeed; i += 1) {
|
||||
const task = spawn('downloading');
|
||||
task.downloadedBytes = Math.floor((task.sizeBytes ?? 0) * rng() * 0.7);
|
||||
task.lastTryAt = task.createdAt;
|
||||
tasks.push(task);
|
||||
initiallyActive.push(task.taskId);
|
||||
}
|
||||
|
||||
return { tasks, initiallyActive, queuedOrder, speedFactor };
|
||||
}
|
||||
|
||||
export const DEFAULT_ACTIVE_CAP = 24;
|
||||
|
||||
const FALLBACK_CATEGORIES: readonly Category[] = [
|
||||
{ categoryId: 'compressed', name: 'Compressed', saveDir: '/home/sami/Downloads/Compressed',
|
||||
extensions: ['zip', 'tar', 'gz'], mimeTypes: [], builtin: true, sortOrder: 0 },
|
||||
];
|
||||
const FALLBACK_QUEUES: readonly Queue[] = [
|
||||
{ queueId: 'main', name: 'Main Queue', state: 'running', maxConcurrent: 3,
|
||||
taskIds: [], schedule: null, onComplete: 'nothing' },
|
||||
];
|
||||
Reference in New Issue
Block a user