Synthetic concurrency benchmark

tools/concurrency.js measures a ladder of exact concurrent client counts against fresh, isolated Scribe servers. It reports throughput, p50/p95/p99/max latency, errors, per-worker progress, and the highest tested load meeting chosen budgets in every repetition. It uses the development-only scribe-perf executable.

Run it

Build current release binaries, then choose new report filenames:

cargo build --release --all-features --locked -p scribe-perf -p scribe-cli -p scribe-server-cli
node tools/concurrency.js --report target/perf/capacity.json --report target/perf/capacity.html
node tools/concurrency.js --help

The default ladder is 1, 2, 4, 8, 16, 32, 64, and 128 clients. Each of four workloads gets two fresh trials per level, with two seconds of warmup and ten seconds of measurement. A smaller exploratory run:

node tools/concurrency.js --levels 1,4,16,64,128 --duration-secs 5 --warmup-secs 1 --repeat 2 --report target/perf/exploratory.json

Use --dry-run to inspect the schedule without creating files or starting processes. --bin-dir chooses prebuilt binaries and --temp-dir chooses the parent of the disposable run directory. Debug measurement helpers are refused. Existing report files are refused rather than overwritten. No saved performance baseline is created or changed. Schema 3 capacity reports are independent of the schema 2 qualification/baseline reports from tools/scenario.js --profile.

Workloads and boundaries

--mixes head,tree,submit,mixed selects workloads. Every trial seeds a fresh repository with 1,024 distinct 1 KiB files (--files, range 256–8,192). Writers use disjoint new paths, so expected merge conflicts cannot disguise errors. Historical tree results must remain identical to the seed pages throughout the run. Final head must equal the seed head plus every successful submit, including warmup and drain. Errors stop that worker; ambiguous writes are never retried.

Setup, seeding, workspace registration, TCP connection establishment, and warmup are outside the measured window. Workers start through a common gate, with one persistent connection and at most one operation at a time each. No client-side think time is added. Throughput counts successful completions inside the measurement window, including requests started during warmup. This avoids reporting zero throughput when a busy server has long-running requests crossing the start boundary. Latency covers every operation overlapping the window, including warmup crossings and operations drained after its end. Draining completions cannot inflate throughput. Failed overlapping attempts also contribute to latency and error counts.

Latency statistics use fixed-size logarithmic histograms per worker/class, merged after joining every worker. Quantiles are bucket upper bounds, with at most 2% rounding above one microsecond. Counts, maximum, and mean are recorded separately. Statistics do not retain an unbounded list of samples or share a mutex in the operation loop. drain_ms exposes work extending beyond the window. Client connection and individual socket-I/O timeouts are 5 and 30 seconds; the outer runner also bounds the helper process lifetime.

Interpreting capacity

Defaults are p95 ≤100 ms for head/tree and ≤1,000 ms for a submit workflow. Change these with --read-p95-ms and --submit-p95-ms. Each operation class needs at least 100 measured attempts (--min-samples), zero errors, and every requested worker must complete work in the window. Too few samples are inconclusive, not evidence of low capacity: rerun with a longer duration.

The summary selects the highest contiguous tested level satisfying those rules in every repetition. It does not skip a failing or missing lower level to claim an isolated higher result. ceiling_not_reached means the largest tested level passed; it is a lower bound, not a discovered maximum. Inspect each row's assessment reasons to distinguish a latency failure, insufficient samples, connection/setup failure, and a correctness failure. A completed harness can have nonqualifying load levels; gates_passed separately reports correctness.

These are short closed-loop synthetic runs, not long-duration production capacity certification. A slow server throttles the client's next request; this is not a fixed-arrival-rate overload test and there is no hidden external request queue. Concurrent active operations are not total logged-in users. Head RPCs, tree pages, and complete submit workflows are different units; use per-class metrics when comparing results.

Client and server share CPU, memory, OS cache, filesystem, and loopback networking. The report records compiler, architecture, CPU, RAM, Git state, executable hashes, and generated server configuration. Server limits and Immediate durability are unchanged. Server logging uses warn. TLS/authentication, WAN latency, payload sync/download, contention on the same path, simultaneous bulk imports, and multi-host load generation are outside this benchmark. Production sizing needs representative data, transport, operation mix, and longer runs on deployment hardware. In particular, this probe does not quantify the chosen v1 policy of allowing small submissions to wait behind an import.

Diagnose submit phases

Use the existing --timings framework through the same runner:

node tools/concurrency.js --timings --levels 1,3 --mixes submit --duration-secs 10 --warmup-secs 1 --repeat 2 --report target/perf/submit-phases.json --report target/perf/submit-phases.html

Build the release binaries with the command above first. The helper installs the shared scribe-log subscriber; it also accepts --timings, --log-format, and --log-dir directly. These diagnostic flags do not change JSON stdout. The runner enables JSON timing logs in both helper and server. Ordinary capacity runs keep timing spans disabled. Profiled runs have config.timings: true, no capacity claim, and no baseline comparison: tracing changes the measured workload.

Each trial reports timings.client.<phase>.* and timings.server.<rpc>.<phase>.*: count, summed lifetime/busy/idle milliseconds, mean, p95, and maximum. The underlying JSON span-close records are the same ones used by the CLI's phase timings. Quantiles use bounded histograms, adding up to 2% rounding above one microsecond to the log's duration rounding. Busy means entered wall time, including blocking waits, not CPU time. Parent spans include children; concurrent spans also overlap, so do not sum all phase rows to estimate end-to-end latency. Means are per span invocation. Retries and batched RPCs can produce several invocations per submit; for a submit-only workload, divide a phase's total_ms by timings.client.client.submit.count to obtain its accumulated cost per submit.

Important submit boundaries include:

• Client add, selection, analysis/cache publication, durable upload intent, transfer, durable candidate, remote submit, and local finalization.
• Server candidate preparation, submit_turn.wait and submit_turn.hold, acceptance preparation, envelope publication, publication-gate wait/hold, durable acceptance, and finalized upload-intent cleanup.
• Metadata writer acquisition, staged commit, recovery archive publication, and the existing file/directory flush spans. metadata.begin_write also includes archive preparation; it is not solely lock wait.

Buffered publication also emits server.submit.prepare_parallel, server.submit.prepare_group, submit.group, and submit.group.wait. A group leader performs durable work for several requests, so its nested spans belong to the leader's RPC. Normalize totals by all completed submits, not by the number of group spans. In-process service diagnostics can also read Metrics::submit_group_queued, submit_groups, submit_group_members, and submit_group_max to distinguish queue entries, accepted group transactions, accepted members, and the largest accepted group. These counters are not exposed through the wire protocol or included in the TCP benchmark report.

The diagnostic scope includes all load-loop operations, warmup and drain included. It differs from the capacity measurement window. Client load spans select RPCs; exact session/request identities correlate their server spans, including admission and upload work. Setup, seeding, and verification are excluded without synchronizing process clocks. Request/session fields are decimal strings in timing JSON so JavaScript preserves all 64 bits.

timing_capture_complete checks the client/server request counts, submit counts, parse errors, and shutdown diagnostics for dropped events/sink failures. Missing or malformed records produce an incomplete profile, never invented zero timings. Readers bound logs to 256 MiB per process, lines to 256 KiB, request correlations to 100,000, and phase groups to 512. Exceeding a bound marks the capture incomplete; use a shorter diagnostic run. The bounded lossy logging queue remains unchanged. Only summaries survive: raw logs are deleted after each trial and on interruption.

Cleanup and tests

The runner owns its temporary directory and processes. After each trial it stops the server before removing its store, working copies, and logs. Final cleanup runs on success, failure, SIGINT, and SIGTERM. Only requested reports survive; there is no keep-data option. Reports record cleanup_complete. SIGKILL or a power loss cannot execute cleanup; the private directory is printed at startup so such an interrupted run can be identified. Do not delete another run's data.

Run the focused tests after building release binaries:

cargo test -p scribe-perf --all-features --locked
node --test tools/scenarios/timings.test.js tools/scenarios/concurrency/runner.test.js tools/scenarios/concurrency/integration.test.js

Tests check window/drain accounting, bounded histogram aggregation, sample/SLO classification, incomplete repetitions, option bounds, existing-root protection, and real workloads. Lifecycle tests exercise missing binaries and interruption of active workers, assert that owned processes/directories are gone, protect an adjacent sentinel, and remove their own reports and test directories afterward.