# Synthetic concurrency benchmark

`tools/concurrency.js` measures a ladder of exact concurrent client counts against
fresh, isolated Scribe servers. It reports throughput, p50/p95/p99/max latency,
errors, per-worker progress, and the highest tested load meeting chosen budgets
in every repetition. It uses the development-only `scribe-perf` executable.

## Run it

Build current release binaries, then choose new report filenames:

```sh
cargo build --release --all-features --locked -p scribe-perf -p scribe-cli -p scribe-server-cli
node tools/concurrency.js --report target/perf/capacity.json --report target/perf/capacity.html
node tools/concurrency.js --help
```

The default ladder is 1, 2, 4, 8, 16, 32, 64, and 128 clients. Each of four
workloads gets two fresh trials per level, with two seconds of warmup and ten
seconds of measurement. A smaller exploratory run:

```sh
node tools/concurrency.js --levels 1,4,16,64,128 --duration-secs 5 --warmup-secs 1 --repeat 2 --report target/perf/exploratory.json
```

Use `--dry-run` to inspect the schedule without creating files or starting
processes. `--bin-dir` chooses prebuilt binaries and `--temp-dir` chooses the
parent of the disposable run directory. Debug measurement helpers are refused.
Existing report files are refused rather than overwritten. No saved performance
baseline is created or changed. Schema 3 capacity reports are independent of the
schema 2 qualification/baseline reports from `tools/scenario.js --profile`.

## Workloads and boundaries

| Workload | One measured operation |
|---|---|
| `head` | One branch-head RPC |
| `tree` | One historical tree-page RPC, up to 256 entries, rotating through the seed tree |
| `submit` | Create a unique 4 KiB file, add it locally, upload and durably submit it |
| `mixed` | Each client cycles through four heads, four tree pages, and two submits; starting positions differ |

`--mixes head,tree,submit,mixed` selects workloads. Every trial seeds a fresh
repository with 1,024 distinct 1 KiB files (`--files`, range 256–8,192). Writers
use disjoint new paths, so expected merge conflicts cannot disguise errors.
Historical tree results must remain identical to the seed pages throughout the
run. Final head must equal the seed head plus every successful submit, including
warmup and drain. Errors stop that worker; ambiguous writes are never retried.

Setup, seeding, workspace registration, TCP connection establishment, and warmup
are outside the measured window. Workers start through a common gate, with one
persistent connection and at most one operation at a time each. No client-side
think time is added. Throughput counts successful **completions inside the
measurement window**, including requests started during warmup. This avoids
reporting zero throughput when a busy server has long-running requests crossing
the start boundary. Latency covers every operation overlapping the window,
including warmup crossings and operations drained after its end. Draining
completions cannot inflate throughput. Failed overlapping attempts also
contribute to latency and error counts.

Latency statistics use fixed-size logarithmic histograms per worker/class,
merged after joining every worker. Quantiles are bucket upper bounds, with at
most 2% rounding above one microsecond. Counts, maximum, and mean are recorded
separately. Statistics do not retain an unbounded list of samples or share a
mutex in the operation loop. `drain_ms` exposes work extending beyond the window.
Client connection and individual socket-I/O timeouts are 5 and 30 seconds;
the outer runner also bounds the helper process lifetime.

## Interpreting capacity

Defaults are p95 ≤100 ms for head/tree and ≤1,000 ms for a submit workflow.
Change these with `--read-p95-ms` and `--submit-p95-ms`. Each operation class
needs at least 100 measured attempts (`--min-samples`), zero errors, and every
requested worker must complete work in the window. Too few samples are
**inconclusive**, not evidence of low capacity: rerun with a longer duration.

The summary selects the highest contiguous tested level satisfying those rules
in **every repetition**. It does not skip a failing or missing lower level to
claim an isolated higher result. `ceiling_not_reached` means the largest tested
level passed; it is a lower bound, not a discovered maximum. Inspect each row's
assessment reasons to distinguish a latency failure, insufficient samples,
connection/setup failure, and a correctness failure. A completed harness can
have nonqualifying load levels; `gates_passed` separately reports correctness.

These are short closed-loop synthetic runs, not long-duration production
capacity certification. A slow server throttles the client's next request;
this is not a fixed-arrival-rate overload test and there is no hidden external
request queue. Concurrent active operations are not total logged-in users.
Head RPCs, tree pages, and complete submit workflows are different units; use
per-class metrics when comparing results.

Client and server share CPU, memory, OS cache, filesystem, and loopback networking.
The report records compiler, architecture, CPU, RAM, Git state, executable hashes,
and generated server configuration. Server limits and Immediate durability are
unchanged. Server logging uses `warn`. TLS/authentication, WAN latency, payload
sync/download, contention on the same path, simultaneous bulk imports, and
multi-host load generation are outside this benchmark. Production sizing needs
representative data, transport, operation mix, and longer runs on deployment
hardware. In particular, this probe does not quantify the chosen v1 policy of
allowing small submissions to wait behind an import.

## Diagnose submit phases

Use the existing `--timings` framework through the same runner:

```sh
node tools/concurrency.js --timings --levels 1,3 --mixes submit --duration-secs 10 --warmup-secs 1 --repeat 2 --report target/perf/submit-phases.json --report target/perf/submit-phases.html
```

Build the release binaries with the command above first. The helper installs
the shared `scribe-log` subscriber; it also accepts `--timings`, `--log-format`,
and `--log-dir` directly. These diagnostic flags do not change JSON stdout.
The runner enables JSON timing logs in both helper and server. Ordinary capacity
runs keep timing spans disabled. Profiled runs have `config.timings: true`, no
capacity claim, and no baseline comparison: tracing changes the measured workload.

Each trial reports `timings.client.<phase>.*` and
`timings.server.<rpc>.<phase>.*`: count, summed lifetime/busy/idle milliseconds,
mean, p95, and maximum. The underlying JSON span-close records are the same ones
used by the CLI's [phase timings](/docs/logging.md#opt-in-phase-timings). Quantiles use
bounded histograms, adding up to 2% rounding above one microsecond to the log's
duration rounding. Busy means entered wall time, including blocking waits, not
CPU time. Parent spans include children; concurrent spans also overlap, so do
not sum all phase rows to estimate end-to-end latency.
Means are per span invocation. Retries and batched RPCs can produce several
invocations per submit; for a submit-only workload, divide a phase's `total_ms`
by `timings.client.client.submit.count` to obtain its accumulated cost per submit.

Important submit boundaries include:

- Client add, selection, analysis/cache publication, durable upload intent,
  transfer, durable candidate, remote submit, and local finalization.
- Server candidate preparation, `submit_turn.wait` and `submit_turn.hold`,
  acceptance preparation, envelope publication, publication-gate wait/hold,
  durable acceptance, and finalized upload-intent cleanup.
- Metadata writer acquisition, staged commit, recovery archive publication,
  and the existing file/directory flush spans. `metadata.begin_write` also
  includes archive preparation; it is not solely lock wait.

Buffered publication also emits `server.submit.prepare_parallel`,
`server.submit.prepare_group`, `submit.group`, and `submit.group.wait`.
A group leader performs durable work for several requests, so its nested spans
belong to the leader's RPC. Normalize totals by all completed submits, not by
the number of group spans. In-process service diagnostics can also read
`Metrics::submit_group_queued`, `submit_groups`, `submit_group_members`, and
`submit_group_max` to distinguish queue entries, accepted group transactions,
accepted members, and the largest accepted group. These counters are not exposed
through the wire protocol or included in the TCP benchmark report.

The diagnostic scope includes **all load-loop operations, warmup and drain
included**. It differs from the capacity measurement window. Client load spans
select RPCs; exact session/request identities correlate their server spans,
including admission and upload work. Setup, seeding, and verification are
excluded without synchronizing process clocks. Request/session fields are decimal
strings in timing JSON so JavaScript preserves all 64 bits.

`timing_capture_complete` checks the client/server request counts, submit counts,
parse errors, and shutdown diagnostics for dropped events/sink failures. Missing
or malformed records produce an incomplete profile, never invented zero timings.
Readers bound logs to 256 MiB per process, lines to 256 KiB, request correlations
to 100,000, and phase groups to 512. Exceeding a bound marks the capture incomplete;
use a shorter diagnostic run. The bounded lossy logging queue remains unchanged.
Only summaries survive: raw logs are deleted after each trial and on interruption.

## Cleanup and tests

The runner owns its temporary directory and processes. After each trial it stops
the server before removing its store, working copies, and logs. Final cleanup
runs on success, failure, SIGINT, and SIGTERM. Only requested reports survive;
there is no keep-data option. Reports record `cleanup_complete`. SIGKILL or a
power loss cannot execute cleanup; the private directory is printed at startup
so such an interrupted run can be identified. Do not delete another run's data.

Run the focused tests after building release binaries:

```sh
cargo test -p scribe-perf --all-features --locked
node --test tools/scenarios/timings.test.js tools/scenarios/concurrency/runner.test.js tools/scenarios/concurrency/integration.test.js
```

Tests check window/drain accounting, bounded histogram aggregation, sample/SLO
classification, incomplete repetitions, option bounds, existing-root protection,
and real workloads. Lifecycle tests exercise missing binaries and interruption
of active workers, assert that owned processes/directories are gone, protect an
adjacent sentinel, and remove their own reports and test directories afterward.