# Parallel client transfers

Uploads and downloads automatically use a bounded pool of session-bound data
connections when enough missing chunks are queued. No server setting or format
migration is required. Ordinary small transfers keep using the control connection.

## Configuration

The default limit is **four simultaneous data transfers per client connection**.
Choose an integer from 1 through 8:

```sh
scribe --transfer-jobs 2 sync
scribe --transfer-jobs 1 submit -m "Serial transfer"
SCRIBE_TRANSFER_JOBS=4 scribe sync
```

The CLI flag overrides `SCRIBE_TRANSFER_JOBS`, which overrides the default.
Invalid values are rejected. The native client uses the same default and reads
`SCRIBE_TRANSFER_JOBS` from its process environment. Set it before launching
the native app; there is no new GUI preference. Rust callers can set `TcpClientOptions::transfer_jobs` directly.
`scribe diagnostics` reports the effective limit.

`1` forces serial transfers. Higher values are ceilings, not a promise that
all connections will be used or that throughput will multiply by that number.
The scheduler selects useful concurrency from the currently queued missing
chunks: roughly one lane per 8 MiB, capped by the configured limit and available
independent batches. A single object is never split. Cached and
locally reusable chunks need no network lane. Connections are opened lazily and
reused within the control session; at most the configured number of data
connections plus the original control connection exist.

This is scheduling based on available work, not bandwidth autotuning. It does
not probe network speed or increase concurrency without a bound. Server admission
and per-principal limits still apply. Connection or authorization errors are
reported; they are not hidden by a serial retry. Lower the client limit if the
server's connection budget or storage calls for it.

## Safety and resource bounds

- All upload identities are durably pinned before any worker transmits bytes.
  Chunk uploads finish durably before manifests and change sets are published.
  Metadata acceptance stays on the original connection, after successful upload.
- Upload workers reread file ranges and verify the original chunk digests.
  Parallel batches share the existing 1,024-object bound across active lanes
  (256 per batch with four lanes), with at most 64 MiB of logical content each;
  payloads stream from disk, rather than being buffered as whole batches.
- Download workers use the same manifest witnesses and authorization as serial
  reads. Each worker hands over one bounded chunk at a time through a rendezvous
  channel. There is no growing queue of downloaded payloads.
- The caller verifies chunk identities, writes temporary files, publishes cache
  objects, reports progress, and verifies complete file identities before file
  replacement. Local-work recovery and journal rules are unchanged.
- Errors stop new jobs. All started workers finish or fail and are joined before
  the workflow returns. Cancellation is cooperative; an in-flight durable server
  publication may complete. Failed streams are discarded before reuse. No
  automatic retry of an unknown metadata outcome is introduced.
- Data lanes inherit TLS trust, session binding, compression and cancellation.
  CLI protocol-byte counters include the pooled connections and handshakes.

Chunk payload memory is bounded by the active producers plus the consuming
thread (each chunk is at most 16 MiB), in addition to transport/compression
buffers and the existing workflow state. Connections remain pooled until the
control session ends. The control connection is silent while the data
connections move bytes; the server's inactivity timeout counts the whole
session, so a transfer of any length keeps it open for the commit that
follows. A gateway replaces its idle upstream connection behind the control
connection before using it again. File analysis has its own existing worker bound.

## Verification and focused comparison

The ordinary regression suite exercises serial and parallel submit/sync,
compression, both test codecs, gateway routing, interrupted transfers, content
identity, warm sync, and temporary-file cleanup. Transport tests cover pool reuse,
failed-stream replacement, cancellation, and TLS authorization on data lanes.

Run the explicit local comparison separately:

```sh
cargo test --release --all-features --locked -p scribe-client --test parallel_transfers compare_serial_and_parallel_transfers -- --ignored --nocapture
```

It compares one and four lanes in alternating order, twice each, using fresh
isolated repositories and 128 MiB of deterministic incompressible files per
trial. Submit and sync use separate fresh control sessions; opening their data
pools is included in the measured workflows. The OS cache is not forcibly
cleared. It checks the resulting file bytes and removes its stores and servers.
Reported submit/sync times include complete client workflows, not just network
transfer. This loopback comparison does not certify WAN throughput or production
capacity. Existing `--timings` profiling includes `transfer.parallel` and the
per-request upload/download spans; overlapping spans must not be summed as CPU
time.