Parallel client transfers

Uploads and downloads automatically use a bounded pool of session-bound data connections when enough missing chunks are queued. No server setting or format migration is required. Ordinary small transfers keep using the control connection.

Configuration

The default limit is four simultaneous data transfers per client connection. Choose an integer from 1 through 8:

scribe --transfer-jobs 2 sync
scribe --transfer-jobs 1 submit -m "Serial transfer"
SCRIBE_TRANSFER_JOBS=4 scribe sync

The CLI flag overrides SCRIBE_TRANSFER_JOBS, which overrides the default. Invalid values are rejected. The native client uses the same default and reads SCRIBE_TRANSFER_JOBS from its process environment. Set it before launching the native app; there is no new GUI preference. Rust callers can set TcpClientOptions::transfer_jobs directly. scribe diagnostics reports the effective limit.

1 forces serial transfers. Higher values are ceilings, not a promise that all connections will be used or that throughput will multiply by that number. The scheduler selects useful concurrency from the currently queued missing chunks: roughly one lane per 8 MiB, capped by the configured limit and available independent batches. A single object is never split. Cached and locally reusable chunks need no network lane. Connections are opened lazily and reused within the control session; at most the configured number of data connections plus the original control connection exist.

This is scheduling based on available work, not bandwidth autotuning. It does not probe network speed or increase concurrency without a bound. Server admission and per-principal limits still apply. Connection or authorization errors are reported; they are not hidden by a serial retry. Lower the client limit if the server's connection budget or storage calls for it.

Safety and resource bounds

• All upload identities are durably pinned before any worker transmits bytes. Chunk uploads finish durably before manifests and change sets are published. Metadata acceptance stays on the original connection, after successful upload.
• Upload workers reread file ranges and verify the original chunk digests. Parallel batches share the existing 1,024-object bound across active lanes (256 per batch with four lanes), with at most 64 MiB of logical content each; payloads stream from disk, rather than being buffered as whole batches.
• Download workers use the same manifest witnesses and authorization as serial reads. Each worker hands over one bounded chunk at a time through a rendezvous channel. There is no growing queue of downloaded payloads.
• The caller verifies chunk identities, writes temporary files, publishes cache objects, reports progress, and verifies complete file identities before file replacement. Local-work recovery and journal rules are unchanged.
• Errors stop new jobs. All started workers finish or fail and are joined before the workflow returns. Cancellation is cooperative; an in-flight durable server publication may complete. Failed streams are discarded before reuse. No automatic retry of an unknown metadata outcome is introduced.
• Data lanes inherit TLS trust, session binding, compression and cancellation. CLI protocol-byte counters include the pooled connections and handshakes.

Chunk payload memory is bounded by the active producers plus the consuming thread (each chunk is at most 16 MiB), in addition to transport/compression buffers and the existing workflow state. Connections remain pooled until the control session ends. The control connection is silent while the data connections move bytes; the server's inactivity timeout counts the whole session, so a transfer of any length keeps it open for the commit that follows. A gateway replaces its idle upstream connection behind the control connection before using it again. File analysis has its own existing worker bound.

Verification and focused comparison

The ordinary regression suite exercises serial and parallel submit/sync, compression, both test codecs, gateway routing, interrupted transfers, content identity, warm sync, and temporary-file cleanup. Transport tests cover pool reuse, failed-stream replacement, cancellation, and TLS authorization on data lanes.

Run the explicit local comparison separately:

cargo test --release --all-features --locked -p scribe-client --test parallel_transfers compare_serial_and_parallel_transfers -- --ignored --nocapture

It compares one and four lanes in alternating order, twice each, using fresh isolated repositories and 128 MiB of deterministic incompressible files per trial. Submit and sync use separate fresh control sessions; opening their data pools is included in the measured workflows. The OS cache is not forcibly cleared. It checks the resulting file bytes and removes its stores and servers. Reported submit/sync times include complete client workflows, not just network transfer. This loopback comparison does not certify WAN throughput or production capacity. Existing --timings profiling includes transfer.parallel and the per-request upload/download spans; overlapping spans must not be summed as CPU time.