# Online integrity checking The serving process gradually verifies the bytes of every indexed object against its full content identity. It uses the same streaming verifier as `doctor`, including envelope checks, decompression, payload length, and digest validation. Packed and loose objects are covered. Verification never repairs, deletes, quarantines, or accepts data. This checks **indexed object bytes**. It does not replace the offline `doctor`: metadata consistency, missing references absent from the locator, unindexed pack entries, and whole-pack inventory still require that broader audit. A clean pass describes the bytes observed during the pass, not an atomic snapshot or a promise that the storage remains healthy afterward. ## Configuration The checker is enabled by default in `serve`, including with an existing configuration that omits this section: ```toml [integrity] enabled = true mib_per_second = 1 idle_secs = 1 interval_secs = 86400 ``` - `mib_per_second`: maximum scheduled decoded payload throughput, 1–1024 MiB/s. Compressed input, object headers, index lookups, and progress-file I/O are additional. One 64 KiB read can start immediately; unused budget does not accumulate into a later burst. - `idle_secs`: quiet time after foreground work, 0–3600 seconds. Queued and executing TCP requests pause verification; idle connections do not. Online checkpoint work also pauses it. The checker checks admission between reads, never holds the publication gate, and consumes no foreground worker permit. - `interval_secs`: wait after a repository pass completes before starting its next pass, 60–31536000 seconds. The default is one day. - `enabled = false`: the checker does not start on the next launch. Saved reports remain available. There is one checker thread for the whole server. It holds one repository's progress and one streaming object reader at a time, with a 64 KiB read buffer and the existing bounded decompressor. It rotates repositories after at most 16 objects, caps object starts at 20 per second, and polls shutdown while waiting. A large object finishes before that rotation, but it can pause between reads. An OS read already in progress cannot be preempted. This is cooperative load control, not a guarantee of zero disk contention. Continuously busy servers can defer verification indefinitely; inspect coverage age rather than assuming a daily pass completes. ## Coverage and reports ```sh scribe-server integrity-status --runtime-dir /srv/scribe/run scribe-server --json integrity-status --runtime-dir /srv/scribe/run --repo REPOSITORY_ID ``` The command reads saved progress without opening or locking the data root. It works while the server runs and after it stops. Reports include the current pass's visited-object count, successfully verified payload bytes, failures, bounded per-object findings, the last completed pass, and the last clean pass. They are saved reports, not a live health or process-liveness probe. Repository selection takes an ID, not a name. Exit codes: - **0:** saved completed passes have no findings, and no current findings/error are recorded. - **1:** current or last-completed coverage contains failures, or a scan error was saved. - **2:** no matching report, no completed pass yet, or reports cannot be read. Progress is saved after each batch, pass completion, or cooperative shutdown. Up to 32 detailed findings are retained per pass; the failure count continues beyond that. Every detected object failure emits `integrity.object_failed` with `check = objects.content`, repository ID, object ID, and error. The offline doctor emits that same event. `integrity.pass_completed` records pass totals; `integrity.scan_failed` reports a checker that could not advance. Diagnostic logs retain their normal bounded/lossy behavior. A pass records an initial upper object-key boundary and seeks forward through the locator with short read transactions. It does not retain a database snapshot for hours. New uploads behind the cursor or beyond that boundary are covered by the next pass; concurrent uploads never become accepted history because the checker sees them. Indexed objects belonging to unfinished uploads are checked too. ## Restart and recovery Checksummed progress lives in `/integrity/.json`. It is disposable operational state, excluded from canonical history, object identity, and the wire protocol. No repository migration is required. Keep the runtime directory between launches if you want to resume coverage. Only completed object checks advance the cursor. Shutdown interrupts waiting or throttled verification; after restart, an unfinished object is read from the beginning. Progress replacement uses a temporary file and rename without durability flushes: losing recent progress repeats work. It never permits a partial check to be reported as verified. Crash leftovers with the matching `.writing` name are replaced on the next progress save. Root and repository incarnation changes discard coverage from the old generation with an explicit event. Malformed, oversized, or checksum-damaged state is reported and refused, rather than silently read as a valid cursor. To reset a damaged report, stop the server, retain the report for diagnosis, and remove only that repository's `.json` and `.writing` progress files. Restart begins a new pass. Do not remove repository files to reset verification. For object failures, preserve the evidence and use the existing [diagnosis and explicit recovery tools](/docs/operations.md#routine-health-checks). With a known-good copy, [verified object repair](/docs/object-repair.md) can replace damaged bytes without rolling back newer accepted history. Online verification is not a substitute for independent backups and restore drills.