scribe.
Download
3. Scale with a gateway
Documentation/Start here
3. Scale with a gateway7 min read

3. Scale across storage instances with a gateway

On this page

Previous: authentication · All tutorials · Next: operate the cluster

You will put one gateway in front of storage A, add storage B behind it, and move the clients to the gateway. Work from the same scribe-demo directory.

The scaling unit is a repository. New repositories can be placed on different storage roots, so separate projects can use different disks or machines. Existing game stays on A. The gateway does not stripe one repository across nodes, rebalance existing repositories, or provide replication or failover.

Diagram
Diagram source
flowchart LR
    Client["Clients: 127.0.0.1:7557"] --> Gateway["Gateway"]
    Gateway --> A["Storage A: 7447 — game"]
    Gateway --> B["Storage B: 7448 — new repositories"]
    A --> Auth["Shared auth: 7443 — studio namespace"]
    B --> Auth

1. Add the gateway

add changes a stopped cluster. Stop it, look at the plan, and apply it:

sh
scribe-ctl --config scribe-cluster.toml down
scribe-ctl --config scribe-cluster.toml add gateway --plan
scribe-ctl --config scribe-cluster.toml add gateway

The plan shows the definition's new [clusters.local.gateway] table and the gateway's own configuration file in full:

text
    @ scribe-gateway.toml (a new file)
    + # Scribe gateway for cluster local, written by `scribe-ctl add gateway`.
    + # Paths are relative to this file. See docs/gateway-usage.md for every setting.
    + data_dir = "data/gateway"
    + listen = "127.0.0.1:7557"
    +
    + [tls]
    + # Issued from the cluster CA (pki/ca), like the storage servers' certificates.
    + cert = "pki/gateway/server.pem"
    + key = "pki/gateway/server-key.pem"
    + # The scribe-gateway service account's credential: a namespace reader and
    + # delegate in every namespace below. Issued by `scribe-ctl add gateway`.
    + credential = "secrets/gateway.credential"
    +
    + [[instances]]
    + name = "storage-a"
    + namespace = "studio"
    + endpoint = "tls://127.0.0.1:7447"
    + ca = "pki/ca/ca.pem"
text
Scribe · local · Gateway added
  wrote scribe-gateway.toml
  rewrote scribe-cluster.toml; the previous definition is kept as scribe-cluster.toml.bak
  issued pki/gateway/server.pem
  initialized data/gateway
  server storage-a: already registered in studio
  storage-a: credential kept (valid until 2027-09-27)
  scribe-gateway: created with its grants in studio
  scribe-gateway: credential issued to secrets/gateway.credential
  scribe-monitor: present with its grants in studio
  scribe-monitor: credential kept (valid until 2027-09-27)
  wrote CLUSTER-INFO.txt

What scribe-ctl did for you

  • The gateway's configuration, scribe-gateway.toml: where it listens, its TLS certificate (from pki/ca, like the stores'), its credential, and one [[instances]] entry per store. Paths are relative to the file; see gateway configuration for every setting.

  • The routing ledger in data/gateway, created by scribe-gateway init: which store holds which repository. Back it up with the data roots.

  • The gateway's own identity. The gateway acts as the scribe-gateway service account: an account with no password that never signs in and only holds credential files. It got two grants in studio, one per job:

    • Namespace reader. The gateway polls every store's inventory as itself. Inventory is a namespace-wide read, so it needs this grant. It can read every repository in studio; it cannot write or administer.
    • Delegate. Alice and Bob present their own access certificates to the gateway, which passes each user's certificate on to storage. Storage checks the delegate grant, then that user's own permissions: the gateway never acts with more than the user's rights. See gateways and delegation.

    Its credential, secrets/gateway.credential, lives 365 days; stage 4's renew replaces it in time. Keep it on the gateway's machine and never give it to clients.

By hand, the same steps are bare metal, stage 3 and Docker Compose, stage 3.

2. Add storage B

There is no add storage: a second store is two short edits and a setup, which provisions whatever the definition names and is missing. Append this table to the end of scribe-cluster.toml:

toml
# A second storage server, added by hand.
[[clusters.local.storage]]
name = "storage-b"
listen = "127.0.0.1:7448"
data_dir = "data/storage-b"
auth = true
namespace = "studio"
credential = "secrets/storage-b.credential"
monitor_credential = "secrets/monitor.credential"
[clusters.local.storage.tls]
cert = "pki/storage-b/server.pem"
key = "pki/storage-b/server-key.pem"

and this entry to the end of scribe-gateway.toml, so the gateway routes to it:

toml
[[instances]]
name = "storage-b"
namespace = "studio"
endpoint = "tls://127.0.0.1:7448"
ca = "pki/ca/ca.pem"

Then plan and apply:

sh
scribe-ctl --config scribe-cluster.toml setup --plan
scribe-ctl --config scribe-cluster.toml setup
text
Scribe · local · Setup complete
  issued pki/storage-b/server.pem
  initialized data/storage-b
  storage-b: bound to namespace studio
  server storage-a: already registered in studio
  storage-a: credential kept (valid until 2027-09-27)
  server storage-b: registered in studio
  storage-b: credential issued to secrets/storage-b.credential
  scribe-gateway: present with its grants in studio
  scribe-gateway: credential kept (valid until 2027-09-27)
  scribe-monitor: present with its grants in studio
  scribe-monitor: credential kept (valid until 2027-09-27)
  wrote CLUSTER-INFO.txt

Setup did for B what add auth did for A: a certificate, a new data root bound to studio, a server registration and B's own credential. Everything that already existed is only checked. A and B share auth and the namespace but have distinct data roots, identities, and credentials. Never copy A's root to create B. By hand: bare metal, stage 4.

3. Start the cluster

sh
scribe-ctl --config scribe-cluster.toml up
text
Scribe · local · running
  SERVICE    STATE        PID  ENDPOINT
  auth       running    17999  tls://127.0.0.1:7443
  storage-a  running    18000  tls://127.0.0.1:7447
  storage-b  running    18001  tls://127.0.0.1:7448
  gateway    running    18002  tls://127.0.0.1:7557

up starts auth, then both stores, then the gateway, and returns only once the gateway has fresh inventory from every instance (require_healthy = true in the definition).

4. Move client connections to the gateway

Edit client-env.sh: change only SCRIBE_ENDPOINT from tls://127.0.0.1:7447 to tls://127.0.0.1:7557. Reload it, then log in for the new endpoint:

sh
. ./client-env.sh
alice login alice --auth tls://127.0.0.1:7443 --auth-ca ./pki/auth-ca/ca.pem
bob login bob --auth tls://127.0.0.1:7443 --auth-ca ./pki/auth-ca/ca.pem
alice repo list
(
  cd work/alice
  alice sync
  alice log
)
text
studio/game              head=3 id=689a965e479c1decf32f3c9ca7e3f913

Use the existing passwords. A saved session belongs to the server it was made for, so the direct-storage session is not a gateway login. Through the gateway, repository names carry their namespace (studio/game); a bare game still resolves. Nothing else changes for clients: they present the same kind of access certificate to the gateway as to storage.

game has the same repository ID and history. The authenticated workspaces from stage 2 remain valid because requests reach the original store. The helper's --server overrides the workspace's recorded direct endpoint for each invocation; a plain scribe sync in that workspace still uses the saved direct address unless SCRIBE_SERVER is set. Use alice and bob consistently.

5. Place new projects and inspect the result

Alice can create repositories because she administers studio:

sh
alice repo create prototypes --storage-compression lz4
alice repo create audio --storage-compression lz4
scribe-ctl --config scribe-cluster.toml repo list
scribe-gateway --config scribe-gateway.toml status
text
Scribe · local · storage inventory

  SERVER     REPOS          USED     AVAILABLE  ENDPOINT
  storage-a      2      2.80 MiB    436.11 GiB  tls://127.0.0.1:7447
  storage-b      1      2.25 MiB    436.11 GiB  tls://127.0.0.1:7448

  Total: 3 repositories · 5.05 MiB used

  storage-a · repositories
    REPOSITORY      HEAD          SIZE  ID
    game               3    608.93 KiB  689a965e479c1decf32f3c9ca7e3f913
    audio              0      2.02 MiB  a5b3de3fcdf6e500450c59bc3fedd4d4

  storage-b · repositories
    REPOSITORY      HEAD          SIZE  ID
    prototypes         0      2.02 MiB  cda14d1dd062c896472cb400953d0a41
text
gateway: running
storage-a: online repos=2 bytes=1940939 free=465212555264
storage-b: online repos=1 bytes=1355998 free=465212555264

repo list asks each store directly (as scribe-monitor); scribe-gateway ... status shows what the gateway itself observes, updated each poll (add --json for details). The gateway placed the two new repositories itself. Placement depends on the observed load, free space and pending reservations, so inspect the result rather than assuming a store. Bob's game grant does not cover the new repositories; add grants for their full IDs if he needs access.

6. Apply this to real scaling

The two demo roots share one machine and likely one filesystem. This proves routing, not additional hardware capacity. For independent disk, CPU, or network capacity, run the stores on separate suitable volumes or hosts:

  1. Give the new store's data_dir a disk of its own, or run it on another machine with its own cluster definition.
  2. Give its [tls] table the DNS names and addresses clients and the gateway use (dns = [...], ip = [...]), so its certificate matches them.
  3. Add the gateway [[instances]] entry with that endpoint. Set optional capacity_bytes consistently across instances if they should have different placement weights. A store in another namespace also needs the scribe-gateway reader and delegate grants there, which setup adds.
  4. Stop the cluster, run setup, then up, which waits for healthy inventory before you create repositories through the gateway.

Existing repositories stay pinned to their original instance; adding C does not move data out of a full A. A failed A makes A's repositories unavailable instead of redirecting them to B. Keep names unique across the cluster, keep every configured instance that owns routes, and back up each data root plus the stopped gateway's routing ledger. The gateway itself is a single process and a potential throughput bottleneck. See placement and limits and capacity planning.

Continue with 4. Operate the cluster.

Source docs/tutorials/03-add-gateway.mdSnapshot 2b8dcccb