# Gauntlet documentation

Generated from https://gauntlet-db.stavros.xyz/docs. Gauntlet is pre-beta; pages marked "Coming" describe work that is not available.

# Gauntlet documentation

> Status: Pre-beta. Gauntlet is a distributed SQL database whose only durable store is one object-store bucket. These pages cover what works today, what is refused, and what is coming.

Gauntlet is a distributed SQL database. Its only durable store is one S3-compatible bucket. Nodes hold nothing you cannot lose: memory and local disk are caches, and a node that dies is replaced by starting another one against the same bucket.

> **Pre-beta.** Gauntlet is pre-release software. Its own beta acceptance run (three hosts, node kills, 48–72 hours of load and a restore drill) has not passed yet. There is no public download; builds go to invited testers. [Join the waitlist](https://gauntlet-db.stavros.xyz/start) to be told when that changes.

## The short version

- **Transactions are strictly serializable**, including transactions that span shards. A write is acknowledged only after the bucket holds what recovery needs.
- **SQL is the SQLite family.** The engine is Turso. It is not PostgreSQL SQL.
- **Clients connect over the PostgreSQL wire protocol** (psql, psycopg, node-postgres, pgjdbc) or over an HTTP JSON API.
- **One process type.** `gauntlet serve` runs on each host and serves many named databases. Nodes find each other through leases in the bucket.
- **Idle databases cost only their bytes.** A released database makes no bucket requests until its next query.

## Start here

| Page | Use it to |
|---|---|
| [Quickstart](https://gauntlet-db.stavros.xyz/docs/quickstart.md) | Run one node, then three nodes on one bucket, and connect a client |
| [Connect an agent](https://gauntlet-db.stavros.xyz/docs/agents.md) | Hand a coding agent an accurate, secret-free prompt |
| [Transactions](https://gauntlet-db.stavros.xyz/docs/concepts/transactions.md) | Learn the guarantee and the two error codes every client must handle |
| [SQL compatibility](https://gauntlet-db.stavros.xyz/docs/sql/compatibility.md) | See which SQL works and what is refused |
| [Limits](https://gauntlet-db.stavros.xyz/docs/limits.md) | Know what pre-beta Gauntlet does not do before you rely on it |

## Who it fits today

It fits you if all of these are true:

- You can run three Linux x86_64 hosts and one bucket, and you are comfortable with TLS certificates, systemd or Docker, and Prometheus.
- Your data can be reloaded or restored from your own backup. There is no in-place upgrade from the beta to v1.
- Your application can retry a transaction on `40001` and can handle an unknown commit outcome (`40003`).
- A SQLite-style dialect is acceptable, and you do not need PostgreSQL catalogs, extensions or an ORM's introspection queries.

It does not fit you if you need any of these today: production guarantees, PostgreSQL SQL, `ALTER TABLE` or `CREATE INDEX`, point-in-time restore, per-user roles, or storage that shrinks when you delete data. Most of those are on the [roadmap](https://gauntlet-db.stavros.xyz/docs/roadmap.md).

## For agents and tooling

Every page has a **Copy as Markdown** button and a raw Markdown URL (add `.md` to the page's address). The whole documentation is indexed at [/llms.txt](https://gauntlet-db.stavros.xyz/llms.txt) and concatenated at [/llms-full.txt](https://gauntlet-db.stavros.xyz/llms-full.txt). Both are generated from the same files as these pages.

---
Source: https://gauntlet-db.stavros.xyz/docs

# Quickstart

> Status: Available. Run one node on your own machine, then three nodes on one bucket with TLS, and connect a client.

| | |
|---|---|
| **Works** | One node on loopback with a local directory; three nodes on one qualified bucket with TLS; psql, psycopg 3, node-postgres, pgjdbc and HTTP clients. |
| **Differs** | The local `fs:` store is for development only. It says nothing about how a real bucket behaves. |
| **Missing** | A public download, a macOS or Windows server binary, a managed service. |
| **Operational impact** | Creating a database reserves an immutable ID. Dropping a database does not reclaim its data in pre-beta. |
| **Evidence** | Steps follow the project's beta guide. The beta acceptance run has not passed yet. |

> **You need a build first.** There is no public download yet. Pre-beta builds are a release directory (binary, checksums, build record, example config and systemd unit) handed to invited testers. [Join the waitlist](https://gauntlet-db.stavros.xyz/start).

## What you need

- **The release directory**, all files from the same release.
- **Linux x86_64 with glibc 2.34 or newer** (Ubuntu 22.04, Debian 12, RHEL 9 or later). The binary does not run on macOS.
- **For part 2:** three hosts that can reach each other on TCP 7101, and one bucket.

Check and install the binary:

```sh
sha256sum --check --ignore-missing SHA256SUMS
install -d -m 0755 "$HOME/.local/bin"
install -m 0755 gauntlet-*-linux-x86_64 "$HOME/.local/bin/gauntlet"
export PATH="$HOME/.local/bin:$PATH"
gauntlet --version        # must list "engines: turso"
```

## Part 1: one node on loopback

This runs one node against a directory on local disk. It needs no bucket and no certificates, because every listener is on 127.0.0.1.

```sh
mkdir -p "$HOME/gauntlet-dev" && cd "$HOME/gauntlet-dev"
export GAUNTLET_CLIENT_TOKEN=$(openssl rand -hex 32)
export GAUNTLET_PEER_TOKEN=$(openssl rand -hex 32)
export GAUNTLET_ADMIN_TOKEN=$(openssl rand -hex 32)

gauntlet --cache-dir "$PWD/cache" --cache-mib 512 serve \
  --store "fs:$PWD/bucket" --id n1 \
  --listen 127.0.0.1:7101 --pg-listen 127.0.0.1:7151 \
  --memory-mib 4096
```

`--memory-mib 4096` is the smallest budget allowed. The node checks its store before it serves:

```sh
curl -s http://127.0.0.1:7101/health     # "ok": the process answers
curl -si http://127.0.0.1:7101/ready     # 200 when it can serve, 503 when it cannot
```

In a second terminal, with the same three tokens exported, create a database and get its token:

```sh
export GAUNTLET_URL=http://127.0.0.1:7101
gauntlet admin create-database app > app-create.json   # keep this file
gauntlet admin list
TOKEN=$(gauntlet admin token app)
```

Connect. The database token is the password, for any user name:

```sh
PGPASSWORD=$TOKEN psql "host=127.0.0.1 port=7151 dbname=app user=app" \
  -c "CREATE TABLE items (id INTEGER PRIMARY KEY, name TEXT NOT NULL)" \
  -c "INSERT INTO items VALUES (1, 'first')" \
  -c "SELECT * FROM items"
```

Stop the node with Ctrl-C. Start it again with the same command and the row is still there: everything it needs is under `bucket/`. The `cache/` directory is disposable. A restart with the same `--id` takes 17–18 s to become ready, because it waits out the previous run's 15 s lease.

## Part 2: three nodes on one bucket

The examples use hosts `n1.example.net`, `n2.example.net` and `n3.example.net`, and an admin machine that is none of them.

### Choose a bucket

Gauntlet needs more than the S3 API. It needs atomic create-if-absent (`If-None-Match: *`), read-after-write consistency and a strongly consistent, ordered LIST. `gauntlet qualify` tests this for you, and every node start runs the same checks again. A bucket that fails is refused.

| Provider | Status |
|---|---|
| Tigris | Run by the project. Recommended: one region, next to the hosts |
| Cloudflare R2 | Run by the project. Conditional writes were slower in the project's runs, so commits are slower |
| AWS S3 | **Unverified.** No run is recorded. Qualify it first |
| MinIO | **Unverified.** No run is recorded. Qualify it first |

Details, including the environment variables, are in [deploy and qualify a bucket](https://gauntlet-db.stavros.xyz/docs/operations/deploy.md).

### Secrets, certificates and config

On the admin machine, write one secrets file that every node uses, test the bucket, and issue certificates from the deployment's private CA:

```sh
umask 077
cat > secrets.env <<ENV
GAUNTLET_S3_ENDPOINT=https://t3.storage.dev
GAUNTLET_S3_REGION=auto
GAUNTLET_S3_BUCKET=my-gauntlet-bucket
GAUNTLET_S3_ACCESS_KEY_ID=<access key id>
GAUNTLET_S3_SECRET_ACCESS_KEY=<secret access key>
GAUNTLET_CLIENT_TOKEN=$(openssl rand -hex 32)
GAUNTLET_PEER_TOKEN=$(openssl rand -hex 32)
GAUNTLET_ADMIN_TOKEN=$(openssl rand -hex 32)
ENV

(set -a; . ./secrets.env; gauntlet qualify --store 's3:#qualify-check')

gauntlet tls init --out ca
for n in 1 2 3; do
  gauntlet tls issue --ca ca --node n$n --dns n$n.example.net --dns localhost --out n$n
done
```

Write `/etc/gauntlet/gauntlet.toml` on each host. Only `id` and `advertise` differ between nodes:

```toml
store = "s3:#gauntlet/beta"          # GAUNTLET_S3_* plus this prefix; same on every node
id = "n1"                            # unique per node
listen = "0.0.0.0:7101"              # HTTP: clients, admin and peers
advertise = "n1.example.net:7101"    # a name in this node's certificate
pg-listen = "0.0.0.0:7151"           # PostgreSQL wire protocol
tls-cert = "/etc/gauntlet/tls/node.crt"
tls-key = "/etc/gauntlet/tls/node.key"
tls-peer-ca = "/etc/gauntlet/tls/ca.crt"
cache-dir = "/var/lib/gauntlet/cache"
cache-mib = 8192
memory-mib = 16384
log-format = "json"
```

Start all three with the packaged systemd unit. They find each other through leases in the bucket. A node answers neither `/health` nor `/ready` until its start-up bucket check passes.

### Create a database

```sh
export GAUNTLET_URL=https://n1.example.net:7101 GAUNTLET_TLS_CA=ca/ca.crt
set -a; . ./secrets.env; set +a
umask 077
gauntlet admin create-database app > app-create.json
TOKEN=$(gauntlet admin token app)
```

If `create-database` reports an unknown outcome, retry with `gauntlet admin create-database --reservation app-create.json`. Never retry by name: that reserves a second ID.

## Connect

Any node serves any database. Use the database name as `dbname`, any user name, and the database token as the password.

```sh
PGPASSWORD=$TOKEN psql "host=n1.example.net port=7151 dbname=app user=app \
  sslmode=verify-full sslrootcert=ca/ca.crt channel_binding=require" \
  -c "SELECT 1"
```

Driver examples are in [PostgreSQL wire](https://gauntlet-db.stavros.xyz/docs/clients/pgwire.md); the JSON API is in [HTTP](https://gauntlet-db.stavros.xyz/docs/clients/http.md).

## Next

- Read [transactions](https://gauntlet-db.stavros.xyz/docs/concepts/transactions.md): your client must handle `40001` and `40003`.
- Take a first [backup](https://gauntlet-db.stavros.xyz/docs/operations/backups.md) and practise a restore.
- Read the [limits](https://gauntlet-db.stavros.xyz/docs/limits.md) before you put data in.

---
Source: https://gauntlet-db.stavros.xyz/docs/quickstart

# Connect an agent

> Status: Available. Give a coding agent an accurate prompt, a database-scoped token and nothing else.

| | |
|---|---|
| **Works** | A copyable prompt that describes the real protocol, dialect and error rules. Database-scoped tokens. Documentation as Markdown and llms.txt. |
| **Differs** | The prompt assumes the token is already in the environment. It never contains a secret. |
| **Missing** | An installer, an SDK package, an MCP server and agent skills. They are planned, not shipped. |
| **Operational impact** | The connectivity check `SELECT 1` may activate that one idle database. Nothing else is woken. |
| **Evidence** | The prompt text is generated from the same source as the site's Copy agent prompt button. |

## The prompt

Use the **Copy agent prompt** button on the [home page](https://gauntlet-db.stavros.xyz/), or copy it from the top of this page. It tells the agent:

- what Gauntlet is (SQLite-family SQL, PostgreSQL wire as a transport only);
- how to connect over PostgreSQL wire or HTTP;
- which errors it must handle, and that `40003` is never retried blindly;
- what it must not do: use an admin token, create or drop databases, or touch other databases.

The prompt points at this documentation's Markdown, so the agent reads the current contract instead of guessing from PostgreSQL habits.

## Environment variables

The prompt uses three names. They are a convention for your project, not a setting read by a Gauntlet SDK:

| Variable | Holds |
|---|---|
| `GAUNTLET_DATABASE_URL` | A PostgreSQL connection string without the password, for example `postgres://app@n1.example.net:7151/app?sslmode=verify-full` |
| `GAUNTLET_HTTP_URL` | The HTTP base URL of a node, for example `https://n1.example.net:7101` |
| `GAUNTLET_DATABASE_TOKEN` | The database token |

If your project already has its own names, keep them and tell the agent.

## Which credential to hand over

| Credential | Give it to an agent? |
|---|---|
| Database token | **Yes.** It opens exactly one database. Anyone with it can do everything in that database |
| Client token | **No.** Every database's token is derived from it |
| Admin token | **No.** It opens `/admin`, `/status` and `/metrics` for the deployment |
| Bucket credentials | **No.** The bucket is the database |

There are no read-only or short-lived tokens yet. A read-only token and platform-issued short-lived tokens are on the [roadmap](https://gauntlet-db.stavros.xyz/docs/roadmap.md).

## One database per agent

The unit of isolation is the database. Clients of different databases are mutually untrusted, and a token is bound to one immutable database ID. Giving each agent its own database keeps one agent's mistakes out of another's state, and an agent that is idle costs only its stored bytes ([scale to zero](https://gauntlet-db.stavros.xyz/docs/concepts/scale-to-zero.md)).

Creating databases needs the admin token today, so an agent should not do it. Run `gauntlet admin create-database` from your own backend. Scoped creation inside a project is planned.

## Coming

- **MCP server and skills.** Not shipped. Do not configure an MCP URL for Gauntlet until one is published here.
- **SDK helpers** that hide `40001` retries and `40003` lookups.
- **Idempotency keys**, so an agent's retried tool call commits at most once.

---
Source: https://gauntlet-db.stavros.xyz/docs/agents

# The bucket is the database

> Status: Available. One object-store bucket holds everything durable. Nodes are disposable compute that recover from it.

Gauntlet has one durable component: an S3-compatible bucket. Memory and local disk are caches. A node that dies is replaced by starting another one against the same bucket and prefix.

## What an acknowledgement means

A write is acknowledged only after the bucket holds what recovery needs. This is the default and, today, the only mode. There is no acknowledgement from memory, from a local disk or from a peer.

That has two consequences worth planning for:

- **Losing every node loses nothing.** Start nodes against the same bucket and prefix and the databases are there.
- **Commit latency follows the bucket.** Each commit waits for conditional writes to the bucket. Put the bucket next to the nodes.

A per-database *peer-durable* mode, which would acknowledge on peers and upload afterwards, is designed but not built. Bucket-only stays the default.

## What is in the bucket

- **A journal per shard**, written with create-if-absent. Two writers can never both win the same slot.
- **Ownership barriers** that fence an old owner once a new one takes over.
- **Write-once decision objects** for transactions that span shards.
- **Checkpoints**, so recovery does not replay the whole journal.
- **Leases**, through which nodes discover each other. There is no member list to maintain.

## What a node is

`gauntlet serve` is the only server process. Each node:

- accepts PostgreSQL wire and HTTP connections for **any** database;
- owns some databases' shards at a given moment and routes requests for the others to their owners;
- keeps a bounded, verified block cache on local disk. The cache is rebuilt from the bucket if it is lost.

Normal SQL does not pass through a control plane. There is no control plane in the query path.

## What the bucket must provide

More than the S3 API: atomic create-if-absent (`If-None-Match: *`), read-after-write consistency and a strongly consistent, ordered LIST. `gauntlet qualify` checks this, and every node start runs the same checks. A bucket that fails is refused. See [deploy and qualify a bucket](https://gauntlet-db.stavros.xyz/docs/operations/deploy.md).

The bucket is **trusted**: Gauntlet does not encrypt or sign its objects. Scope the bucket key to that bucket, and never set lifecycle or expiry rules on the prefix.

## Failure, in one table

| Event | What happens |
|---|---|
| One node is killed | The other nodes take over its databases through normal recovery |
| All nodes are lost | Nothing is lost. Start nodes against the same bucket |
| The local cache disk is lost | Nothing is lost. The cache is rebuilt |
| The bucket is unreachable | No node can commit. `/ready` turns 503. More nodes do not help |

No failover time is guaranteed in pre-beta. See [node failure and recovery](https://gauntlet-db.stavros.xyz/docs/operations/failover.md).

## Evidence and its limits

The commit and recovery protocol is checked by fault-injection tests, deterministic simulation and bounded TLA+ models, which gate changes to the engine. These check limited compositions. They are engineering evidence, not an independent audit and not a proof about arbitrary deployments.

---
Source: https://gauntlet-db.stavros.xyz/docs/concepts/architecture

# Transactions

> Status: Available. Every transaction is strictly serializable, including across shards. Clients must retry conflicts and look up unknown outcomes.

| | |
|---|---|
| **Works** | Strict serializability as the default and only isolation level. Atomic, isolated commits across shards inside one database. Interactive transactions and atomic HTTP batches. |
| **Differs** | Locking is no-wait, so a conflicting transaction fails at once with 40001 and the client retries. |
| **Missing** | Savepoints. Transactions that span separate databases. Server-side automatic retry. |
| **Operational impact** | A commit waits for conditional writes to the bucket. |
| **Evidence** | Fault-injection tests, deterministic simulation and bounded TLA+ models. Not an independent audit. |

## The guarantee

Transactions are **strictly serializable**: the result is the same as running them one at a time, in an order consistent with real time. If transaction A is acknowledged before transaction B starts, B sees A.

This holds for every transaction. There is no weaker default to opt out of, and a stale read is never given silently.

It also holds **across shards**. A transaction that touches several shards of one database commits atomically and in isolation: prepare on each shard, one write-once decision object, then resolve. Index entries are written in the same attempt as the row, so a unique index is never briefly wrong.

The guarantee is per database. There is no transaction across two separate databases.

## What you must handle

Two error codes are part of normal operation. No stock PostgreSQL driver handles either for you.

### 40001: conflict, retry the whole transaction

Locking is **no-wait**. A transaction that meets a conflicting lock is not queued; it fails with `40001`. An ownership change while a transaction is open gives the same code.

Roll back, then run the whole transaction again. Do not retry only the failed statement.

```python
import psycopg

def run(conn, work, attempts=5):
    for attempt in range(attempts):
        try:
            with conn.transaction():
                return work(conn)
        except psycopg.errors.SerializationFailure:
            continue  # 40001: retry the whole transaction
    raise RuntimeError("still conflicting after retries")
```

### 40003: the commit outcome is unknown

`40003` means the commit may or may not have happened, for example because a bucket write timed out or a node stopped mid-commit. **Do not retry yet.** The error carries the transaction ID. Ask any node:

```sh
curl --cacert ca/ca.crt -H "Authorization: Bearer $TOKEN" \
  https://n1.example.net:7101/db/app/txn/<txid>
```

| `status` | Meaning |
|---|---|
| `committed` | Final. Do not run it again |
| `aborted` | It did not happen. You may run the operation again, as a new transaction |
| `unknown`, `quarantined`, or HTTP 503 `lookup_failed` | Not resolved. Repeat the lookup. Do not replay the operation |

Only `aborted` permits a new attempt. An unknown outcome is not a confirmed rollback.

## Read-only transactions

`BEGIN READ ONLY` gives a consistent read-only image of the whole database while writers continue. A write inside it returns `25006`.

## Things that differ from PostgreSQL

- **Any error inside `BEGIN … COMMIT` aborts the block.** Later statements return `25P02` until you send `ROLLBACK`.
- **No savepoints.** `ON CONFLICT ABORT`, `FAIL` and `ROLLBACK` discard the whole transaction, not one statement.
- **`CREATE TABLE` is not transactional.** It is one standalone statement and commits at once.
- **A multi-statement simple query is one implicit transaction.**

## Coming

- **Server-side retry of standalone statements and SDK helpers** for `40001` and `40003`.
- **Idempotency keys** recorded in the same commit, for exactly-once writes inside a published window.

---
Source: https://gauntlet-db.stavros.xyz/docs/concepts/transactions

# Scale to zero

> Status: Available. A database that is not in use is released. While released it makes no bucket requests; its next query activates it.

| | |
|---|---|
| **Works** | Release after 30 s without use. Zero bucket requests while released. Activation on the first request. Readiness checks that do not wake a database. |
| **Differs** | The first request after a release is slower, because it activates the database. |
| **Missing** | A published activation latency for your deployment. A qualification above 1,000 databases. |
| **Operational impact** | Listing databases and `GET /db/<name>/ready` do not activate a database. A SQL statement does. A scheduled backup does too. |
| **Evidence** | The project's own run at 1,000 databases. Larger qualifications are planned. |

## What happens

A node serves many named databases. A database that has gone **30 seconds without use** is *released*: its owner writes what it must and lets go. From then on the database makes **no bucket requests**. Its incremental idle footprint is the bytes it already has in the bucket.

The next request for that database *activates* it on some node. That request is slower than the ones after it.

| State | Meaning |
|---|---|
| Released (idle) | No current owner. No bucket requests. Healthy |
| Activating | A node is loading it for a request |
| Serving | A node owns it and answers requests |

A released database is not down. It has no owner because it needs none.

## What does and does not wake a database

| Action | Activates it? |
|---|---|
| A SQL statement over PostgreSQL wire or HTTP | Yes |
| `GET /db/<name>/ready` with the database token | No |
| `gauntlet admin list` | No |
| A scheduled backup | **Yes, today.** Backups that skip idle databases are planned |

This matters for tools: a dashboard or agent that lists a fleet must never run a query per database to do it.

## What "free while idle" does and does not mean

- It means the database's **incremental** cost is its stored bytes: no polling, no heartbeats, no bucket requests.
- It does **not** make a machine you run free, or an allocated node, or the bucket's storage bill.
- Storage only grows in pre-beta. Deleted rows and dropped databases are not reclaimed yet.

## How far this has been shown

The project's own evidence is a run with **1,000 databases** on one deployment. In that run the first query against a released database took about 250 ms at the median. That is one deployment's measurement and not a promise for your hosts and bucket. Measure activation on your own deployment.

Qualifications at 100,000 and at one million databases are planned and have not been run. The bytes and object count of one empty database are not yet published.

---
Source: https://gauntlet-db.stavros.xyz/docs/concepts/scale-to-zero

# Databases, identity and tokens

> Status: Available. A database has a name, an immutable ID and one token. The database is the unit of isolation.

## Name and ID

`gauntlet admin create-database app` reserves an **immutable ID** and gives the database the name `app`. Clients use the name: it is the `dbname` on PostgreSQL wire and the path segment in `/db/<name>/…` over HTTP.

Keep the JSON that `create-database` prints. If the command reports an unknown outcome, retry with `--reservation <file>`, never by name: a retry by name reserves a second ID.

`gauntlet admin drop-database app --id <id>` drops one exact ID. Its data is not reclaimed in pre-beta.

## One token per database

Each database has exactly one token.

- It is derived from the deployment's client token and the database's ID. `gauntlet admin token app` computes it locally.
- It is the PostgreSQL password (for any user name) and the HTTP bearer token.
- Anyone who holds it can do everything in that database. There are no SQL roles and no table privileges.

| Token | Scope |
|---|---|
| Database token | One database. Hand this to applications and agents |
| Client token | The whole deployment's client access: every database token is derived from it. Guard it like the admin token |
| Peer token | Node-to-node authentication, with certificates |
| Admin token | `/admin`, `/status` and `/metrics` |

## The database is the boundary

Clients of different databases are mutually untrusted. A token is bound to one immutable ID and cannot open another database.

This is why Gauntlet suits **one database per agent, per user or per tenant**: isolation is a property of the boundary, not of row-level rules you have to get right. Idle databases are released and cost only their stored bytes ([scale to zero](https://gauntlet-db.stavros.xyz/docs/concepts/scale-to-zero.md)).

What the project has shown is a deployment with 1,000 databases. A fleet of that size works today; qualification at 100,000 and one million is [planned](https://gauntlet-db.stavros.xyz/docs/roadmap.md).

## Today's limits

- **One shard per new database.** One database is limited to what one shard owner can serve. Shard split and move are not built.
- **No listing with labels or paging.** `gauntlet admin list` is a flat list. Labels for agent, tenant and environment, and a paged listing that never activates a database, are planned.
- **Creation needs the admin token.** Scoped creation inside a project is planned.

## Coming

- **Read-only tokens** and **short-lived tokens**.
- **Tenant scopes**: create, list and drop inside a scope, with per-scope limits.
- **A fleet listing** with labels, filters and paging, read from usage rows so that listing never wakes a database.

---
Source: https://gauntlet-db.stavros.xyz/docs/concepts/databases

# Branches and point-in-time recovery

> Status: Coming. Copy-on-write branches and restore to an exact transaction are planned. Neither is available today.

> **Coming.** Nothing on this page is available in pre-beta. Today a backup is one whole image, and a restore creates a new database. See [backups and restore](https://gauntlet-db.stavros.xyz/docs/operations/backups.md).

## What is planned

| Feature | Plan item | What it will do |
|---|---|---|
| Fork by copy | P10.16 | Create a new database from a copy of another. Not instant: the data is copied |
| Copy-on-write branches | P6.8 | Fork with no data copied |
| Restore to a transaction ID | P10.17 | Restore to an exact transaction, one cut across all shards, into a new identity |
| Schema-only branches | P10.39 | A new database with the parent's schema and no data |

## Why they wait

Branches and point-in-time restore both need Gauntlet to keep objects "back to here". That depends on storage reclamation being switched on and stable first, and reclamation is off in pre-beta. So these features are built after it, not before.

## What will not be promised

- **No automatic data merge.** Merging two SQL databases is an application problem. The pattern is: fork, let the work run, then keep the child or drop it.
- **No wall-clock exactness.** The exact cut is a transaction ID. A wall-clock time is a way to find one, with the resolved ID shown before you confirm.
- **No in-place restore.** A restore always creates a new database with a new ID and a new token.

---
Source: https://gauntlet-db.stavros.xyz/docs/concepts/branches-and-recovery

# SQL compatibility

> Status: Partial. Gauntlet speaks the SQLite dialect over the PostgreSQL wire protocol. Those are two separate promises.

| | |
|---|---|
| **Works** | SELECT with joins, CTEs, subqueries and compound queries; INSERT with upsert and RETURNING; UPDATE; DELETE; CREATE TABLE; BEGIN, COMMIT, ROLLBACK; BEGIN READ ONLY; EXPLAIN; SHOW TABLES. |
| **Differs** | Types use SQLite affinity. Every table needs an explicit primary key. CREATE TABLE is standalone and commits at once. |
| **Missing** | ALTER TABLE, DROP TABLE, CREATE INDEX, savepoints, COPY, CHECK, foreign keys, triggers, PostgreSQL catalogs and extensions. Each is refused with 0A000 or 42601. |
| **Operational impact** | A refused statement changes nothing. Nothing returns success and does nothing. |
| **Evidence** | A tested subset checked against SQLite. Passing tests do not mean complete SQLite compatibility. |

- The **wire protocol** lets PostgreSQL drivers connect, authenticate, send statements and read results.
- The **SQL** is run by Turso, a SQLite-family engine. PostgreSQL syntax, catalogs (`pg_catalog`, `information_schema`), extensions and types are not there.

## What works

| Statement | Notes |
|---|---|
| `SELECT` | Joins, CTEs, subqueries and compound queries, as far as they are tested |
| `INSERT` | Multi-row `VALUES`, `INSERT … SELECT`, `DEFAULT VALUES`, `ON CONFLICT` (upsert), `OR IGNORE`/`OR REPLACE`, `RETURNING` |
| `UPDATE`, `DELETE` | One table, with `WHERE` and `RETURNING` |
| `CREATE TABLE` | With limits; see below |
| `BEGIN`, `COMMIT`, `ROLLBACK` | Interactive transactions |
| `BEGIN READ ONLY` | A consistent read-only image of the whole database. A write inside it returns `25006` |
| `EXPLAIN` | For `SELECT` and plain DML |
| `SHOW TABLES` | Lists the tables. This is the only `SHOW` |

Other things to know:

- **Parameters** are `$1`, `?1` or a bare `?`. Named parameters are refused.
- **Names are single-part.** `schema.table` is refused.
- **Types use SQLite affinity.** A declared length or precision is not enforced. `TEXT` is UTF-8.
- **Every table needs an explicit primary key.** Integer keys are not generated for you: your `INSERT` supplies them.

## What is refused

These return `0A000` (feature not supported), or `42601` when the parser rejects them.

| Refused | Use instead |
|---|---|
| `ALTER TABLE`, `DROP TABLE`, `CREATE INDEX` | Decide the schema and its `UNIQUE` constraints at `CREATE TABLE`. There is no way to change a table yet |
| `SAVEPOINT`, `ROLLBACK TO`, `COMMIT AND CHAIN` | Retry the whole transaction |
| `BEGIN` with a mode other than `READ ONLY` | Plain `BEGIN` |
| `UPDATE` or `DELETE` with `ORDER BY` or `LIMIT` | Select the keys first |
| Several statements in one prepared statement | One statement per prepare |
| Writing `rowid`, `_rowid_` or `oid` | Use your own key column |
| `COPY`, `SHOW <setting>`, SQL `DEALLOCATE`, procedural blocks | — |
| `CHECK`, foreign keys, triggers, generated columns, `STRICT`, non-`BINARY` collations in a table definition | Enforce in the application |
| Cancel requests | Ignored. The statement runs to its end or to the 20 s limit |

## CREATE TABLE

```sql
CREATE TABLE IF NOT EXISTS orders (
  id INTEGER PRIMARY KEY,
  customer TEXT NOT NULL,
  ref TEXT UNIQUE,
  qty INTEGER DEFAULT 1
);
```

- **One standalone statement.** Send it outside a transaction and not mixed with other statements. Otherwise it returns `25001`.
- **It commits at once.** A later error in the same batch does not undo it. This is not PostgreSQL's transactional DDL.
- **An explicit primary key is required**, inline or as a table constraint. Composite keys are fine.
- **Supported:** `UNIQUE` (column or table), `NOT NULL`, literal `DEFAULT`, `WITHOUT ROWID`.
- **`UNIQUE` is how you get an index.** It is built with the table, while the table is empty.
- **`IF NOT EXISTS`** only checks the name. It does not check that the existing table matches your definition.
- Limits: 4,096 tables, 1,024 indexes per table, 2,000 columns per new table.

## Not inherited from Turso

Gauntlet runs Turso's SQL layer over its own storage. Features that live in Turso's own storage are **not** available: its indexes, full-text search, vector index, change capture and MVCC. Do not read a Turso feature list as a Gauntlet feature list.

## Size limits

| Limit | Value | Past it |
|---|---|---|
| One transaction | 16 MiB, including overhead | `54000` |
| One value / one key | 8 MiB / 64 KiB | `54000` |
| One statement | 20 s or 500 million engine instructions | `57014` |
| Sort, `DISTINCT` and `GROUP BY` temporary data | 64 MiB | `54000` |
| Result columns | 2,000 | `54000` |

Queries build their intermediate results in memory. There is no spill to disk, so a large unindexed join can be slow or hit a limit.

## Coming

`ALTER TABLE … ADD COLUMN`, `CREATE INDEX`, `DROP TABLE`, generated integer keys, `DEFAULT CURRENT_TIMESTAMP`, a schema introspection surface, and a certified list of ORMs are v1 roadmap items. See the [roadmap](https://gauntlet-db.stavros.xyz/docs/roadmap.md).

---
Source: https://gauntlet-db.stavros.xyz/docs/sql/compatibility

# Error codes

> Status: Available. Errors use PostgreSQL SQLSTATE codes on both protocols. Four of them need handling in every client.

Over HTTP the same code is in the JSON body under `error`.

## The four you must handle

| Code | Meaning | What to do |
|---|---|---|
| `40001` | Serialization conflict or ownership change | Retry the whole transaction. Locking is no-wait, so this is normal under contention |
| `40003` | Commit outcome unknown | **Do not retry yet.** [Look the transaction up first](https://gauntlet-db.stavros.xyz/docs/concepts/transactions.md#40003-the-commit-outcome-is-unknown) |
| `57P03` | Cannot serve right now | Retry with backoff. Inside a transaction: `ROLLBACK`, then retry the whole transaction |
| `53300` | Too many connections | Back off or use another node. Keep your pool below the node's limit |

### 57P03 in detail

You can see `57P03` when connecting (the node is overloaded, the database could not be activated, or startup took too long), when a node is shutting down, under memory pressure, or when the bucket or a shard owner is unavailable. It is always retryable. Over HTTP it is status 503.

### 53300 in detail

Each node accepts **128 PostgreSQL connections** by default; the operator can change that with `max-connections` (1 to 10000). The count includes connections that are starting up and connections that are idle. One connection past the limit gets `FATAL: sorry, too many clients already`. The limit is per node, so other nodes may have room.

## Other codes

| Code | Meaning |
|---|---|
| `23505` / `23502` | UNIQUE / NOT NULL violation |
| `42P01` / `42703` / `42702` | Unknown table / unknown column / ambiguous column |
| `42P07` / `42P16` | Table already exists / invalid table definition |
| `42601` / `42000` | Syntax error / other planning error |
| `42883` / `42803` | Unknown function or wrong argument count / aggregate used where it is not allowed |
| `0A000` | Feature not supported |
| `25001` | `CREATE TABLE` inside a transaction or mixed with other statements |
| `25006` | Write in a read-only transaction |
| `25P02` | The transaction is aborted; send `ROLLBACK` |
| `25P01` | No transaction is open |
| `54000` | A size limit was exceeded (transaction, value, result, SQL text) |
| `57014` | The statement hit its limit: 20 s or 500 million engine instructions |
| `22003` | Integer or numeric overflow |
| `28P01` | Wrong password (database token) |
| `28000` | Plaintext connection when TLS is required |
| `3D000` | Database does not exist, or was dropped |
| `08P01` | Malformed protocol message or missing parameter |
| `53200` / `53100` | The engine could not allocate memory / storage full |
| `XX000` / `XX001` | Internal error / invalid stored data. Never retried automatically; report it |

## HTTP status mapping

| SQLSTATE | HTTP status |
|---|---|
| `40001` | 409 |
| `57P03` | 503 |
| `40003` | 504 |
| `XX…` | 500 |
| Every other SQL error | 400 |

---
Source: https://gauntlet-db.stavros.xyz/docs/sql/errors

# PostgreSQL wire

> Status: Available. Connect with psql, psycopg, node-postgres or pgjdbc. The wire protocol is PostgreSQL's; the SQL is not.

| | |
|---|---|
| **Works** | Simple and extended protocols, prepared statements, portals with a row limit, pipelining, SCRAM-SHA-256 and SCRAM-SHA-256-PLUS, TLS. |
| **Differs** | Ordinary columns are described as text in prepared statements. The user name is ignored. There are no date or time types. |
| **Missing** | COPY, cancel requests, named (server-side) cursors, per-user authorization, `\dt` and other catalog-backed psql commands. |
| **Operational impact** | A connection activates the database it names. An idle connection is closed after 300 s. |
| **Evidence** | psql 16.2 and 17, psycopg 3.3.6, node-postgres 8.23.1 and pgjdbc 42.7.14, on source builds. node-postgres and pgjdbc were run without TLS. |

Any node serves any database. Use the database name as `dbname`, any user name, and the [database token](https://gauntlet-db.stavros.xyz/docs/concepts/databases.md) as the password. Authentication is SCRAM-SHA-256; cleartext passwords are not offered. With TLS configured, a plaintext connection is refused with `28000`.

On a loopback development node, drop the TLS options from each example.

## psql

```sh
PGPASSWORD=$TOKEN psql "host=n1.example.net port=7151 dbname=app user=app \
  sslmode=verify-full sslrootcert=ca/ca.crt channel_binding=require" \
  -c "CREATE TABLE items (id INTEGER PRIMARY KEY, name TEXT NOT NULL)" \
  -c "INSERT INTO items VALUES (1, 'first')" -c "SELECT * FROM items"
```

`\dt`, `\copy` and `FETCH_COUNT` are not supported. Use `SHOW TABLES`.

## psycopg 3

```python
import os, psycopg

with psycopg.connect(host="n2.example.net", port=7151, dbname="app", user="app",
                     password=os.environ["GAUNTLET_DATABASE_TOKEN"], sslmode="verify-full",
                     sslrootcert="ca/ca.crt") as conn:
    print(conn.execute("SELECT name FROM items WHERE id = %s", (1,)).fetchone())
```

## node-postgres

```js
import fs from "node:fs";
import pg from "pg";

const client = new pg.Client({
  host: "n3.example.net", port: 7151, database: "app", user: "app",
  password: process.env.GAUNTLET_DATABASE_TOKEN,
  ssl: { ca: fs.readFileSync("ca/ca.crt", "utf8") },
});
await client.connect();
const { rows } = await client.query("SELECT name FROM items WHERE id = $1", [1]);
console.log(rows);
await client.end();
```

- node-postgres sends `true` and `false` as untyped text, so they are stored as the text `'true'` and `'false'`.
- A `Buffer` parameter is stored as a BLOB.
- **Unverified:** the project's node-postgres suite runs on loopback without TLS. The `ssl` option above has no recorded test.

## JDBC (pgjdbc)

```java
String url = "jdbc:postgresql://n1.example.net:7151/app"
           + "?sslmode=verify-full&sslrootcert=ca/ca.crt";
Properties props = new Properties();
props.setProperty("user", "app");
props.setProperty("password", System.getenv("GAUNTLET_DATABASE_TOKEN"));
try (Connection conn = DriverManager.getConnection(url, props);
     PreparedStatement st = conn.prepareStatement("SELECT name FROM items WHERE id = ?")) {
    st.setLong(1, 1);
    try (ResultSet rs = st.executeQuery()) { while (rs.next()) System.out.println(rs.getString(1)); }
}
```

- pgjdbc's default settings work, including binary transfer.
- `setQueryTimeout` is not enforced: cancel requests are ignored.
- **Unverified:** as with node-postgres, the JDBC suite runs without TLS.

## Driver behaviour

- **Column types in prepared statements are `text`** for ordinary columns, because SQLite affinity does not fix a type. Stable expressions keep their type: `CAST(x AS INTEGER)` is `int8`, `count(*)` is `int8`. Cast when your driver needs a number.
- **Results are text unless the driver asks for binary.** Binary works for int8, float8, text, bytea, numeric, bool, int2, int4, oid, float4, name, char and varchar.
- **`numeric` is not arbitrary precision.** A numeric parameter becomes an INTEGER if it has no fraction and fits in 64 bits, otherwise the nearest REAL.
- **`SET` works only when it keeps the server's value:** `client_encoding` UTF8, `DateStyle` ISO, `TimeZone` UTC, `extra_float_digits` 1 to 3. `application_name` can be set.
- **Per connection:** 256 prepared statements, 64 portals, 4,096 parameters, 1 MiB of SQL per statement.

## ORMs

No ORM is certified. Most ORMs start by reading PostgreSQL catalogs that Gauntlet does not have, so expect their migrations and introspection to fail. ORM support through the libSQL protocol is a [v1 roadmap item](https://gauntlet-db.stavros.xyz/docs/clients/libsql.md).

---
Source: https://gauntlet-db.stavros.xyz/docs/clients/pgwire

# HTTP JSON API

> Status: Available. Run a statement, an atomic batch or an interactive session over HTTPS with a bearer token.

| | |
|---|---|
| **Works** | POST /db/<name>/sql, POST /db/<name>/tx for an atomic list of statements, interactive sessions, GET /db/<name>/txn/<txid> for outcome lookup, GET /db/<name>/ready. |
| **Differs** | This is Gauntlet's own JSON shape. It is not the libSQL (Hrana) protocol. |
| **Missing** | A published OpenAPI schema, a libSQL-compatible endpoint, CORS for browsers. |
| **Operational impact** | /sql, /tx and sessions activate the database. /ready does not. HTTP requests are not counted against the PostgreSQL connection limit. |
| **Evidence** | Covered by the project's gateway tests. The request and response shapes may still change in pre-beta. |

All routes are under `/db/<name>/` on any node's HTTP listener (port 7101 in the examples). Send the [database token](https://gauntlet-db.stavros.xyz/docs/concepts/databases.md) as a bearer token over TLS.

## One statement

```sh
curl --cacert ca/ca.crt \
  -H "Authorization: Bearer $GAUNTLET_DATABASE_TOKEN" \
  -H 'Content-Type: application/json' \
  https://n3.example.net:7101/db/app/sql \
  -d '{"sql": "SELECT name FROM items WHERE id = $1", "params": [1]}'
```

The body is `{"sql": "…", "params": […]}`; `params` is optional. A result has `columns`, `rows` (an array of arrays) and `rows_affected`.

## An atomic batch

`/tx` runs a list of statements as one transaction. Either all commit or none do.

```sh
curl --cacert ca/ca.crt \
  -H "Authorization: Bearer $GAUNTLET_DATABASE_TOKEN" \
  -H 'Content-Type: application/json' \
  https://n3.example.net:7101/db/app/tx \
  -d '{"statements": [
    {"sql": "UPDATE accounts SET balance = balance - 5 WHERE id = $1", "params": [1]},
    {"sql": "UPDATE accounts SET balance = balance + 5 WHERE id = $1", "params": [2]}
  ]}'
```

The response has one entry in `results` per statement.

## Sessions

`POST /db/<name>/session` opens an interactive session and returns its ID; pass `"session": <id>` in later `/sql` bodies to run `BEGIN … COMMIT` across requests. `DELETE /db/<name>/session/<id>` closes it. Prefer `/tx` when the statements are known up front: it has no session to leak.

## Errors

An error is a JSON body with an `error` object that carries the same SQLSTATE code as PostgreSQL wire.

| SQLSTATE | HTTP status | Do |
|---|---|---|
| `40001` | 409 | Retry the whole transaction |
| `57P03` | 503 | Retry with backoff |
| `40003` | 504 | [Look the outcome up](https://gauntlet-db.stavros.xyz/docs/concepts/transactions.md#40003-the-commit-outcome-is-unknown). Do not retry |
| `XX…` | 500 | Report it |
| Any other | 400 | Fix the request |

## Outcome lookup

```sh
curl --cacert ca/ca.crt -H "Authorization: Bearer $GAUNTLET_DATABASE_TOKEN" \
  https://n1.example.net:7101/db/app/txn/<txid>
```

Returns `{"txid": "…", "status": "committed" | "aborted" | "unknown" | "quarantined"}`, or HTTP 503 with `lookup_failed`.

## Readiness without waking

`GET /db/<name>/ready` with the database token reports that database's readiness **without activating it**. Use it for health checks; never use `SELECT 1` for a fleet-wide probe.

## Values

Integers and floats are JSON numbers, text is a string, a BLOB is `{"$blob": "<hex>"}`, and infinite floats are `{"$float": "Infinity"}` or `{"$float": "-Infinity"}`.

## The CLI

```sh
GAUNTLET_CLIENT_TOKEN=$TOKEN gauntlet sql --url https://n3.example.net:7101/db/app 'SELECT * FROM items'
```

---
Source: https://gauntlet-db.stavros.xyz/docs/clients/http

# libSQL clients and ORMs

> Status: Coming. A libSQL-compatible HTTP endpoint, and the ORMs that come with it, are planned for v1. They are not available today.

> **Coming.** Gauntlet's [HTTP API](https://gauntlet-db.stavros.xyz/docs/clients/http.md) is its own JSON shape. A libSQL SDK pointed at a Gauntlet node will not work today.

## What is planned

| Item | Plan | What it will add |
|---|---|---|
| libSQL-compatible HTTP endpoint | P10.2 | A route to the existing libSQL client and ORM ecosystem, in the dialect Gauntlet already speaks |
| Certified client and ORM list | P10.8 | One page of what works, what is partial and what is refused, per combination |
| SDK helpers | P10.7 | Retry on `40001` and outcome lookup on `40003`, which no stock driver does |

## Why libSQL and not PostgreSQL

Gauntlet's SQL is the SQLite family. The libSQL protocol carries that dialect, so tools built for it send SQL Gauntlet can run. Promising PostgreSQL compatibility would mean PostgreSQL types, `pg_catalog` and a second SQL dialect; that is not planned for v1. PostgreSQL wire stays a transport.

## Until then

- Use a PostgreSQL driver with hand-written SQL: [PostgreSQL wire](https://gauntlet-db.stavros.xyz/docs/clients/pgwire.md).
- From an edge runtime that has only `fetch`, call the [HTTP API](https://gauntlet-db.stavros.xyz/docs/clients/http.md) directly.

---
Source: https://gauntlet-db.stavros.xyz/docs/clients/libsql

# Deploy and qualify a bucket

> Status: Available. What a deployment is, what the bucket must guarantee, and how nodes are secured.

A deployment is one bucket prefix and the `gauntlet serve` nodes that point at it. The supported shape in pre-beta is `gauntlet serve` on Linux x86_64 (glibc 2.34 or newer), from a systemd unit or the container image.

One rule comes first: **the bucket is the database.** Never delete, expire or edit objects under the deployment's prefix, and never set a lifecycle rule on it.

## Qualify the bucket

Gauntlet needs atomic create-if-absent (`If-None-Match: *`), read-after-write consistency and a strongly consistent, ordered LIST. An S3-compatible endpoint alone is not enough.

```sh
(set -a; . ./secrets.env; gauntlet qualify --store 's3:#qualify-check')
```

`gauntlet qualify` exits 0 on a pass, 2 when a check fails and 3 when the result is inconclusive. Every node start runs the same checks. Never run `qualify` against a live prefix.

| Provider | Status |
|---|---|
| Tigris | Run by the project |
| Cloudflare R2 | Run by the project. Slower conditional writes in those runs |
| AWS S3 | Unverified. No recorded run |
| Google Cloud Storage | Unverified. Qualification is planned |
| MinIO | Unverified. No recorded run |

One strongly consistent, **single-region** bucket is the only supported shape. A multi-region or eventually consistent bucket is not safe as the only arbiter.

## Bucket settings

| Variable | Default | Meaning |
|---|---|---|
| `GAUNTLET_S3_ENDPOINT` | required | The service URL |
| `GAUNTLET_S3_BUCKET` | required | The bucket name |
| `GAUNTLET_S3_ACCESS_KEY_ID`, `GAUNTLET_S3_SECRET_ACCESS_KEY` | required | A key that can read and write this bucket only |
| `GAUNTLET_S3_REGION` | `auto` | The signing region |
| `GAUNTLET_S3_PATH_STYLE` | `true` | Set `false` for virtual-hosted bucket addressing |
| `GAUNTLET_S3_TIMEOUT_MS` | `10000` | Deadline for one control request |

## Security

- **TLS on every listener.** A listener that is not on loopback must use TLS, or the node refuses to start. The deployment has its own private CA; each node's certificate is both its server certificate and its client certificate to the other nodes.
- **SCRAM-SHA-256** for PostgreSQL clients. One token per database; a separate admin token.
- **Tokens are at least 32 characters** and never go in the config file. Each can come from a file named by `GAUNTLET_<NAME>_FILE`.
- **No SQL roles** or table privileges.
- **The bucket is trusted.** Objects are not encrypted or signed by Gauntlet.

A node refuses to start, before it binds or writes anything, if TLS is half configured, the key file is readable by others, the certificate does not cover the advertised name, a token is missing or short, or the bucket fails qualification.

## Health endpoints

| Endpoint | Auth | Meaning |
|---|---|---|
| `GET /health` | none | Liveness. Answers `ok` while the process runs |
| `GET /ready` | none | Readiness. 200 when the node can serve; 503 when it cannot |
| `GET /db/<name>/ready` | database token | That database's readiness. Does not wake an idle database |

Point a load balancer's health check at `GET /ready`.

## Configuration

The config file's keys are the flag names of `gauntlet serve --help`. A flag on the command line wins over an option's environment variable, which wins over the file. A misspelt key, a value of the wrong type, or a key that looks like a secret stops the start.

## Coming

Managed Gauntlet Cloud and packaged bring-your-own-cloud installs are designed and not built. Qualification of AWS S3 and Google Cloud Storage is planned.

---
Source: https://gauntlet-db.stavros.xyz/docs/operations/deploy

# Backups and restore

> Status: Available. A backup is one complete, consistent image of one database. A restore creates a new database.

| | |
|---|---|
| **Works** | Whole-database backup while writes continue. Image verification. Scheduled backups with retention and restore drills. Restore into a new database. |
| **Differs** | A restore never overwrites the original. The restored database has a new name, a new ID and a new token. |
| **Missing** | Point-in-time restore. Incremental backups. Backups that skip idle databases. |
| **Operational impact** | A scheduled backup activates each database and copies it in full. |
| **Evidence** | Covered by the project's backup tests. How large a database fits in one backup window has not been measured. |

Put backups in a second bucket, ideally at a second provider. Losing the bucket loses everything since the last finished backup.

## Take a backup

```sh
image="s3:$PWD/backup.env#images/app/$(date -u +%Y%m%dT%H%M%SZ)"
gauntlet backup --source https://n1.example.net:7101/db/app --store "$image"
gauntlet verify-image --store "$image"
```

- Use a new prefix for every backup. A non-empty destination is refused.
- The database keeps serving writes while the backup runs.
- Keep the printed receipt.
- A backup that cannot finish copying in time fails cleanly. It never leaves a partial image marked ready.

## Schedule backups

`gauntlet backups` runs from cron on a backup host. It backs up every database on an interval, applies retention, and runs restore drills.

```sh
*/15 * * * *  gauntlet backups run   --config /etc/gauntlet/backups.json
30 3 * * *    gauntlet backups drill --config /etc/gauntlet/backups.json --cache-dir /srv/gauntlet-cache/drill
0 * * * *     gauntlet backups status --config /etc/gauntlet/backups.json >/dev/null
```

The defaults are a daily backup, keeping 7 images and 30 days.

## Restore

A restore creates a **new database with a new name, a new ID and a new token**.

1. Verify the image: `gauntlet verify-image --store "$image"`.
2. Make a working copy of the image if the original must stay an archive. Serving an image turns it into a live cluster.
3. Serve the copy on loopback with `gauntlet node`, wait for `/ready`, then run `gauntlet restore-database --store 's3:#gauntlet/beta' --name app-restored --source http://127.0.0.1:7201`.
4. Get the new token: `gauntlet admin token app-restored`.
5. Check the data, then point clients at the new name and token.

**Do a restore drill before you need one.** An image you have never restored is not yet a backup.

## Coming

- **Restore to a transaction ID** (P10.17): exact, one cut across shards, into a new identity.
- **Backups that skip unchanged databases** without waking them (P10.50).

See [branches and point-in-time recovery](https://gauntlet-db.stavros.xyz/docs/concepts/branches-and-recovery.md).

---
Source: https://gauntlet-db.stavros.xyz/docs/operations/backups

# Node failure and recovery

> Status: Available. Nodes are disposable. This page says what each failure does and what clients see.

| Event | What happens | What you do |
|---|---|---|
| Planned stop (SIGTERM) | The node stops taking new work, lets admitted requests finish for up to 25 s, and releases its databases. Other nodes take them over | Nothing |
| Kill or host loss | The other nodes take over its databases through normal recovery. Every acknowledged write is in the bucket | Send clients to the surviving nodes. Restart the node when you can |
| All nodes lost | Nothing is lost | Start nodes against the same bucket and prefix |
| Bucket unreachable | No node can commit. `/ready` turns 503; `/health` stays up | Restore bucket access. More nodes do not help |
| Local cache disk lost | Nothing is lost. The cache is rebuilt from the bucket | Restart the node |

## What clients see when a node dies

- Connections to that node break. Reconnect to any node.
- A commit that was in flight may return `40003`, or no answer at all. Its outcome is unknown. [Look it up before retrying](https://gauntlet-db.stavros.xyz/docs/concepts/transactions.md#40003-the-commit-outcome-is-unknown).
- Other transactions may get `40001` or `57P03` while ownership moves. Retry them.

## Restarting a node

- A restart with the same `id` is not ready for 17–18 s. The old process's 15 s lease in the bucket has to expire first.
- Every start also checks the bucket. Transient bucket errors are retried for up to 315 s before the process exits.
- The packaged systemd unit restarts a failed process after 5 s.

Do not "help" recovery. Never delete leases, journals, decisions or checkpoints to make a node ready.

## No failover time is guaranteed

The project's fault tests allow 90 s from a kill until data can be read again, as a test ceiling, not a promise. Recovery time depends on database size and bucket latency. Measure it on your own deployment: kill a node under load and time it.

## Read leases and clocks

`serve` has a `read-leases` option: `on` (the default), `off` and `lease-free`. With `on`, a node can answer read-only transactions from memory for up to 2 seconds without asking the bucket. That is safe only if the host's clock runs at a steady rate: no suspend, no VM pause or live migration that freezes the guest clock.

If you have not qualified your hosts, set `read-leases = "lease-free"` on every node **before the first start**. A clock fault under `on` can break strict serializability for read-only transactions. Writes and durability are not affected.

## Coming

A v1 target of failover in 10 s or less at the median (P5.11), warm-follower takeover (P10.58), query buffering during handoffs (P10.36) and zone-local gateways (P10.59).

---
Source: https://gauntlet-db.stavros.xyz/docs/operations/failover

# Metrics, memory and connections

> Status: Available. What to scrape, how the memory budget works and how to size connections.

## Metrics

Every node serves Prometheus metrics at `GET /metrics` with the admin token. `GET /status` (`gauntlet admin status`) has the same and more as JSON. All metric names start with `gauntlet_`.

| Watch | Metric | Why |
|---|---|---|
| Is it serving | `ready` | 0 means the node is not ready |
| Which build | `build_info{version, revision}` | Must be identical on every node |
| Refused connections | `pg_rejected_connections_total{outcome}` | Above zero: the connection limit was hit |
| Memory pressure | `memory_bytes{pool}` against `memory_limit_bytes{pool}` | A pool near its limit comes before `57P03` |
| Conflicts and unknown outcomes | `sql_terminal_errors_total{code}` | `40003` should be rare; each one needs a lookup |
| Bucket health | `bucket_errors_total`, `bucket_throttled_total`, `bucket_request_duration_seconds{op}` | Commit latency follows the bucket |
| Stalled writes | `shards_durable_io_stalled` | Above zero: a shard cannot write to the bucket |
| Ownership | `shards{role}`, `ownership_changes_total{change}` | Long-lived `recovering` shards need attention |
| Checkpoints | `checkpoint_lag_records` | Steady growth means checkpoints are not keeping up |
| Stuck transactions | `prepared_oldest_age_seconds` | Should stay small |
| Databases | `databases{state}` | How many are known and how many are resident |

- Gauntlet does not export process RSS, bucket size or object count. Collect RSS from the host and bucket figures from your provider.
- **There are no published alert thresholds.** Record a baseline under your normal load and alert on change.
- With `log-format = "json"`, each log line is one JSON object on stderr. Fields that look like secrets are redacted.

## Terms the console uses

| Term | Meaning |
|---|---|
| Window | An absolute start and end. Every figure states its window |
| As of | When the figure was last exported. A figure can be stale while the database is healthy |
| Unavailable | No export exists for the window. It is never shown as zero |
| Runtime share | A query pattern's total time divided by all patterns' total time in the window |
| Read amplification | Rows read divided by rows returned. A high value usually means a missing index |

Query Insights and per-database usage export are planned; today these figures come from node metrics.

## Memory budget

One setting controls memory: `memory-mib`. Every database the process hosts shares it.

| Case | Budget used |
|---|---|
| You set `memory-mib` | That value. Minimum 4096 |
| Not set, no container memory limit | 16384 MiB |
| Not set, smaller container limit | The container limit minus 1024 MiB |
| Not set, and that leaves less than 4096 MiB | The node refuses to start |

**The budget is not a cap on the process's memory (RSS).** It limits what Gauntlet reserves for its own work. On top of it, plan for the mapped block cache (`cache-mib`, default 8192), SQL buffers and the operating system.

When the budget is exhausted the node does not crash. It returns `57P03` (HTTP 503) and releases idle databases.

## Connections

Each node accepts **128 PostgreSQL connections** by default. Set `max-connections` (1 to 10000) to change it; use the same value on every node.

- Starting and idle connections count. A connection with no traffic for 300 s is closed.
- One connection past the limit gets `FATAL 53300`.
- HTTP requests are not counted. They have their own queue; when it is full the answer is HTTP 503.
- Prefer a connection pool to a very large limit.

---
Source: https://gauntlet-db.stavros.xyz/docs/operations/observability

# Upgrades

> Status: Available. Beta builds carry no upgrade promise. An upgrade is a coordinated stop, or a new deployment and a restore.

Two rules:

1. **Never run two builds against one bucket prefix.** There is no rolling upgrade.
2. **Read the new build's notes first.** They say whether the storage format changed.

## Same storage format: stop everything, then start everything

1. Back up every database. Keep the old binary and its build record.
2. Stop every node. Check that no `gauntlet` process of the old build is left on any host, including `backup` and `restore-database` runs.
3. Install the new binary on every node, after `sha256sum --check`.
4. Start every node. On each, check `gauntlet probe ready` and that `gauntlet admin status` shows the new build.

This costs one outage, from the last old process stopping to the new ones being ready. To roll back, stop every node and start the old build.

## Changed storage format: a new deployment

The new build runs as a separate deployment on a **new bucket prefix**, and each database moves by backup and restore. The old deployment stays untouched until you have verified the copy.

## From the beta to v1

There will be no in-place upgrade. Data moves by backup and restore, or by export.

## Coming

Rolling upgrades are deferred and not near-term. A dump-and-load export path (P10.6) is a v1 item and is also the beta-to-v1 route.

---
Source: https://gauntlet-db.stavros.xyz/docs/operations/upgrades

# Limits and known gaps

> Status: Available. What pre-beta Gauntlet does not do. Read this before you rely on it.

## The ones that change how you use it

| Limit | What it means for you |
|---|---|
| No in-place upgrade to v1 | Plan to move data by backup and restore |
| Coordinated stop between beta builds | An upgrade is one outage. Never run two builds on one bucket prefix |
| Storage only grows | Budget the bucket for every write ever made, not for live rows. Dropping a database frees nothing |
| Backups are whole images | No point-in-time restore. Losing the bucket loses everything since the last finished backup |
| Commit latency follows the bucket | Each commit waits for conditional writes to the bucket. Put the bucket next to the nodes |
| One token per database, no roles | Anyone with the token can do everything in that database |
| One shard per new database | One database is limited to what one shard owner can serve |
| The bucket is trusted | Gauntlet does not encrypt or sign objects. Scope the bucket key to this bucket |
| No-wait locking | A conflicting transaction gets `40001` and must retry |
| `CREATE TABLE` only | No `ALTER`, `DROP` or `CREATE INDEX` |
| 128 PostgreSQL connections per node by default | One past the limit gets `FATAL 53300` |
| Memory budget is not an RSS ceiling | Size the host above `memory-mib` |
| Slow restart with the same node ID | A node restarted with the same `id` is not ready for 17–18 s |

## Not supported in pre-beta

| Area | Not available |
|---|---|
| Upgrades | No rolling upgrade and no mixed builds |
| Schema changes | No `ALTER`, `DROP` or `CREATE INDEX`. Indexes come only from `UNIQUE` at `CREATE TABLE` |
| SQL features | No savepoints, `COPY`, `SHOW <setting>`, query cancel, named parameters or schema-qualified names |
| Access control | No SQL roles or table privileges. No read-only or short-lived tokens |
| Storage | Nothing is reclaimed. The bucket only grows, even after a database is dropped |
| Recovery | No point-in-time restore. No branches |
| Scaling | One shard per new database. No shard split or move |
| Regions | One deployment, one single-region bucket. No region moves that keep the ID and token. No multi-region writes |
| Platforms | No macOS or Windows server binary. An aarch64 build exists but has not been run as a deployment |
| Clients | No certified ORM. No libSQL-compatible endpoint. No SDK, MCP server or agent skill |
| Service | No managed cloud, no public download, no SLA |

## Not yet verified

These are stated plainly so you do not assume more than has been shown.

- **The beta acceptance run has not passed.** The 48–72 hour run with node kills and a restore drill exists as a test driver.
- **No performance figure applies to the build you receive.** Latency figures the project has recorded come from earlier commits. Measure your own workload.
- **No failover time is guaranteed.**
- **No maximum database count has been qualified above 1,000.**
- **AWS S3, Google Cloud Storage and MinIO have no recorded run.** Only Tigris and Cloudflare R2 appear in the project's results.
- **node-postgres and pgjdbc over TLS** are documented, but the client suites run them on loopback without TLS.

## Where Gauntlet is behind

Product surface, by years: a table cannot change, an index exists only as `UNIQUE`, keys are not generated, no ORM is certified, nothing deletes, and there are no users or roles. Databases that acknowledge on local disks are faster per write than one that waits for a bucket. The [roadmap](https://gauntlet-db.stavros.xyz/docs/roadmap.md) is the plan to close the surface gap; an optional peer-durable mode is the answer to latency.

---
Source: https://gauntlet-db.stavros.xyz/docs/limits

# Roadmap

> Status: Pre-beta. What is true today, what is planned for v1 and what comes after. Planned work is worth nothing until it is built.

Phases have exit conditions, not dates. Plan item numbers are given so that a claim here can be checked against the project's plan.

## Today

- Strict serializability for every transaction, including across shards.
- Every acknowledged write in the bucket; disposable nodes.
- PostgreSQL wire and an HTTP JSON API.
- Many named databases per deployment; activation on first request and release when idle.
- Whole-image backup, verification, scheduled backups and restore into a new database.
- Tigris and Cloudflare R2 exercised by the project.

## Before the beta

- The beta acceptance run: three hosts, node kills, 48–72 hours of load and a restore drill.
- Storage reclamation, switched on step by step with evidence.

## v1

| Area | Planned | Item |
|---|---|---|
| Schema changes | `ADD COLUMN`, `CREATE INDEX`, `DROP TABLE` | P10.1 |
| Clients | libSQL-compatible endpoint and ORM support | P10.2 |
| Keys and defaults | Generated integer keys; `DEFAULT CURRENT_TIMESTAMP` | P10.3, P10.4 |
| Introspection | A schema surface for migration tools | P10.5 |
| Leaving | Dump and load; one-command import from SQLite-dialect sources | P10.6, P10.6a |
| Client helpers | SDK retry on `40001`, lookup on `40003`; server-side retry of standalone statements | P10.7 |
| Certified list | What works, what is partial, what is refused | P10.8 |
| Vectors | A published contract for exact vector search | P10.9 |
| Tokens | A read-only token; platform-issued short-lived tokens | P10.10, P10.13a |
| Fleet proof | Qualification at 100,000 databases | P10.11 |
| Tenancy | Scopes, paged listing, per-scope limits | P10.12 |
| Failover | 10 s or less at the median, kill to first SQL | P5.11 |

## After v1

| Area | Planned | Item |
|---|---|---|
| Branches | Fork by copy, then copy-on-write | P10.16, P6.8 |
| Recovery | Restore to a transaction ID | P10.17 |
| Insights | Query fingerprints, runtime share, read amplification | P10.23 |
| Fleet | Labels for agent, tenant and environment; listing that never activates | P10.53 |
| Change stream | One resumable cursor in commit order | P10.14 |
| Deploy requests | Diff, lint, gated apply, revert | P10.37 |
| Moves | One procedure for every data move; a home region per database | P10.41, P10.30 |
| Search | Full-text and approximate vector indexes | P10.18, P10.19 |
| Fleet proof | Qualification at one million databases | P10.25 |
| MCP | A server per database with a credential per query | P10.40 |
| Latency | Optional peer-durable mode; shared commit objects | P5.9, P10.55 |

## Gauntlet Cloud

A managed service is designed and not built. The plan is an invited alpha on dedicated deployments in one EU region, then a private beta with billing, then open sign-up. Bring-your-own-cloud packaging follows. No prices are published; see [pricing](https://gauntlet-db.stavros.xyz/pricing) for the model.

## Not planned

- An in-process library with offline sync. A write made offline cannot be strictly serializable against the bucket.
- PostgreSQL SQL in v1.
- Embedding generation inside the server.

---
Source: https://gauntlet-db.stavros.xyz/docs/roadmap

# Licence

> Status: Available. The server is source-available under FSL-1.1-ALv2 and converts to Apache 2.0 two years after each release. Clients are Apache-2.0.

| Component | Licence |
|---|---|
| Server and engine | **FSL-1.1-ALv2**, the Functional Source License 1.1 with an Apache 2.0 future licence. Licensor: ComputerEyes |
| Client SDKs, drivers, the MCP server and examples | **Apache-2.0**. These are planned; none is published yet |

## What FSL-1.1-ALv2 means

- The source is available. You may use, copy, modify and redistribute it for any purpose **other than a competing use**, as the licence defines it.
- **Each release converts to Apache 2.0 two years after it is published.**
- It is a source-available licence. It is not an OSI-approved open-source licence until that conversion, and this site does not call the server open source.

Read the licence text itself before you rely on this summary. The text governs.

## Bring your own cloud

When packaged bring-your-own-cloud installs exist, they will sell updates and support. A deployment keeps serving if its subscription lapses; only updates and support stop.

## Status

The licence decision is made. Pre-beta builds are distributed to invited testers; a public source release has not happened yet.

---
Source: https://gauntlet-db.stavros.xyz/docs/licence