Docs / Operations
Node failure and recovery
Nodes are disposable. This page says what each failure does and what clients see.
Available View Markdown
# Node failure and recovery
> Status: Available. Nodes are disposable. This page says what each failure does and what clients see.
| Event | What happens | What you do |
|---|---|---|
| Planned stop (SIGTERM) | The node stops taking new work, lets admitted requests finish for up to 25 s, and releases its databases. Other nodes take them over | Nothing |
| Kill or host loss | The other nodes take over its databases through normal recovery. Every acknowledged write is in the bucket | Send clients to the surviving nodes. Restart the node when you can |
| All nodes lost | Nothing is lost | Start nodes against the same bucket and prefix |
| Bucket unreachable | No node can commit. `/ready` turns 503; `/health` stays up | Restore bucket access. More nodes do not help |
| Local cache disk lost | Nothing is lost. The cache is rebuilt from the bucket | Restart the node |
## What clients see when a node dies
- Connections to that node break. Reconnect to any node.
- A commit that was in flight may return `40003`, or no answer at all. Its outcome is unknown. [Look it up before retrying](https://gauntlet-db.stavros.xyz/docs/concepts/transactions.md#40003-the-commit-outcome-is-unknown).
- Other transactions may get `40001` or `57P03` while ownership moves. Retry them.
## Restarting a node
- A restart with the same `id` is not ready for 17–18 s. The old process's 15 s lease in the bucket has to expire first.
- Every start also checks the bucket. Transient bucket errors are retried for up to 315 s before the process exits.
- The packaged systemd unit restarts a failed process after 5 s.
Do not "help" recovery. Never delete leases, journals, decisions or checkpoints to make a node ready.
## No failover time is guaranteed
The project's fault tests allow 90 s from a kill until data can be read again, as a test ceiling, not a promise. Recovery time depends on database size and bucket latency. Measure it on your own deployment: kill a node under load and time it.
## Read leases and clocks
`serve` has a `read-leases` option: `on` (the default), `off` and `lease-free`. With `on`, a node can answer read-only transactions from memory for up to 2 seconds without asking the bucket. That is safe only if the host's clock runs at a steady rate: no suspend, no VM pause or live migration that freezes the guest clock.
If you have not qualified your hosts, set `read-leases = "lease-free"` on every node **before the first start**. A clock fault under `on` can break strict serializability for read-only transactions. Writes and durability are not affected.
## Coming
A v1 target of failover in 10 s or less at the median (P5.11), warm-follower takeover (P10.58), query buffering during handoffs (P10.36) and zone-local gateways (P10.59).
---
Source: https://gauntlet-db.stavros.xyz/docs/operations/failover
Connect this project to my existing Gauntlet database.
Context:
- Documentation index for agents: https://gauntlet-db.stavros.xyz/llms.txt
- Quickstart: https://gauntlet-db.stavros.xyz/docs/quickstart.md
- SQL compatibility and error codes: https://gauntlet-db.stavros.xyz/docs/sql/compatibility.md
- Database: the one already configured for this project
- Protocol: PostgreSQL wire or the native HTTP JSON API, whichever this project already uses
- Connection values are in environment variables: GAUNTLET_DATABASE_URL (PostgreSQL wire),
GAUNTLET_HTTP_URL (HTTP base URL) and GAUNTLET_DATABASE_TOKEN (the database token).
These names are a convention, not a shipped SDK setting. Keep this project's own names if it has them.
Before editing, read the Gauntlet documentation above and inspect how this project
already talks to its database. Gauntlet is pre-beta software; where the documentation
and my deployment disagree, tell me instead of guessing.
What Gauntlet is:
- A distributed SQL database. SQL is the SQLite family, run by the Turso engine.
- The PostgreSQL wire protocol is a transport only. There is no PostgreSQL SQL dialect,
no pg_catalog or information_schema, no extensions, and no ORM is certified yet.
- libSQL client compatibility is not available yet. Do not assume it.
Connecting:
- PostgreSQL wire: dbname is the database name, the user name is ignored, and the
password is the database token. Authentication is SCRAM-SHA-256. Keep TLS
verification on (sslmode=verify-full with the deployment's CA certificate).
- HTTP: POST <GAUNTLET_HTTP_URL>/db/<name>/sql with a JSON body
{"sql": "...", "params": [...]} and the header "Authorization: Bearer <token>".
POST /db/<name>/tx runs a list of statements atomically.
SQL rules that differ from PostgreSQL:
- Every table needs an explicit primary key, and INSERT supplies the key value.
- CREATE TABLE is a standalone statement outside a transaction. There is no ALTER TABLE,
DROP TABLE or CREATE INDEX; an index comes only from a UNIQUE constraint at creation.
- Parameters are $1, ?1 or ?. Names are single-part (no schema.table).
- Column types follow SQLite affinity. In prepared statements ordinary columns are
described as text; CAST(x AS INTEGER) when the driver needs a number.
Errors you must handle (SQLSTATE):
- 40001: conflict. Roll back and retry the whole transaction. Locking is no-wait, so
this is normal under contention.
- 40003: commit outcome unknown. Do NOT retry. Look the transaction up with
GET /db/<name>/txn/<txid> and replay only if the status is `aborted`.
- 57P03: cannot serve right now. Roll back, then retry the whole transaction with backoff.
- 53300: too many connections. Back off; keep the pool below the node's limit.
Credentials and scope:
- Use only the database token for this one database. Keep it server-side and out of
source, logs, browser bundles and this conversation.
- Never use or ask for a deployment admin token, a client token or bucket credentials.
- Do not create, drop, back up, restore or branch databases, and do not rotate credentials.
Task:
1. Add the smallest connection integration that fits this project.
2. Run one explicit `SELECT 1` connectivity check. It may activate this database if it
is idle; do not probe or wake any other database.
3. Do not change the schema or write application data unless I ask for it.
Report the files changed, the protocol and client used, the result of the check, and
anything this project needs that Gauntlet does not support.| Event | What happens | What you do |
|---|---|---|
| Planned stop (SIGTERM) | The node stops taking new work, lets admitted requests finish for up to 25 s, and releases its databases. Other nodes take them over | Nothing |
| Kill or host loss | The other nodes take over its databases through normal recovery. Every acknowledged write is in the bucket | Send clients to the surviving nodes. Restart the node when you can |
| All nodes lost | Nothing is lost | Start nodes against the same bucket and prefix |
| Bucket unreachable | No node can commit. /ready turns 503; /health stays up |
Restore bucket access. More nodes do not help |
| Local cache disk lost | Nothing is lost. The cache is rebuilt from the bucket | Restart the node |
#What clients see when a node dies
- Connections to that node break. Reconnect to any node.
- A commit that was in flight may return
40003, or no answer at all. Its outcome is unknown. Look it up before retrying. - Other transactions may get
40001or57P03while ownership moves. Retry them.
#Restarting a node
- A restart with the same
idis not ready for 17–18 s. The old process's 15 s lease in the bucket has to expire first. - Every start also checks the bucket. Transient bucket errors are retried for up to 315 s before the process exits.
- The packaged systemd unit restarts a failed process after 5 s.
Do not "help" recovery. Never delete leases, journals, decisions or checkpoints to make a node ready.
#No failover time is guaranteed
The project's fault tests allow 90 s from a kill until data can be read again, as a test ceiling, not a promise. Recovery time depends on database size and bucket latency. Measure it on your own deployment: kill a node under load and time it.
#Read leases and clocks
serve has a read-leases option: on (the default), off and lease-free. With on, a node can answer read-only transactions from memory for up to 2 seconds without asking the bucket. That is safe only if the host's clock runs at a steady rate: no suspend, no VM pause or live migration that freezes the guest clock.
If you have not qualified your hosts, set read-leases = "lease-free" on every node before the first start. A clock fault under on can break strict serializability for read-only transactions. Writes and durability are not affected.
#Coming
A v1 target of failover in 10 s or less at the median (P5.11), warm-follower takeover (P10.58), query buffering during handoffs (P10.36) and zone-local gateways (P10.59).