# Node failure and recovery

> Status: Available. Nodes are disposable. This page says what each failure does and what clients see.

| Event | What happens | What you do |
|---|---|---|
| Planned stop (SIGTERM) | The node stops taking new work, lets admitted requests finish for up to 25 s, and releases its databases. Other nodes take them over | Nothing |
| Kill or host loss | The other nodes take over its databases through normal recovery. Every acknowledged write is in the bucket | Send clients to the surviving nodes. Restart the node when you can |
| All nodes lost | Nothing is lost | Start nodes against the same bucket and prefix |
| Bucket unreachable | No node can commit. `/ready` turns 503; `/health` stays up | Restore bucket access. More nodes do not help |
| Local cache disk lost | Nothing is lost. The cache is rebuilt from the bucket | Restart the node |

## What clients see when a node dies

- Connections to that node break. Reconnect to any node.
- A commit that was in flight may return `40003`, or no answer at all. Its outcome is unknown. [Look it up before retrying](https://gauntlet-db.stavros.xyz/docs/concepts/transactions.md#40003-the-commit-outcome-is-unknown).
- Other transactions may get `40001` or `57P03` while ownership moves. Retry them.

## Restarting a node

- A restart with the same `id` is not ready for 17–18 s. The old process's 15 s lease in the bucket has to expire first.
- Every start also checks the bucket. Transient bucket errors are retried for up to 315 s before the process exits.
- The packaged systemd unit restarts a failed process after 5 s.

Do not "help" recovery. Never delete leases, journals, decisions or checkpoints to make a node ready.

## No failover time is guaranteed

The project's fault tests allow 90 s from a kill until data can be read again, as a test ceiling, not a promise. Recovery time depends on database size and bucket latency. Measure it on your own deployment: kill a node under load and time it.

## Read leases and clocks

`serve` has a `read-leases` option: `on` (the default), `off` and `lease-free`. With `on`, a node can answer read-only transactions from memory for up to 2 seconds without asking the bucket. That is safe only if the host's clock runs at a steady rate: no suspend, no VM pause or live migration that freezes the guest clock.

If you have not qualified your hosts, set `read-leases = "lease-free"` on every node **before the first start**. A clock fault under `on` can break strict serializability for read-only transactions. Writes and durability are not affected.

## Coming

A v1 target of failover in 10 s or less at the median (P5.11), warm-follower takeover (P10.58), query buffering during handoffs (P10.36) and zone-local gateways (P10.59).

---
Source: https://gauntlet-db.stavros.xyz/docs/operations/failover