# Metrics, memory and connections

> Status: Available. What to scrape, how the memory budget works and how to size connections.

## Metrics

Every node serves Prometheus metrics at `GET /metrics` with the admin token. `GET /status` (`gauntlet admin status`) has the same and more as JSON. All metric names start with `gauntlet_`.

| Watch | Metric | Why |
|---|---|---|
| Is it serving | `ready` | 0 means the node is not ready |
| Which build | `build_info{version, revision}` | Must be identical on every node |
| Refused connections | `pg_rejected_connections_total{outcome}` | Above zero: the connection limit was hit |
| Memory pressure | `memory_bytes{pool}` against `memory_limit_bytes{pool}` | A pool near its limit comes before `57P03` |
| Conflicts and unknown outcomes | `sql_terminal_errors_total{code}` | `40003` should be rare; each one needs a lookup |
| Bucket health | `bucket_errors_total`, `bucket_throttled_total`, `bucket_request_duration_seconds{op}` | Commit latency follows the bucket |
| Stalled writes | `shards_durable_io_stalled` | Above zero: a shard cannot write to the bucket |
| Ownership | `shards{role}`, `ownership_changes_total{change}` | Long-lived `recovering` shards need attention |
| Checkpoints | `checkpoint_lag_records` | Steady growth means checkpoints are not keeping up |
| Stuck transactions | `prepared_oldest_age_seconds` | Should stay small |
| Databases | `databases{state}` | How many are known and how many are resident |

- Gauntlet does not export process RSS, bucket size or object count. Collect RSS from the host and bucket figures from your provider.
- **There are no published alert thresholds.** Record a baseline under your normal load and alert on change.
- With `log-format = "json"`, each log line is one JSON object on stderr. Fields that look like secrets are redacted.

## Terms the console uses

| Term | Meaning |
|---|---|
| Window | An absolute start and end. Every figure states its window |
| As of | When the figure was last exported. A figure can be stale while the database is healthy |
| Unavailable | No export exists for the window. It is never shown as zero |
| Runtime share | A query pattern's total time divided by all patterns' total time in the window |
| Read amplification | Rows read divided by rows returned. A high value usually means a missing index |

Query Insights and per-database usage export are planned; today these figures come from node metrics.

## Memory budget

One setting controls memory: `memory-mib`. Every database the process hosts shares it.

| Case | Budget used |
|---|---|
| You set `memory-mib` | That value. Minimum 4096 |
| Not set, no container memory limit | 16384 MiB |
| Not set, smaller container limit | The container limit minus 1024 MiB |
| Not set, and that leaves less than 4096 MiB | The node refuses to start |

**The budget is not a cap on the process's memory (RSS).** It limits what Gauntlet reserves for its own work. On top of it, plan for the mapped block cache (`cache-mib`, default 8192), SQL buffers and the operating system.

When the budget is exhausted the node does not crash. It returns `57P03` (HTTP 503) and releases idle databases.

## Connections

Each node accepts **128 PostgreSQL connections** by default. Set `max-connections` (1 to 10000) to change it; use the same value on every node.

- Starting and idle connections count. A connection with no traffic for 300 s is closed.
- One connection past the limit gets `FATAL 53300`.
- HTTP requests are not counted. They have their own queue; when it is full the answer is HTTP 503.
- Prefer a connection pool to a very large limit.

---
Source: https://gauntlet-db.stavros.xyz/docs/operations/observability