Docs / Operations
Metrics, memory and connections
What to scrape, how the memory budget works and how to size connections.
Available View Markdown
# Metrics, memory and connections
> Status: Available. What to scrape, how the memory budget works and how to size connections.
## Metrics
Every node serves Prometheus metrics at `GET /metrics` with the admin token. `GET /status` (`gauntlet admin status`) has the same and more as JSON. All metric names start with `gauntlet_`.
| Watch | Metric | Why |
|---|---|---|
| Is it serving | `ready` | 0 means the node is not ready |
| Which build | `build_info{version, revision}` | Must be identical on every node |
| Refused connections | `pg_rejected_connections_total{outcome}` | Above zero: the connection limit was hit |
| Memory pressure | `memory_bytes{pool}` against `memory_limit_bytes{pool}` | A pool near its limit comes before `57P03` |
| Conflicts and unknown outcomes | `sql_terminal_errors_total{code}` | `40003` should be rare; each one needs a lookup |
| Bucket health | `bucket_errors_total`, `bucket_throttled_total`, `bucket_request_duration_seconds{op}` | Commit latency follows the bucket |
| Stalled writes | `shards_durable_io_stalled` | Above zero: a shard cannot write to the bucket |
| Ownership | `shards{role}`, `ownership_changes_total{change}` | Long-lived `recovering` shards need attention |
| Checkpoints | `checkpoint_lag_records` | Steady growth means checkpoints are not keeping up |
| Stuck transactions | `prepared_oldest_age_seconds` | Should stay small |
| Databases | `databases{state}` | How many are known and how many are resident |
- Gauntlet does not export process RSS, bucket size or object count. Collect RSS from the host and bucket figures from your provider.
- **There are no published alert thresholds.** Record a baseline under your normal load and alert on change.
- With `log-format = "json"`, each log line is one JSON object on stderr. Fields that look like secrets are redacted.
## Terms the console uses
| Term | Meaning |
|---|---|
| Window | An absolute start and end. Every figure states its window |
| As of | When the figure was last exported. A figure can be stale while the database is healthy |
| Unavailable | No export exists for the window. It is never shown as zero |
| Runtime share | A query pattern's total time divided by all patterns' total time in the window |
| Read amplification | Rows read divided by rows returned. A high value usually means a missing index |
Query Insights and per-database usage export are planned; today these figures come from node metrics.
## Memory budget
One setting controls memory: `memory-mib`. Every database the process hosts shares it.
| Case | Budget used |
|---|---|
| You set `memory-mib` | That value. Minimum 4096 |
| Not set, no container memory limit | 16384 MiB |
| Not set, smaller container limit | The container limit minus 1024 MiB |
| Not set, and that leaves less than 4096 MiB | The node refuses to start |
**The budget is not a cap on the process's memory (RSS).** It limits what Gauntlet reserves for its own work. On top of it, plan for the mapped block cache (`cache-mib`, default 8192), SQL buffers and the operating system.
When the budget is exhausted the node does not crash. It returns `57P03` (HTTP 503) and releases idle databases.
## Connections
Each node accepts **128 PostgreSQL connections** by default. Set `max-connections` (1 to 10000) to change it; use the same value on every node.
- Starting and idle connections count. A connection with no traffic for 300 s is closed.
- One connection past the limit gets `FATAL 53300`.
- HTTP requests are not counted. They have their own queue; when it is full the answer is HTTP 503.
- Prefer a connection pool to a very large limit.
---
Source: https://gauntlet-db.stavros.xyz/docs/operations/observability
Connect this project to my existing Gauntlet database.
Context:
- Documentation index for agents: https://gauntlet-db.stavros.xyz/llms.txt
- Quickstart: https://gauntlet-db.stavros.xyz/docs/quickstart.md
- SQL compatibility and error codes: https://gauntlet-db.stavros.xyz/docs/sql/compatibility.md
- Database: the one already configured for this project
- Protocol: PostgreSQL wire or the native HTTP JSON API, whichever this project already uses
- Connection values are in environment variables: GAUNTLET_DATABASE_URL (PostgreSQL wire),
GAUNTLET_HTTP_URL (HTTP base URL) and GAUNTLET_DATABASE_TOKEN (the database token).
These names are a convention, not a shipped SDK setting. Keep this project's own names if it has them.
Before editing, read the Gauntlet documentation above and inspect how this project
already talks to its database. Gauntlet is pre-beta software; where the documentation
and my deployment disagree, tell me instead of guessing.
What Gauntlet is:
- A distributed SQL database. SQL is the SQLite family, run by the Turso engine.
- The PostgreSQL wire protocol is a transport only. There is no PostgreSQL SQL dialect,
no pg_catalog or information_schema, no extensions, and no ORM is certified yet.
- libSQL client compatibility is not available yet. Do not assume it.
Connecting:
- PostgreSQL wire: dbname is the database name, the user name is ignored, and the
password is the database token. Authentication is SCRAM-SHA-256. Keep TLS
verification on (sslmode=verify-full with the deployment's CA certificate).
- HTTP: POST <GAUNTLET_HTTP_URL>/db/<name>/sql with a JSON body
{"sql": "...", "params": [...]} and the header "Authorization: Bearer <token>".
POST /db/<name>/tx runs a list of statements atomically.
SQL rules that differ from PostgreSQL:
- Every table needs an explicit primary key, and INSERT supplies the key value.
- CREATE TABLE is a standalone statement outside a transaction. There is no ALTER TABLE,
DROP TABLE or CREATE INDEX; an index comes only from a UNIQUE constraint at creation.
- Parameters are $1, ?1 or ?. Names are single-part (no schema.table).
- Column types follow SQLite affinity. In prepared statements ordinary columns are
described as text; CAST(x AS INTEGER) when the driver needs a number.
Errors you must handle (SQLSTATE):
- 40001: conflict. Roll back and retry the whole transaction. Locking is no-wait, so
this is normal under contention.
- 40003: commit outcome unknown. Do NOT retry. Look the transaction up with
GET /db/<name>/txn/<txid> and replay only if the status is `aborted`.
- 57P03: cannot serve right now. Roll back, then retry the whole transaction with backoff.
- 53300: too many connections. Back off; keep the pool below the node's limit.
Credentials and scope:
- Use only the database token for this one database. Keep it server-side and out of
source, logs, browser bundles and this conversation.
- Never use or ask for a deployment admin token, a client token or bucket credentials.
- Do not create, drop, back up, restore or branch databases, and do not rotate credentials.
Task:
1. Add the smallest connection integration that fits this project.
2. Run one explicit `SELECT 1` connectivity check. It may activate this database if it
is idle; do not probe or wake any other database.
3. Do not change the schema or write application data unless I ask for it.
Report the files changed, the protocol and client used, the result of the check, and
anything this project needs that Gauntlet does not support.#Metrics
Every node serves Prometheus metrics at GET /metrics with the admin token. GET /status (gauntlet admin status) has the same and more as JSON. All metric names start with gauntlet_.
| Watch | Metric | Why |
|---|---|---|
| Is it serving | ready |
0 means the node is not ready |
| Which build | build_info{version, revision} |
Must be identical on every node |
| Refused connections | pg_rejected_connections_total{outcome} |
Above zero: the connection limit was hit |
| Memory pressure | memory_bytes{pool} against memory_limit_bytes{pool} |
A pool near its limit comes before 57P03 |
| Conflicts and unknown outcomes | sql_terminal_errors_total{code} |
40003 should be rare; each one needs a lookup |
| Bucket health | bucket_errors_total, bucket_throttled_total, bucket_request_duration_seconds{op} |
Commit latency follows the bucket |
| Stalled writes | shards_durable_io_stalled |
Above zero: a shard cannot write to the bucket |
| Ownership | shards{role}, ownership_changes_total{change} |
Long-lived recovering shards need attention |
| Checkpoints | checkpoint_lag_records |
Steady growth means checkpoints are not keeping up |
| Stuck transactions | prepared_oldest_age_seconds |
Should stay small |
| Databases | databases{state} |
How many are known and how many are resident |
- Gauntlet does not export process RSS, bucket size or object count. Collect RSS from the host and bucket figures from your provider.
- There are no published alert thresholds. Record a baseline under your normal load and alert on change.
- With
log-format = "json", each log line is one JSON object on stderr. Fields that look like secrets are redacted.
#Terms the console uses
| Term | Meaning |
|---|---|
| Window | An absolute start and end. Every figure states its window |
| As of | When the figure was last exported. A figure can be stale while the database is healthy |
| Unavailable | No export exists for the window. It is never shown as zero |
| Runtime share | A query pattern's total time divided by all patterns' total time in the window |
| Read amplification | Rows read divided by rows returned. A high value usually means a missing index |
Query Insights and per-database usage export are planned; today these figures come from node metrics.
#Memory budget
One setting controls memory: memory-mib. Every database the process hosts shares it.
| Case | Budget used |
|---|---|
You set memory-mib |
That value. Minimum 4096 |
| Not set, no container memory limit | 16384 MiB |
| Not set, smaller container limit | The container limit minus 1024 MiB |
| Not set, and that leaves less than 4096 MiB | The node refuses to start |
The budget is not a cap on the process's memory (RSS). It limits what Gauntlet reserves for its own work. On top of it, plan for the mapped block cache (cache-mib, default 8192), SQL buffers and the operating system.
When the budget is exhausted the node does not crash. It returns 57P03 (HTTP 503) and releases idle databases.
#Connections
Each node accepts 128 PostgreSQL connections by default. Set max-connections (1 to 10000) to change it; use the same value on every node.
- Starting and idle connections count. A connection with no traffic for 300 s is closed.
- One connection past the limit gets
FATAL 53300. - HTTP requests are not counted. They have their own queue; when it is full the answer is HTTP 503.
- Prefer a connection pool to a very large limit.