# Monitoring

> Scrape QueryProxy with Prometheus, import the example Grafana dashboard, and use queryproxy:doctor and the scheduler to see problems before your users do.

- Documentation: Guides
- Updated: 2026-10-25
- Source: https://queryproxy.com/docs/monitoring/
- Language: en-US
- Author: Muhammet Şafak

---
QueryProxy has three ways to tell you it is healthy: a Prometheus endpoint, an
example Grafana dashboard, and the `queryproxy:doctor` command. The endpoint is
off until you turn it on.

## The `/metrics` endpoint

`GET /metrics` returns the Prometheus text format. It answers `404` until one of
these is set, and then lets a scrape through when it carries the right bearer
token **or** comes from an allowed address:

| Variable | Meaning |
| :-- | :-- |
| `QUERYPROXY_METRICS_TOKEN` | Bearer token for scrapes. Use 24 or more random characters: `openssl rand -hex 24`. `QUERYPROXY_METRICS_TOKEN_FILE` reads it from a file |
| `QUERYPROXY_METRICS_ALLOWED_IPS` | Comma-separated addresses or CIDR ranges that may scrape without a token |

Set one or both. Anything else gets `401`. The route runs outside the web
session: no cookies or CSRF.

Two things decide whether a scrape is accepted besides the token:

- **The client address** follows the `TRUSTED_PROXIES` rules. Behind a reverse
  proxy, the allowlist sees the proxy's address unless the proxy is listed. Never
  set `TRUSTED_PROXIES=*`: any client could then send its own `X-Forwarded-For`
  and pass the address allowlist.
- **The `Host` header** must be `APP_URL`'s host or a name in `TRUSTED_HOSTS`;
  otherwise the request is refused with `400`. Scrape the name users reach.

A Prometheus scrape configuration with a token read from a file:

```yaml
scrape_configs:
  - job_name: queryproxy
    scheme: https
    metrics_path: /metrics
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/queryproxy-metrics-token
    static_configs:
      - targets: ['queryproxy.example.com']
```

### What it exposes

Counters start at 0 for the known label sets, so `rate()` has a starting point.

| Metric | Type | Labels | Meaning |
| :-- | :-- | :-- | :-- |
| `queryproxy_requests_total` | counter | `kind` (`read`, `write`), `status` | Requests, counted each time one enters a status. `queued` is not counted. Stop and cancel both count as `cancelled` |
| `queryproxy_requests_pending` | gauge | | Requests waiting for a decision |
| `queryproxy_approval_latency_seconds` | histogram | | Seconds from submission to an approve or reject decision |
| `queryproxy_execution_duration_seconds` | histogram | `driver` | Seconds a request ran on its target database |
| `queryproxy_guard_rejections_total` | counter | `rule` | Statements the SQL guard refused, by rule family: `where_clause`, `privilege_management`, `server_control`, `file_io`, `dynamic_sql`, `transaction_control`, `session_setting`, `unreadable_sql`, `other` |
| `queryproxy_masked_values_total` | counter | `rule` | Result values replaced by a masking rule, by rule type: `column`, `regex`, `validator` |
| `queryproxy_slot_deferred_requests_total` | counter | | Requests held back because every connection slot was busy |
| `queryproxy_queue_depth` | gauge | `queue` | Jobs waiting on a queue |
| `queryproxy_failed_jobs` | gauge | | Rows in the failed jobs table |

Labels never carry SQL, user names, team names or host names. The `rule` label
of masked values is the rule's match type, not its name, because rule names are
chosen by each team.

Notes on how the counts behave:

- A guard rejection is counted per user action. Validating a statement and then
  submitting the same statement can count it twice.
- A rejection with several rule families counts once in each family.
- On SQLite every counted event is a database write. The table stays small
  because labels are bounded.
- `/metrics` needs the `metric_samples` table. Run migrations before you start
  new workers.

### Useful queries

```text
# Requests waiting for a decision for a long time
queryproxy_requests_pending > 0

# 95th percentile approval latency over 1 hour
histogram_quantile(0.95, sum by (le) (rate(queryproxy_approval_latency_seconds_bucket[1h])))

# 95th percentile execution time per driver
histogram_quantile(0.95, sum by (le, driver) (rate(queryproxy_execution_duration_seconds_bucket[15m])))

# Failed requests per minute
sum(rate(queryproxy_requests_total{status="failed"}[5m])) * 60
```

Alert on `queryproxy_failed_jobs > 0` and on a `queryproxy_queue_depth` that
keeps growing: both mean the worker is not keeping up or not running.

## The Grafana dashboard

[`docs/grafana-queryproxy.json`](https://github.com/QueryProxy/QueryProxy/blob/main/docs/grafana-queryproxy.json)
in the repository is an example dashboard (uid `queryproxy-overview`). Import it
with Grafana's dashboard import, then pick your Prometheus data source in the
**Data source** variable at the top. It shows pending requests, failed jobs,
queue depth, requests by status and kind, approval latency (p50 and p95),
execution duration by driver (p95), guard rejections, masked values and
requests held for a connection slot. Treat it as a starting point; copy it and
edit the copy.

## The doctor

`queryproxy:doctor` checks configuration, database, queue, scheduler, storage,
TLS and the audit chain. Run it after every deploy, and in a pipeline with
`--strict --json`. Each check is explained in
[Troubleshooting](/docs/troubleshooting/#reading-the-doctor).

The doctor can tell whether the **scheduler** runs because the scheduler writes
a heartbeat to the cache every minute. A scheduler that stopped means expired
results are not pruned, reminders are not sent and the daily audit checkpoint
does not run.

## What the scheduler runs

| Job | Schedule |
| :-- | :-- |
| `results:prune` | Daily |
| `queue:prune-failed` | Weekly |
| `audit:checkpoint` | Daily |
| `requests:remind` | Every 15 minutes |
| Scheduler heartbeat | Every minute |

## Next steps

- [Audit chain and SIEM export](/docs/audit-export/)
- [Troubleshooting](/docs/troubleshooting/)
- [Configuration](/docs/configuration/)