Monitoring

QueryProxy has three ways to tell you it is healthy: a Prometheus endpoint, an example Grafana dashboard, and the queryproxy:doctor command. The endpoint is off until you turn it on.

The /metrics endpoint

GET /metrics returns the Prometheus text format. It answers 404 until one of these is set, and then lets a scrape through when it carries the right bearer token or comes from an allowed address:

Set one or both. Anything else gets 401. The route runs outside the web session: no cookies or CSRF.

Two things decide whether a scrape is accepted besides the token:

• The client address follows the TRUSTED_PROXIES rules. Behind a reverse proxy, the allowlist sees the proxy’s address unless the proxy is listed. Never set TRUSTED_PROXIES=*: any client could then send its own X-Forwarded-For and pass the address allowlist.
• The Host header must be APP_URL’s host or a name in TRUSTED_HOSTS; otherwise the request is refused with 400. Scrape the name users reach.

A Prometheus scrape configuration with a token read from a file:

scrape_configs:
  - job_name: queryproxy
    scheme: https
    metrics_path: /metrics
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/queryproxy-metrics-token
    static_configs:
      - targets: ['queryproxy.example.com']

What it exposes

Counters start at 0 for the known label sets, so rate() has a starting point.

Labels never carry SQL, user names, team names or host names. The rule label of masked values is the rule’s match type, not its name, because rule names are chosen by each team.

Notes on how the counts behave:

• A guard rejection is counted per user action. Validating a statement and then submitting the same statement can count it twice.
• A rejection with several rule families counts once in each family.
• On SQLite every counted event is a database write. The table stays small because labels are bounded.
• /metrics needs the metric_samples table. Run migrations before you start new workers.

Useful queries

# Requests waiting for a decision for a long time
queryproxy_requests_pending > 0

# 95th percentile approval latency over 1 hour
histogram_quantile(0.95, sum by (le) (rate(queryproxy_approval_latency_seconds_bucket[1h])))

# 95th percentile execution time per driver
histogram_quantile(0.95, sum by (le, driver) (rate(queryproxy_execution_duration_seconds_bucket[15m])))

# Failed requests per minute
sum(rate(queryproxy_requests_total{status="failed"}[5m])) * 60

Alert on queryproxy_failed_jobs > 0 and on a queryproxy_queue_depth that keeps growing: both mean the worker is not keeping up or not running.

The Grafana dashboard

docs/grafana-queryproxy.json in the repository is an example dashboard (uid queryproxy-overview). Import it with Grafana’s dashboard import, then pick your Prometheus data source in the Data source variable at the top. It shows pending requests, failed jobs, queue depth, requests by status and kind, approval latency (p50 and p95), execution duration by driver (p95), guard rejections, masked values and requests held for a connection slot. Treat it as a starting point; copy it and edit the copy.

The doctor

queryproxy:doctor checks configuration, database, queue, scheduler, storage, TLS and the audit chain. Run it after every deploy, and in a pipeline with --strict --json. Each check is explained in Troubleshooting.

The doctor can tell whether the scheduler runs because the scheduler writes a heartbeat to the cache every minute. A scheduler that stopped means expired results are not pruned, reminders are not sent and the daily audit checkpoint does not run.

What the scheduler runs

Next steps

• Audit chain and SIEM export
• Troubleshooting
• Configuration