# Troubleshooting

> Start with queryproxy:doctor, read each check, and fix the errors operators meet most, from stuck requests and TLS failures to chat buttons and secrets.

- Documentation: Guides
- Updated: 2026-10-25
- Source: https://queryproxy.com/docs/troubleshooting/
- Language: en-US
- Author: Muhammet Şafak

---
Start with the doctor. It reads the installation and tells you what is wrong
before you go digging in logs.

```bash
docker exec queryproxy php artisan queryproxy:doctor
```

Without Docker, run `php artisan queryproxy:doctor` in the application
directory.

## Reading the doctor

The command prints a table of checks with a status each:

| Status | Meaning |
| :-- | :-- |
| `ok` | The check passed |
| `warn` | Something deserves attention; the run still exits with `0` |
| `fail` | A problem that breaks or endangers the install; the run exits with `1` |
| `skip` | The check does not apply (no SIEM sink is turned on, the cache or queue is not the kind the check can read, and so on) |

Options:

- `--strict` makes warnings exit with `1` as well. Use it in a deploy pipeline.
- `--full-audit` walks the whole audit chain. Without it the doctor checks the
  newest records and the chain head, which is faster.
- `--json` prints `{"ok": …, "strict": …, "checks": […]}` for scripts.

The doctor only reads. It does not change anything on the installation.

### What each check means

| Check | If it is not `ok`, do this |
| :-- | :-- |
| `app_environment` | `fail`: `APP_DEBUG` is on with `APP_ENV=production`. `warn`: it is on in another environment. Turn `APP_DEBUG` off |
| `app_key` | `fail`: the key is missing or malformed, or it is valid but cannot decrypt stored credentials, which means the key changed. Restore the old key or list it in `APP_PREVIOUS_KEYS` |
| `app_key_location` | `warn`: the `APP_KEY` file is on the same volume as the SQLite database. Supply `APP_KEY` or `APP_KEY_FILE` from outside the volume |
| `database` | `fail`: the application database is unreachable, or migrations are pending. Run `php artisan migrate --force` |
| `queue_timing` | `fail`: the queue's `retry_after` is not greater than the worker timeout, so a long query would be started a second time. Raise `retry_after` or lower `QUERYPROXY_EXECUTION_TIMEOUT` |
| `worker` | `warn`: jobs have waited longer than the execution timeout plus a minute; no queue worker seems to run. Start `php artisan queue:work --queue=queries,default`. `fail`: the queue table does not exist, run `php artisan migrate --force`. `skip`: it measures the `database` queue only |
| `scheduler` | `warn`: no heartbeat, or an old one. Start `php artisan schedule:work`. Right after a start, wait a minute |
| `failed_jobs` | `warn`: jobs failed. Inspect with `php artisan queue:failed`; failed SIEM deliveries end up here too |
| `storage_permissions` | `fail`: a path under `storage/` is not writable by the user running QueryProxy |
| `result_disk` | `fail`: the disk named by `QUERYPROXY_RESULT_DISK` does not exist, or is public. Use a private disk |
| `trusted_proxies` | `fail`: `TRUSTED_PROXIES` is `*` or another catch-all (`**`, `0.0.0.0/0`, `::/0`). List the proxy's IPs or CIDRs. A broad but valid range such as `10.0.0.0/8` passes, so keep it as narrow as your network allows |
| `sqlite_allowed_dir` | `warn`: SQLite connections exist and `QUERYPROXY_SQLITE_ALLOWED_DIR` is empty, or it points at no existing directory. With no SQLite connection and no setting the check is `ok` |
| `masking_coverage` | `warn`: a team has no enabled masking rule, so its results are not masked. Install the default rules on the masking page |
| `tls_enforcement` | `warn`: some connections do not require TLS (see [TLS failures](#tls-failures)). Edit them and require TLS |
| `tls_files` | `fail`: a stored PEM cannot be decrypted, usually because the key changed. `warn`: the files under `storage/app/tls` have wrong permissions |
| `secret_files` | `fail`: a `*_FILE` variable is unreadable, empty, or set together with its plain variable |
| `metrics` | `warn`: the token is shorter than 24 characters, or `QUERYPROXY_METRICS_ALLOWED_IPS` holds an entry that is not an IP address or CIDR range (it never matches). `fail`: the `_FILE` of the token cannot be read. `skip`: `/metrics` is off |
| `siem_sinks` | `fail`: a configured sink would not be used, so audit events are not exported. `warn`: a webhook on plain `http://`, or syslog over UDP (unencrypted and unacknowledged); `--strict` turns these into failures of the run |
| `audit_chain` | `fail`: the chain is broken or records were removed. Run `php artisan audit:verify` |

## Requests

### A request stays `queued`

Three causes, in order of likelihood:

1. **No worker.** The doctor's `worker` check warns. Start the worker, or set
   `QUERYPROXY_RUN_WORKER=true` in the container.
2. **Every connection slot is busy.** At most `QUERYPROXY_CONNECTION_SLOTS`
   requests (default `2`) run on one connection at once; the rest wait and
   start on their own. The audit log shows `request.execution_deferred` with
   `reason=connection_slots_busy`, and the metric
   `queryproxy_slot_deferred_requests_total` counts it. Raise the setting or set
   it to `0` if the target database can take more.
3. **The worker listens to the wrong queue.** It must serve `queries` and
   `default`: `--queue=queries,default`.

### A read fails with "tried to change data"

The message reads: *This statement tried to change data, but read requests run
in a read-only database session. Submit it as a write request if the change is
intended.* The database refused a write inside the read-only session. On
PostgreSQL this includes `SELECT … FOR UPDATE` and `FOR SHARE`. Submit the statement as a write, or remove the locking clause.

### A read fails after the database dropped the connection

QueryProxy does not reconnect once a statement has started, so a brief network
drop fails the run. Submit the request again.

### A write runs past the execution timeout on MySQL

The database-side statement timeout reaches SELECT only on MySQL
(`MAX_EXECUTION_TIME`), so a long MySQL write is not cut by the database. The
job ends when `QUERYPROXY_EXECUTION_TIMEOUT` elapses and the request fails; the
write may still have run on the server. Use **Stop query** to cancel it earlier.

### "Stop query" says the query could not be signalled

The request is marked `cancelled`, but the database did not receive the cancel
signal and the statement ends at its timeout. The button appears only for a
running request whose database session is known, and not for SQLite. Check the
privileges the connection's user needs:

- **PostgreSQL:** the same user, or a role with `pg_signal_backend`.
- **MySQL and MariaDB:** the same user, or `CONNECTION_ADMIN`. Only the
  statement is cancelled.
- **SQL Server:** `VIEW SERVER STATE` (without it the session of the run is
  invisible to the control connection) and `ALTER ANY CONNECTION` (`KILL` needs
  it). Stop on SQL Server ends the **session**, and an open transaction rolls
  back on the server.

### The SQL preview in chat reads "SQL hidden"

The statement could not be redacted safely, so no preview is shown.
`QUERYPROXY_CHAT_INCLUDE_SQL=none` hides the preview everywhere; `full` shows the
statement as submitted.

## TLS failures

A connection created in v0.3.0 requires TLS and fails if the server does not
offer it. Read the connection test result: it shows the TLS state.

| Symptom | Cause and fix |
| :-- | :-- |
| PostgreSQL: *server does not support SSL, but SSL was required* | The server has no TLS. Enable it on the server. Only on a network you fully trust, choose `prefer` or `disable` in the connection's SSL mode, which allows plain text |
| MySQL or MariaDB: connection closes right after the handshake (error 2006) | The server offers no TLS. Enable it on the server |
| PostgreSQL `verify-full` or MySQL "Verify the server certificate" fails | The server certificate does not chain to the CA you pasted, or its name does not match the host. Paste the correct CA PEM; for `verify-full` the host must match the certificate |
| SQL Server: certificate error | The server certificate is not from a trusted CA. Install the CA on the QueryProxy host, or set `TrustServerCertificate` for a self-signed one on purpose |
| SQL Server: `RuntimeException` that `pdo_sqlsrv` is missing | A required-TLS SQL Server connection needs the `pdo_sqlsrv` extension. Without it Laravel would fall back to an unencrypted `dblib` connection, so QueryProxy refuses to connect |

Connections from before the upgrade keep their old behaviour; `tls_enforcement`
lists them.

## Chat

### A Slack or Teams button says it "has expired"

The message *this button has expired. Use the button in the newer message* means
the 24-hour action token of that card is gone while the request is still
pending. Requests do not expire; open the request in QueryProxy, or wait for the
reminder, which posts a card with a new token (see `QUERYPROXY_PENDING_REMIND_AFTER`).

### Teams receives nothing, or the flow reads empty fields

Check the webhook host first. `*.powerplatform.com` and `*.logic.azure.com`
receive an Adaptive Card; legacy connector URLs (`*.webhook.office.com`, and
`outlook.office.com` once you have added it to the allowlist) a MessageCard. A custom flow that reads the old MessageCard fields from a
Workflows URL gets empty values. See [Teams integration](/docs/teams-integration/).
A host outside the allowlist is rejected; extend it with
`QUERYPROXY_CHAT_WEBHOOK_ALLOWED_HOSTS`.

### Slack **Approve** answers with a link to a web form

That click would be an admin override, and overrides need a written reason on
every channel. Nothing was approved; open the link and fill in the reason.

## Monitoring and export

### `/metrics` answers `404`, `401` or `400`

- `404`: it is off. Set `QUERYPROXY_METRICS_TOKEN` or
  `QUERYPROXY_METRICS_ALLOWED_IPS`.
- `401`: wrong token, or the client address is not in the allowlist. Behind a
  reverse proxy the address is the proxy's unless you set `TRUSTED_PROXIES`.
- `400`: the `Host` header of the scrape is not `APP_URL`'s host and is not in
  `TRUSTED_HOSTS`. Set the `Host` header in the scrape config or list the name.

See [Monitoring](/docs/monitoring/).

### SIEM events do not arrive

1. Run the doctor; `siem_sinks` names a sink that would not be used (a webhook
   without a 16-character secret is never sent unsigned).
2. Run a worker for the SIEM queue. The default is `default`, which the stock
   worker serves; a custom `QUERYPROXY_SIEM_QUEUE` needs its own worker.
3. Look at `php artisan queue:failed`. A delivery that exhausted its retries
   lands there.

Delivery is at least once, so receivers should deduplicate on `id`.

## Secrets and startup

### The container exits with "Both X and X_FILE are set"

Define only one of the two. A plain variable set to `null`, `(null)` or
`(empty)` counts as not set. The message names the variable and the path, never
the value.

### The container exits with "`X_FILE` points at … which is not a readable file"

Mount the secret at that path and make it readable by the user the container
runs as. An empty file is refused too. Trailing newlines are stripped; nothing
else is.

### Stored credentials are unreadable after a restart

The `APP_KEY` changed. Restore the original key, or list it in
`APP_PREVIOUS_KEYS`. See [Backup and restore](/docs/backup-restore/#the-app_key).

### A secret read from a file still shows in the configuration cache

`php artisan config:cache` writes every configuration value, including secrets
read from files, in plain text to `bootstrap/cache/config.php`. That is how the
cache has always worked, and it partly cancels the point of `*_FILE`. The image
re-caches the configuration on every start. Keep that path unreadable to other
users and out of images and backups that are shared.

## Upgrade and database

### The migration fails with error 1419

MySQL or MariaDB refused to create the audit triggers. The migration user needs
the `TRIGGER` privilege and, with binary logging on, `SUPER` or
`log_bin_trust_function_creators=1`. See [Upgrading](/docs/upgrading/).

### `audit:verify` reports a break

`audit:verify` names the first broken record and why: a record that carries no
hash, does not link to the one before it, does not match its hash, a first
record that does not start the chain, a length that differs from the number of
records written, or a head that is not the recorded one. Compare the chain with
your SIEM copy before you act; see
[Audit chain and SIEM export](/docs/audit-export/).

### A MySQL request is refused with a `sql_mode` message

The target session's `sql_mode` contains `NO_BACKSLASH_ESCAPES` or
`ANSI_QUOTES`, which the guard reads differently. See
[Configuration](/docs/configuration/#mysql--mariadb-targets-sql_mode).

## Next steps

- [Backup and restore](/docs/backup-restore/)
- [Production hardening](/docs/production-hardening/)
- [Threat model](/docs/threat-model/)