QueryProxy
Documentation menu

Troubleshooting

Start with queryproxy:doctor, read each check, and fix the errors operators meet most, from stuck requests and TLS failures to chat buttons and secrets.

Updated:

Start with the doctor. It reads the installation and tells you what is wrong before you go digging in logs.

docker exec queryproxy php artisan queryproxy:doctor

Without Docker, run php artisan queryproxy:doctor in the application directory.

Reading the doctor

The command prints a table of checks with a status each:

Status Meaning
ok The check passed
warn Something deserves attention; the run still exits with 0
fail A problem that breaks or endangers the install; the run exits with 1
skip The check does not apply (no SIEM sink is turned on, the cache or queue is not the kind the check can read, and so on)

Options:

  • --strict makes warnings exit with 1 as well. Use it in a deploy pipeline.
  • --full-audit walks the whole audit chain. Without it the doctor checks the newest records and the chain head, which is faster.
  • --json prints {"ok": …, "strict": …, "checks": […]} for scripts.

The doctor only reads. It does not change anything on the installation.

What each check means

Check If it is not ok, do this
app_environment fail: APP_DEBUG is on with APP_ENV=production. warn: it is on in another environment. Turn APP_DEBUG off
app_key fail: the key is missing or malformed, or it is valid but cannot decrypt stored credentials, which means the key changed. Restore the old key or list it in APP_PREVIOUS_KEYS
app_key_location warn: the APP_KEY file is on the same volume as the SQLite database. Supply APP_KEY or APP_KEY_FILE from outside the volume
database fail: the application database is unreachable, or migrations are pending. Run php artisan migrate --force
queue_timing fail: the queue’s retry_after is not greater than the worker timeout, so a long query would be started a second time. Raise retry_after or lower QUERYPROXY_EXECUTION_TIMEOUT
worker warn: jobs have waited longer than the execution timeout plus a minute; no queue worker seems to run. Start php artisan queue:work --queue=queries,default. fail: the queue table does not exist, run php artisan migrate --force. skip: it measures the database queue only
scheduler warn: no heartbeat, or an old one. Start php artisan schedule:work. Right after a start, wait a minute
failed_jobs warn: jobs failed. Inspect with php artisan queue:failed; failed SIEM deliveries end up here too
storage_permissions fail: a path under storage/ is not writable by the user running QueryProxy
result_disk fail: the disk named by QUERYPROXY_RESULT_DISK does not exist, or is public. Use a private disk
trusted_proxies fail: TRUSTED_PROXIES is * or another catch-all (**, 0.0.0.0/0, ::/0). List the proxy’s IPs or CIDRs. A broad but valid range such as 10.0.0.0/8 passes, so keep it as narrow as your network allows
sqlite_allowed_dir warn: SQLite connections exist and QUERYPROXY_SQLITE_ALLOWED_DIR is empty, or it points at no existing directory. With no SQLite connection and no setting the check is ok
masking_coverage warn: a team has no enabled masking rule, so its results are not masked. Install the default rules on the masking page
tls_enforcement warn: some connections do not require TLS (see TLS failures). Edit them and require TLS
tls_files fail: a stored PEM cannot be decrypted, usually because the key changed. warn: the files under storage/app/tls have wrong permissions
secret_files fail: a *_FILE variable is unreadable, empty, or set together with its plain variable
metrics warn: the token is shorter than 24 characters, or QUERYPROXY_METRICS_ALLOWED_IPS holds an entry that is not an IP address or CIDR range (it never matches). fail: the _FILE of the token cannot be read. skip: /metrics is off
siem_sinks fail: a configured sink would not be used, so audit events are not exported. warn: a webhook on plain http://, or syslog over UDP (unencrypted and unacknowledged); --strict turns these into failures of the run
audit_chain fail: the chain is broken or records were removed. Run php artisan audit:verify

Requests

A request stays queued

Three causes, in order of likelihood:

  1. No worker. The doctor’s worker check warns. Start the worker, or set QUERYPROXY_RUN_WORKER=true in the container.
  2. Every connection slot is busy. At most QUERYPROXY_CONNECTION_SLOTS requests (default 2) run on one connection at once; the rest wait and start on their own. The audit log shows request.execution_deferred with reason=connection_slots_busy, and the metric queryproxy_slot_deferred_requests_total counts it. Raise the setting or set it to 0 if the target database can take more.
  3. The worker listens to the wrong queue. It must serve queries and default: --queue=queries,default.

A read fails with “tried to change data”

The message reads: This statement tried to change data, but read requests run in a read-only database session. Submit it as a write request if the change is intended. The database refused a write inside the read-only session. On PostgreSQL this includes SELECT … FOR UPDATE and FOR SHARE. Submit the statement as a write, or remove the locking clause.

A read fails after the database dropped the connection

QueryProxy does not reconnect once a statement has started, so a brief network drop fails the run. Submit the request again.

A write runs past the execution timeout on MySQL

The database-side statement timeout reaches SELECT only on MySQL (MAX_EXECUTION_TIME), so a long MySQL write is not cut by the database. The job ends when QUERYPROXY_EXECUTION_TIMEOUT elapses and the request fails; the write may still have run on the server. Use Stop query to cancel it earlier.

“Stop query” says the query could not be signalled

The request is marked cancelled, but the database did not receive the cancel signal and the statement ends at its timeout. The button appears only for a running request whose database session is known, and not for SQLite. Check the privileges the connection’s user needs:

  • PostgreSQL: the same user, or a role with pg_signal_backend.
  • MySQL and MariaDB: the same user, or CONNECTION_ADMIN. Only the statement is cancelled.
  • SQL Server: VIEW SERVER STATE (without it the session of the run is invisible to the control connection) and ALTER ANY CONNECTION (KILL needs it). Stop on SQL Server ends the session, and an open transaction rolls back on the server.

The SQL preview in chat reads “SQL hidden”

The statement could not be redacted safely, so no preview is shown. QUERYPROXY_CHAT_INCLUDE_SQL=none hides the preview everywhere; full shows the statement as submitted.

TLS failures

A connection created in v0.3.0 requires TLS and fails if the server does not offer it. Read the connection test result: it shows the TLS state.

Symptom Cause and fix
PostgreSQL: server does not support SSL, but SSL was required The server has no TLS. Enable it on the server. Only on a network you fully trust, choose prefer or disable in the connection’s SSL mode, which allows plain text
MySQL or MariaDB: connection closes right after the handshake (error 2006) The server offers no TLS. Enable it on the server
PostgreSQL verify-full or MySQL “Verify the server certificate” fails The server certificate does not chain to the CA you pasted, or its name does not match the host. Paste the correct CA PEM; for verify-full the host must match the certificate
SQL Server: certificate error The server certificate is not from a trusted CA. Install the CA on the QueryProxy host, or set TrustServerCertificate for a self-signed one on purpose
SQL Server: RuntimeException that pdo_sqlsrv is missing A required-TLS SQL Server connection needs the pdo_sqlsrv extension. Without it Laravel would fall back to an unencrypted dblib connection, so QueryProxy refuses to connect

Connections from before the upgrade keep their old behaviour; tls_enforcement lists them.

Chat

A Slack or Teams button says it “has expired”

The message this button has expired. Use the button in the newer message means the 24-hour action token of that card is gone while the request is still pending. Requests do not expire; open the request in QueryProxy, or wait for the reminder, which posts a card with a new token (see QUERYPROXY_PENDING_REMIND_AFTER).

Teams receives nothing, or the flow reads empty fields

Check the webhook host first. *.powerplatform.com and *.logic.azure.com receive an Adaptive Card; legacy connector URLs (*.webhook.office.com, and outlook.office.com once you have added it to the allowlist) a MessageCard. A custom flow that reads the old MessageCard fields from a Workflows URL gets empty values. See Teams integration. A host outside the allowlist is rejected; extend it with QUERYPROXY_CHAT_WEBHOOK_ALLOWED_HOSTS.

That click would be an admin override, and overrides need a written reason on every channel. Nothing was approved; open the link and fill in the reason.

Monitoring and export

/metrics answers 404, 401 or 400

  • 404: it is off. Set QUERYPROXY_METRICS_TOKEN or QUERYPROXY_METRICS_ALLOWED_IPS.
  • 401: wrong token, or the client address is not in the allowlist. Behind a reverse proxy the address is the proxy’s unless you set TRUSTED_PROXIES.
  • 400: the Host header of the scrape is not APP_URL’s host and is not in TRUSTED_HOSTS. Set the Host header in the scrape config or list the name.

See Monitoring.

SIEM events do not arrive

  1. Run the doctor; siem_sinks names a sink that would not be used (a webhook without a 16-character secret is never sent unsigned).
  2. Run a worker for the SIEM queue. The default is default, which the stock worker serves; a custom QUERYPROXY_SIEM_QUEUE needs its own worker.
  3. Look at php artisan queue:failed. A delivery that exhausted its retries lands there.

Delivery is at least once, so receivers should deduplicate on id.

Secrets and startup

The container exits with “Both X and X_FILE are set”

Define only one of the two. A plain variable set to null, (null) or (empty) counts as not set. The message names the variable and the path, never the value.

The container exits with “X_FILE points at … which is not a readable file”

Mount the secret at that path and make it readable by the user the container runs as. An empty file is refused too. Trailing newlines are stripped; nothing else is.

Stored credentials are unreadable after a restart

The APP_KEY changed. Restore the original key, or list it in APP_PREVIOUS_KEYS. See Backup and restore.

A secret read from a file still shows in the configuration cache

php artisan config:cache writes every configuration value, including secrets read from files, in plain text to bootstrap/cache/config.php. That is how the cache has always worked, and it partly cancels the point of *_FILE. The image re-caches the configuration on every start. Keep that path unreadable to other users and out of images and backups that are shared.

Upgrade and database

The migration fails with error 1419

MySQL or MariaDB refused to create the audit triggers. The migration user needs the TRIGGER privilege and, with binary logging on, SUPER or log_bin_trust_function_creators=1. See Upgrading.

audit:verify reports a break

audit:verify names the first broken record and why: a record that carries no hash, does not link to the one before it, does not match its hash, a first record that does not start the chain, a length that differs from the number of records written, or a head that is not the recorded one. Compare the chain with your SIEM copy before you act; see Audit chain and SIEM export.

A MySQL request is refused with a sql_mode message

The target session’s sql_mode contains NO_BACKSLASH_ESCAPES or ANSI_QUOTES, which the guard reads differently. See Configuration.

Next steps

Arrow keys to move, Enter to open.