Kubernetes, Helm and high availability

QueryProxy ships a Helm chart in the repository, at deploy/helm/queryproxy. It runs the one published image. With an external application database it starts that image as separate web, worker and scheduler Deployments that scale independently. This page is a summary; the chart’s README has the full list of values and the details. The chart needs Kubernetes 1.27 or newer. From 0.4.1 the chart is also published to a registry (see Install from the OCI registry); otherwise install it from the chart directory of a checkout at the release tag you want to run.

Two modes

SQLite means one pod. A SQLite file lives on one volume and cannot be shared between pods, so with SQLite the chart installs a single all-in-one Deployment, as docker-compose does. It refuses to render with more than one web replica, with worker autoscaling, or without a persistent volume. Use this mode to evaluate QueryProxy or for a small team; to scale, move to an external database first.

The role split applies only with an external application database. There the roles are switched with QUERYPROXY_RUN_WORKER and QUERYPROXY_RUN_SCHEDULER: the web Deployment runs neither, the worker Deployment runs only the queue worker, and the scheduler Deployment runs only the scheduler. A crash loop or an expensive query in a worker no longer takes the UI with it. docker-compose is unchanged and still runs everything in one container.

Install from the OCI registry

Every release tag publishes the chart to GitHub Container Registry as oci://ghcr.io/queryproxy/charts/queryproxy. The chart version is the release version without the v. Use these commands in place of the local path in the examples below:

helm upgrade --install queryproxy oci://ghcr.io/queryproxy/charts/queryproxy --version 0.4.1 \
  --namespace queryproxy --create-namespace \
  --set [email protected] --set secrets.adminPassword="$(openssl rand -hex 16)"

# Or fetch it first, for example to read values-ha.yaml:
helm pull oci://ghcr.io/queryproxy/charts/queryproxy --version 0.4.1 --untar

The release workflow signs the chart keylessly with cosign (Sigstore, GitHub OIDC identity), the same way it signs the images, and checks the signature itself before it finishes. Verify it with cosign v3 or later before you install. The identity must be the release workflow of the QueryProxy repository at a version tag, issued by GitHub Actions:

cosign verify \
  --certificate-identity-regexp '^https://github\.com/QueryProxy/QueryProxy/\.github/workflows/release\.yml@refs/tags/v[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  ghcr.io/queryproxy/charts/queryproxy:0.4.1

Replace 0.4.1 with the version you install. A release is only published when the tag matches version and appVersion in the chart’s Chart.yaml, so the chart you pull always belongs to the image of the same number.

Try it on SQLite

helm upgrade --install queryproxy deploy/helm/queryproxy \
  --namespace queryproxy --create-namespace \
  --set [email protected] \
  --set secrets.adminPassword="$(openssl rand -hex 16)"

kubectl -n queryproxy port-forward svc/queryproxy 7432:80

Then open http://localhost:7432. config.appUrl defaults to that address. QueryProxy answers only to the host of APP_URL, the Service’s in-cluster names (which the chart adds) and config.trustedHosts, so for a real install set config.appUrl to the address users type, usually the HTTPS URL of the Ingress.

High availability

The reference profile is values-ha.yaml:

                     Ingress / LoadBalancer
                              |
                      Service (80 -> 7432)
                              |
        +---------------------+----------------------+
        |                                            |
 web x N (nginx + php-fpm)           worker x M (queue:work)      scheduler x 1
        |                                            |                  |
        +--------------------+-----------------------+------------------+
                             |
          external PostgreSQL / MySQL (users, requests, queue, sessions, audit chain)
                             |
          S3 bucket (result files)         target databases (reached by the workers)

The profile sets an external PostgreSQL with sslMode: require, S3 result storage, two web and two worker pods, spread over zones where the scheduler can (ScheduleAnyway), one scheduler, an Ingress with TLS, a PodDisruptionBudget for web and worker, and a NetworkPolicy. Install it like this:

1. Provision PostgreSQL (a managed service or an operator) with a database and a role for QueryProxy, and a private S3 bucket for result files.
2. Create the Secret named in secrets.existingSecret before you install; the migration hook needs it.
3. Replace every example value in values-ha.yaml, then:

helm upgrade --install queryproxy deploy/helm/queryproxy \
  -f deploy/helm/queryproxy/values-ha.yaml \
  --namespace queryproxy --create-namespace

What you size yourself:

• PostgreSQL is the one stateful dependency. It holds the queue, the sessions and the append-only audit chain, so use a replicated, backed-up service. Every pod opens its own connections; put a pooler in front for large worker counts.
• Workers run approved queries. Parallelism is worker.replicas times config.workerProcesses. A stopping worker pod gets config.executionTimeout + 30 seconds to finish a running query.
• The scheduler is one replica with the Recreate strategy. Two schedulers would run every scheduled task (retention pruning, reminders, expiry of time-bound grants) twice.

Secrets

All pods need the same APP_KEY, plus the database password and, when used, the Redis password, the metrics token and the S3 keys.

• Your own Secret (production). Set secrets.existingSecret. The expected keys are APP_KEY, DB_PASSWORD and, as needed, REDIS_PASSWORD, QUERYPROXY_METRICS_TOKEN, QUERYPROXY_ADMIN_PASSWORD, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. The Secret can come from the External Secrets Operator; the chart README has an example. Restart the pods yourself after rotating it.
• Chart-managed Secret (evaluation). Leave secrets.existingSecret empty. The chart generates the APP_KEY on first install and keeps it. The Secret survives helm uninstall on purpose, because stored connection credentials cannot be decrypted without the key: back it up.
• Secrets as files. secrets.mountAsFiles=true mounts the Secret and sets the *_FILE variables, so the values do not appear in the pod spec. With IRSA or Workload Identity you can leave the S3 keys out (results.s3.useSecretCredentials=false).

Connection passwords do not have to live in QueryProxy at all. With external secret references a pod reads them from Vault, AWS Secrets Manager or Azure Key Vault when a connection opens: IRSA for AWS, workload identity for Azure and Vault’s Kubernetes login need no stored credential. Pass the plain settings (QUERYPROXY_VAULT_ADDR, QUERYPROXY_VAULT_AUTH, QUERYPROXY_VAULT_K8S_ROLE, QUERYPROXY_AWS_REGION, QUERYPROXY_AZURE_TENANT_ID and the like) through config.extraEnv. Values in config.extraEnv land in a ConfigMap and in the pod spec as plain text, so never put QUERYPROXY_VAULT_TOKEN, QUERYPROXY_VAULT_SECRET_ID or QUERYPROXY_AZURE_CLIENT_SECRET there. Keep them in a Kubernetes Secret whose keys carry those names and reference it from config.extraEnvFrom:

config:
  extraEnvFrom:
    - secretRef:
        name: queryproxy-secret-store

For single sign-on, pass the QUERYPROXY_OIDC_* variables through config.extraEnv, and keep the client secret in a Secret that you reference from config.extraEnvFrom. See Single sign-on.

Migrations and upgrades

With an external database a pre-install,pre-upgrade Job runs php artisan migrate --force before any pod of the new version starts. If it fails, the release does not proceed. The ConfigMap, Secret and ServiceAccount are ordinary release resources, so helm rollback restores the previous configuration. A rollback does not reverse migrations; follow Upgrading and rolling back and keep a database backup.

To move an existing SQLite installation to PostgreSQL, stop it and use php artisan queryproxy:migrate-appdb --to=<dsn>, then install the chart with the new database.* values.

Autoscaling the workers

worker.autoscaling.enabled=true renders a HorizontalPodAutoscaler on the external metric queryproxy_queue_depth. It is off by default, because it needs:

1. metrics.enabled=true with secrets.metricsToken.
2. Prometheus scraping /metrics through the Service name. QueryProxy answers only to trusted host names, so a scrape of a pod IP gets 400.
3. A metrics adapter that serves the metric; the chart’s values.yaml has a prometheus-adapter rule.

Scale-down is deliberately slow, because stopping a pod that runs a query waits for it. See Monitoring for the metric.

Network, availability and hardening

• Probes call GET /up with the host of config.appUrl.
• PodDisruptionBudget with maxUnavailable: 1 for web and worker. The scheduler has none: it is one pod, and a drain restarts it.
• NetworkPolicy (networkPolicy.enabled): only networkPolicy.ingressFrom peers reach port 7432; with an empty ingressFrom (the default) any source can still reach it, and every other port is closed. NetworkPolicy takes effect only if your network plugin enforces it, for example Calico or Cilium. Egress is open by default, because workers must reach every target database you register. With networkPolicy.egress.restrict=true list DNS, the application database, S3, your target databases and your chat and SIEM endpoints in networkPolicy.egress.rules. The ingress rule covers every pod of the release, so add the namespace of a Prometheus that scrapes /metrics to ingressFrom.
• Security context. The pods run as a non-root user with all capabilities dropped and no privilege escalation.
• The root filesystem is still writable. The image’s entrypoint writes to it at start, so readOnlyRootFilesystem is off. The chart cannot promise more today.

S3 and Redis

The image carries the S3 adapter and the predis Redis client. Both are optional. results.disk=s3 stores result files on S3 or an S3-compatible endpoint; without S3, use a ReadWriteMany volume shared by web and worker. redis.enabled moves cache, queue and sessions to Redis; everything also works on the database alone.

Next steps

• Production hardening for the checklist that still applies.
• Configuration for every environment variable the chart can pass.
• Backup and restore for the database and the APP_KEY.