Production deployment

Start

export POSTGRES_PASSWORD="$(openssl rand -hex 24)"
export CLICKHOUSE_PASSWORD="$(openssl rand -hex 24)"
export MEILI_MASTER_KEY="$(openssl rand -hex 32)"
export CRYPTO_SECRET="$(openssl rand -hex 16)"
export SERVER_CSRF_HMAC_SECRET="$(openssl rand -hex 32)"
export GF_SECURITY_ADMIN_PASSWORD="$(openssl rand -hex 24)"
COMPOSE_FILE=compose.prod.yaml scripts/compose/up.sh

Set all six values in the shell or copy .env.example to .env and fill it. Persist them in your secret manager and reuse them across server restarts. CRYPTO_SECRET and SERVER_CSRF_HMAC_SECRET must be distinct. Startup rejects empty values and reused server secrets in any environment.

For Kubernetes, use the setup script. It preserves valid, complete pre-created Secrets and generates missing bootstrap material:

scripts/k8s/setup.sh --mode production --context <context>

Production publishes:

• Dashboard and proxied API through Nginx on port 80.

Grafana is not host-published by default (no port 3000); use the opt-in Compose loopback override (compose.grafana.yaml → 127.0.0.1:${GRAFANA_HOST_PORT:-3000}:3000) or kubectl -n chronoverse port-forward svc/lgtm 3000:3000.

Per-IP rate limits assume nginx sees real client addresses. On Linux hosts Docker preserves client IPs through published ports via DNAT; on Docker Desktop, published ports traverse the userland proxy (notably for IPv6), so nginx may see the gateway address instead and those clients aggregate into one rate-limit bucket.

Kubernetes production is expected to be accessed through the configured Ingress host over HTTPS. Authentication cookies are scoped from SERVER_HOST_URL, so a plain http://localhost port-forward will not preserve login cookies when the server is configured for https://chronoverse.example.com. The production Ingress references chronoverse-ingress-tls; replace the generated self-signed fallback with a trusted certificate before exposing real traffic.

Application services and infrastructure otherwise remain on Compose networks. The Compose production profile still represents one Docker runtime named local-docker. In Kubernetes, production is self-hosted: PostgreSQL, Redis, Kafka, ClickHouse, Meilisearch, services, workers, and runtime-agent run inside the cluster. Each Docker-capable node should run runtime-agent beside docker-proxy and register tcp://$(NODE_IP):2376 via hostPort:2376 (127.0.0.1:2376 health); workers can schedule anywhere and route Docker work through the runtime registry. Kubernetes setup generates docker-proxy-ca/server/client-* mTLS material plus docker-proxy-auth token — the proxy binds :2376 ssl ... verify required and enforces token + exact method/path allowlist via WithDockerProxyTLS/WithDockerProxyToken.

Maintenance-window upgrades

Deploy schema-changing releases as an offline cutover; mixed application versions are not supported across an idempotency-ledger migration.

1. Quiesce mutations

   Back up the databases, stop public traffic, and stop the server, workers, processors, and outbox relay. Drain in-flight requests and transactions.
2. Apply the new schema

   Run the new release's database-migration image. Resolve any preflight failure before changing application versions. Then use an approved Redis SCAN plus UNLINK procedure to delete keys matching workflow:* and workflows:*, and verify both scans are empty while application processes remain stopped. Migration-time threshold reconciliation updates PostgreSQL directly and cannot invalidate Redis atomically. Do not use FLUSHDB, because Redis also stores sessions and coordination state.
3. Start internal processing

   Start one version of the domain services, then the relay and workers. Keep public traffic stopped while checking health, consumer progress, outbox publication, and runtime registration.
4. Canary and resume

   Exercise replay, changed-input conflict, scheduling, termination, deletion, and deterministic outbox keys with approved fixtures. Clean them up before restoring public traffic.

The standard migration executable applies upgrades only. Rollback requires an operator-reviewed down migration or a verified pre-upgrade database restore while all mutation paths remain stopped. Starting old images against the new schema is unsafe. Prefer restoring the backup if the legacy schema's documented loss of non-workflow command identities, completed terminal identities, and superseded reused manual keys is unacceptable.

Images

Application images are pulled from ghcr.io/hitesh22rana/chronoverse/*:latest. Tagged releases are built for supported architectures by GitHub Actions.

Worker replicas

The production profile declares replicated workers and resource reservations. Effective parallelism still depends on Kafka partitions, registered runtime capacity, Docker host capacity, database pools, and lease configuration.

Required hardening

• Replace database and Meilisearch credentials.
• Manage certificates and private keys with environment-specific controls.
• Pre-create Kubernetes Secrets when you need operator-owned credentials or certificates; otherwise the setup script generates missing bootstrap material and never overwrites valid, complete existing Secrets.
• Configure the public host URL, allowed origins, secure cookies, and same-site policy.
• Pin reviewed image tags instead of relying on mutable latest in controlled environments.
• Restrict access to the LGTM dashboard.
• Restrict Docker proxy hostPort:2376 (bypasses NetworkPolicy across CNIs) to trusted Chronoverse traffic at infra layer (node firewall / security group / CNI host policy) in addition to mTLS verify required + docker-proxy-auth token + exact allowlist.
• Define backup, restore, and retention procedures for PostgreSQL and ClickHouse.