Local certificate bootstrap
init-certs generates a local CA, server certificates, client certificates, Kafka keystore and truststore material, and Ed25519 authentication keys in the shared certificate volume.
The Kubernetes local overlay uses the same bootstrap idea through Jobs and a local certificate PVC. The production setup script creates missing Kubernetes Secrets for bootstrap material and preserves valid, complete operator-provided Secrets. PostgreSQL certificates cover the PgBouncer client endpoint (postgres) and direct backend endpoint (postgres-primary).
Encrypted infrastructure
Compose enables TLS or mTLS for PostgreSQL, ClickHouse, Redis, Meilisearch, Kafka, and gRPC service connections. Kubernetes uses the same posture: data-store clients mount the generated or operator-provided client certificate, and PostgreSQL host connections require TLS plus a CA-verified client certificate.
Service authorization
The HTTP gateway signs downstream authorization metadata. gRPC services validate that context in addition to transport identity.
Each service signs JWTs with its own Ed25519 private key and a kid header. The gateway and every verifier mount only its own certs/issuers/<issuer>/auth.ed (0440, fsGroup: 101 in Kubernetes) plus a shared certs/issuers/trusted.json bundle and the 13 public keys certs/issuers/*/auth.ed.pub; Auth.ValidateToken requires the kid and verifies the signature only against the matching bundle entry with issuer binding. New builds inject the per-issuer paths through Makefile ldflags (for example certs/issuers/users-service/auth.ed) and both Compose profiles generate the per-issuer keys and trusted.json in init-certs and mount only the calling service's private key plus the shared pubs/trusted bundle (and certs/ca/certs/clients for TLS) — the previous whole-tree ./certs:/certs:ro mount is removed so a compromised service cannot read another issuer's private key — gating startup on the bundle. Infrastructure services (postgres, redis, meilisearch, kafka) mount only their own TLS subtrees and ca/clients, not the issuer keys. Kubernetes generates the same layout in cert-bootstrap and projects per-workload the bundle, the single issuers/<svc>/auth.ed private, and all issuers/*/auth.ed.pub via mounted volumes; the local overlay keeps issuer material on a dedicated auth-certs-pvc, separate from the TLS PVC mounted by infrastructure workloads. Rotation is graceful: scripts/compose/rotate-auth-key.sh <issuer> adds a new key alongside the old kid in the bundle — automatically using a privileged helper container (docker run -v ./certs:/certs:rw alpine) when the host operator (e.g. UID 1000) cannot write the 100:101 0440 bind-mounted keys — verifiers reload the dual-key bundle before the signer switches to the new kid (docker compose -f compose.prod.yaml up -d --force-recreate <verifiers> then ... <signer>; COMPOSE_FILE is respected and compose.yaml does not exist), and the old kid is pruned after its 15-minute expiry window (all auth services restart to drop it); make compose/validate guards the wiring and rejects whole-tree /certs mounts.
Internal Ed25519 JWT validation allows at most 30 seconds of clock skew for
nbf and expiry checks so ordinary multi-node time differences do not reject a
fresh token. Keep every Kubernetes node synchronized with a reliable time
source; the bounded leeway is not a substitute for host time synchronization.
Browser security
Sessions are stored in encrypted cookies. Mutating browser requests require CSRF validation, and token HMACs are compared in constant time. CORS origins and same-site behavior are configurable.
The HTTP API adds X-Content-Type-Options: nosniff, X-Frame-Options: DENY, Referrer-Policy: no-referrer, and an API-only Content Security Policy that disables resource loading and framing.
Docker access
Execution components use an mTLS socket proxy with a narrow API surface instead of broad direct daemon access. A dedicated proxy CA, a generated high-entropy token, and exact certificate-subject ACLs authenticate and authorize three roles: runtime-agent can only ping/read version; workflow-worker can inspect/pull images and log/stop/delete containers for cancellation cleanup; execution-worker additionally receives container inspect/create/start/wait and network inspect/create. Unknown subjects and cross-role operations are denied. Workload containers never receive a token or certificate, and /networks/*/connect and /networks/*/disconnect are denied.
Compose stores the proxy issuer, server identity, and each client role outside the shared certs tree and mounts only the exact role directory into a runtime service. Kubernetes uses separate projected volumes and non-root-readable 0440 role keys; the proxy server key is never mounted into runtime-agent. TLS clients reload the CA bundle and client keypair on every new handshake, which supports staged rotation without pinning cached clients to old files.
The 2375 to 2376 upgrade is backward-compatible with persisted runtime references. When proxy mTLS is configured, runtime-agent normalizes its configured health endpoint, while both Docker-using workers preserve each stored DNS name or node IP and normalize only the legacy TCP port before their bounded endpoint-cache lookup. Historical jobs and command snapshots therefore require no database rewrite. Plaintext Docker endpoints are not changed when proxy TLS is disabled.
The runtime uses the two allowed network operations solely to guarantee the dedicated chronoverse-workloads isolation network exists (bridge driver, inter-container communication disabled) before any workload runs. It revalidates the network immediately before each container creation, recreates a safely pruned network, and fails closed if the network has been replaced with an unsafe configuration.
The execution-worker identity is node-root-equivalent if its private key and token are compromised because Docker container creation is not a tenant security boundary. Role authorization prevents runtime-agent and workflow-worker credentials from creating containers, but workflow cleanup credentials can still read logs and stop/delete any known container ID on a reachable runtime. Per-container authorization needs a purpose-built execution broker. Keep these workers trusted and use isolated or rootless runtime nodes when stronger blast-radius control is required.
Production secrets
Treat generated local material as a development convenience. Production should separately manage private keys, CA lifecycle, database credentials, Meilisearch master key, CSRF HMAC secret, encryption secret, hostnames, and certificate rotation. CRYPTO_SECRET and SERVER_CSRF_HMAC_SECRET must be persistent, distinct values; startup rejects empty or reused values.
For Kubernetes, pre-create chronoverse-server-security, chronoverse-auth, chronoverse-ca, chronoverse-ingress-tls, chronoverse-client-tls, chronoverse-service-tls, chronoverse-infra-tls, chronoverse-kafka-tls, the docker-proxy-ca/docker-proxy-server/three docker-proxy-client-* Secrets, postgres-secret, postgres-app-secret, clickhouse-secret, and meilisearch-secret when you need to bring your own material. postgres-secret is restricted to PostgreSQL and role bootstrap; application workloads use the chronoverse_app non-superuser identity from postgres-app-secret through PgBouncer transaction pooling. Both PostgreSQL Secrets must name the same database; PgBouncer generates its single route from postgres-app-secret.POSTGRES_DB, and setup rejects mismatches or names unsafe for its configuration. The setup script validates complete pre-created server-security Secrets before preserving them. Internal TLS trust-chain and Docker proxy PKI Secrets are treated as atomic sets; partial material is rejected to avoid mixing certificates from different CAs. Ingress TLS is edge-facing and can be managed independently. Otherwise scripts/k8s/setup.sh --mode production generates missing fallback Secrets.
Use scripts/k8s/setup.sh --mode production --context <context> --rotate-docker-proxy-certs for an overlapping-CA Kubernetes rotation after applying current manifests; the standalone scripts/k8s/rotate-docker-proxy-certs.sh performs the same rotation without an apply. For Compose, stop the stack, run init-certs once with -e DOCKER_PROXY_ROTATE_CERTS=true, and recreate the stack. Docker proxy clients mount their isolated role at /docker-proxy-certs, separately from the read-only /certs application-certificate mount; make compose/validate rejects nested read-only mount targets and incomplete role wiring. Both rotations are maintenance operations; the Kubernetes workflow stages old+new trust before replacing clients and server certificates.
The development profile publishes databases and gRPC services for debugging. That topology is not the production security boundary.