Chronoverse ships Kustomize overlays and an interactive setup script under infra/k8s.
Deployment strategies
scripts/k8s/setup.sh --mode local
scripts/k8s/setup.sh --mode production --context <context>local is a single-node, self-contained validation strategy. It includes PostgreSQL, Redis, ClickHouse, Kafka, Meilisearch, LGTM, hostPath storage, generated local certificate bootstrap jobs, Kafka topic initialization, one replica per application deployment, Nginx, and the Docker socket proxy with a runtime-agent sidecar. After generating certificates on the local PVC, a resource-name-scoped bootstrap Job mirrors only the CA and generic client identity into chronoverse-ca and chronoverse-client-tls Secrets so KEDA can authenticate to Kafka.
production is the self-hosted Chronoverse strategy for your Kubernetes infrastructure. It includes the same infrastructure dependencies in-cluster, plus production-oriented PVCs, HorizontalPodAutoscalers for stateless services, KEDA Kafka-lag scaling for consumers, runtime-agent, Docker proxy, PgBouncer, migrations, and topic initialization.
The script keeps valid, complete operator-provided Secrets and generates only missing bootstrap material. Partial Secrets fail with a clear missing-key error. For chronoverse-server-security, it also rejects reuse of one value for CRYPTO_SECRET and SERVER_CSRF_HMAC_SECRET, and requires CRYPTO_SECRET to be exactly 32 bytes. It also creates docker-proxy-auth (DOCKER_PROXY_TOKEN) and the docker-proxy mTLS material (docker-proxy-ca ca.crt, docker-proxy-server server.pem, and per-client docker-proxy-client-* tls.crt/tls.key for runtime-agent/workflow-worker/execution-worker).
Useful non-interactive options include:
scripts/k8s/setup.sh --mode local --create-kind
scripts/k8s/setup.sh --mode production --context <context> --storage-class <name>
scripts/k8s/setup.sh --mode production --context <context> --skip-apply
scripts/k8s/setup.sh --mode production --context <context> --rotate-docker-proxy-certs--skip-apply prepares prerequisites without applying the overlay. --context keeps every kubectl operation on the named context. Run scripts/k8s/setup.sh --help for the current option list.
--rotate-docker-proxy-certs requires an existing deployment and performs a staged old+new-CA rotation after the manifest apply; it cannot be combined with --dry-run or --skip-apply.
Cluster prerequisites
Kubernetes does not include a generic kubectl command to create a cluster. Create the cluster with your lifecycle tool, such as kind, minikube, kubeadm, or managed Kubernetes provisioning.
The setup workflow requires kubectl and OpenSSL. Production Secret generation also requires Java keytool; --create-kind additionally requires kind. The production overlay needs a default dynamic StorageClass or an explicit --storage-class, and its service HPAs need metrics-server or an equivalent autoscaling/v2 resource-metrics provider.
KEDA is a cluster-level prerequisite for both overlays. Chronoverse does not install or upgrade platform controllers. One supported installation is:
helm repo add kedacore https://kedacore.github.io/charts
helm repo update
helm upgrade --install keda kedacore/keda --version 2.20.2 \
--namespace keda --create-namespace
kubectl label namespace keda chronoverse.io/keda-kafka-access=true --overwrite
kubectl get crd scaledobjects.keda.sh
kubectl get apiservice v1beta1.external.metrics.k8s.ioThe Kafka NetworkPolicy authorizes KEDA through the stable namespace label
chronoverse.io/keda-kafka-access=true, not a hard-coded namespace name. When
KEDA is installed in another platform-managed namespace, apply the same label
there. scripts/k8s/setup.sh discovers that namespace from the external
metrics APIService and stops with an actionable command when the label, KEDA
CRD, external-metrics API, or Docker-node label is absent. It warns when
resource metrics or IngressClass nginx are unavailable. An ingress controller
is needed only for Ingress-based access; port forwarding remains available
without one.
Internal clients, Kafka broker metadata, and KEDA use namespace-qualified
partial Service names such as kafka.chronoverse.svc. They do not assume the
default cluster.local suffix and resolve through the standard Kubernetes Pod
DNS search list. Preserve normal ClusterFirst DNS behavior when customizing
workload DNS settings.
Container workflows require Docker-capable runtime nodes. Label every node that should own Docker containers:
kubectl label node <node-name> chronoverse.io/docker-workloads=trueThose nodes must expose Docker Engine at /var/run/docker.sock. Docker Desktop's built-in docker-desktop Kubernetes context is not sufficient for Docker-backed workers because it does not expose Docker Engine to pods.
For local kind validation:
kind create cluster --name chronoverse --config infra/k8s/overlays/local/kind-cluster.yaml
kubectl config use-context kind-chronoverse
scripts/k8s/setup.sh --mode localSecrets and storage
Production uses Kubernetes Secrets for credentials, server cookie/CSRF secrets, auth keys, CA material, Ingress TLS, service TLS, client mTLS, infrastructure TLS, and Kafka JKS material. PostgreSQL itself and the role-bootstrap Job receive postgres-secret; application pods receive postgres-app-secret for the dedicated chronoverse_app non-superuser role. You can pre-create these Secrets to control credentials and certificates. The setup script creates only missing non-TLS Secrets and never overwrites valid, complete existing Secrets. Invalid local server-security Secrets are replaced with safe local values; invalid production server-security Secrets stop setup so the operator can correct them. Internal production TLS trust-chain Secrets are atomic: provide all of chronoverse-ca, chronoverse-client-tls, chronoverse-service-tls, chronoverse-infra-tls, and chronoverse-kafka-tls, or let setup generate the full fallback set. The PostgreSQL certificate must cover both postgres and postgres-primary; setup rejects an older certificate missing the direct-backend SAN. chronoverse-ingress-tls is edge TLS and can be managed independently. Local generated data-store credentials are deterministic development defaults so retained hostPath data can survive a resource delete/recreate without drifting away from regenerated Secrets; production generated fallback credentials remain random.
postgres-secret and postgres-app-secret must use the same POSTGRES_DB.
Setup derives a missing peer Secret from an existing custom value and rejects a
mismatch. PgBouncer generates its single allowed database mapping from the
application Secret at startup instead of hard-coding chronoverse or exposing
a wildcard route. Database names may contain at most 63 ASCII letters, digits,
or underscores; the reserved name pgbouncer is rejected.
The local strategy uses hostPath PVs for single-node validation. Those PVs use retained node-local paths; delete the kind cluster or clear those paths when you intentionally want an empty local data set. Production uses dynamic PVCs by default; provide --storage-class <name> to the setup script or rely on the cluster default StorageClass. Production requires dynamic storage; use the local strategy for single-node hostPath validation.
Ingress client-IP recovery
Rate limits key on the client address. Behind ingress-nginx, the nginx service would otherwise see the controller address and collapse every client into one shared bucket. The base config therefore recovers the client from X-Forwarded-For through a dedicated nginx-realip-config ConfigMap mounted into the nginx Deployment.
scripts/k8s/setup.sh --mode production resolves the trust ranges, in order: an explicit --realip-cidrs <list> flag (stable, recommended for scalable production — e.g. the kube-controller-manager --cluster-cidr), then auto-detection from the running ingress-nginx controllers. HostNetwork controllers contribute all current node InternalIPs (snapshot — new nodes need a re-run or --realip-cidrs). Pod-network controllers are covered by node pod CIDRs when available (snapshot — new nodes with disjoint CIDRs need a re-run or a stable --realip-cidrs; for static clusters, pass --allow-realip-snapshot to acknowledge), otherwise by ephemeral controller pod IPs (prefer the flag). For direct Kustomize applies, set set_real_ip_from in infra/k8s/overlays/production/nginx-realip.yaml to the source range your ingress-nginx controller connects from, then run kubectl -n chronoverse rollout restart deployment/nginx — ConfigMap volume updates reach the mounted file but nginx does not reload it on its own. Too narrow, and every client shares one bucket; too broad, and in-cluster callers can forge X-Forwarded-For to obtain unlimited fresh buckets. Clusters whose CNI does not populate node pod CIDRs (for example EKS with the Amazon VPC CNI, where pods draw VPC subnet addresses) are covered by the controller-pod fallback or the flag. After applying, setup restarts the nginx Deployment so updated trust ranges take effect immediately.
When a cloud load balancer sits in front of ingress-nginx, the X-Forwarded-For chain gains one more hop. Preserving the client address across that hop is ingress-nginx's own concern (use-forwarded-headers plus its proxy trust configuration); Chronoverse's real_ip block covers only the ingress-nginx-to-nginx hop.
Slow-client limits follow the same split: ingress-nginx terminates public client connections and buffers each request before forwarding a fast controller-to-pod request, so the client_header_timeout 10s and client_body_timeout 15s directives in the nginx service configuration only govern that internal hop and cannot brake external slow clients. Enforce them at the public edge by setting client-header-timeout: "10" and client-body-timeout: "15" in the ingress-nginx controller ConfigMap — these are controller-global settings with no per-Ingress annotation equivalent, and without them the edge keeps the controller's 60-second defaults.
Scaling
The local strategy runs one replica per app deployment. The production strategy includes CPU/memory HorizontalPodAutoscalers for app services. Backend service requests are 100m CPU and 128Mi memory; at a 70% CPU target this represents 70m instead of the unstable 35m target produced by the former 50m request. Limits and HPA maxima remain separate safety ceilings. Install metrics-server or provide equivalent autoscaling/v2 resource metrics before relying on these HPAs.
The five domain gRPC Services are headless. Their DNS records publish ready Pod
IPs so the gRPC clients' round_robin policy can balance calls across resolved
replicas; the HTTP server and dashboard retain normal ClusterIP Services. A
healthy, long-lived gRPC-Go channel may not immediately re-resolve DNS solely
because an HPA added a replica, so proactive endpoint refresh is a separate
scaling concern.
Kafka consumers scale from lag through base KEDA ScaledObject resources. Local maxima follow the base topic partition counts: workflows 2, jobs 4, job logs 4, and analytics 2. Production raises both topic partitions and consumer maxima to 6, 12, 12, and 6 respectively. KEDA cannot usefully exceed the topic's available partitions; CPU/memory HPAs therefore do not compete for ownership of these four consumer Deployments.
PostgreSQL connection capacity
Applications connect to the postgres Service, backed by PgBouncer transaction pooling. PgBouncer reaches the database through the private postgres-primary Service. The database-migration Job bypasses PgBouncer and connects to postgres-primary because golang-migrate relies on session-level PostgreSQL advisory locks, which transaction pooling cannot preserve. Production runs two poolers; each permits 20 normal and 5 reserve backend connections, bounding the pair at 50 of PostgreSQL's 100 slots. The remaining capacity covers bootstrap, migrations, administration, and failure handling.
Client pools are budgeted per workload rather than inheriting ten connections per pod: API services, job-log processing, and outbox use a maximum of four; scheduling, analytics processing, and runtime-agent use two. Idle minima are zero except for one outbox connection. Datastore-client Deployments use maxSurge: 0, so a rollout cannot create an extra wave of stateful dependency clients. Production PostgreSQL requests 500m CPU / 1Gi memory and limits 2 CPU / 2Gi memory.
PgBouncer and PostgreSQL limits are safety boundaries, not a throughput guarantee. Before raising replicas, pool sizes, or partitions, load-test representative transaction duration, pool wait time, PostgreSQL CPU/memory, locks, and active connections.
Controllers and their pod templates carry chronoverse.io/datastore-client plus specific postgres-client, redis-client, kafka-client, clickhouse-client, and meilisearch-client dependency labels for operational inventory. Query them with commands such as kubectl -n chronoverse get pods -l chronoverse.io/kafka-client=true. Every datastore-client Deployment uses maxSurge: 0, preventing a rollout from creating an extra wave of database, cache, search, or broker clients. This does not replace datastore readiness and application retry behavior during a cold cluster start.
Startup behavior
Kubernetes does not provide deployment startup ordering for long-running pods. Chronoverse handles that with cancellable PostgreSQL startup retries using a bounded backoff of roughly 30 seconds, readiness/liveness probes, and bootstrap Jobs for the application database role, migrations, Kafka topics, and local certificate material. The setup entrypoint waits for those Jobs, forces the KEDA Kafka scalers to reconcile after the broker is usable, waits for PgBouncer, rolls the Docker proxy DaemonSet, and then restarts and waits for stateless application Deployments. A successful setup return therefore represents post-bootstrap availability; direct Kustomize users must perform equivalent Job and rollout checks. gRPC services handle SIGTERM, drain for up to 20 seconds, then close database pools and telemetry providers with bounded cleanup.
Docker execution model
Container workflows execute through runtime ownership. Each labeled Docker-capable node runs one docker-proxy DaemonSet pod with a runtime-agent sidecar. Official overlays register an IPv4/IPv6-safe node endpoint on 2376 via hostPort:2376; a load-balanced ClusterIP cannot preserve runtime ownership. Health probes use tcp://127.0.0.1:2376 while the advertised endpoint uses the node IP. The proxy binds :2376 ssl crt /certs/docker-proxy/server.pem ca-file /certs/docker-proxy/ca.crt verify required plus X-Chronoverse-Docker-Proxy-Token and exact role/method/path ACLs. Runtime-agent gets ping/version only; workflow-worker gets image operations and container log/stop/delete cleanup; execution-worker additionally gets container execution and network inspect/create. Unknown certificate subjects and cross-role operations are denied, and workload containers have neither token nor certificate.
The proxy server identity and runtime-agent client identity are separate projected volumes. Workers mount only their own client-role Secret with explicit non-root UID/GID and fsGroup ownership. Go clients reload the projected CA and client keypair on new handshakes and bound endpoint clients with idle/LRU eviction, so certificate rotation and runtime-node churn do not require an unbounded cache.
workflow-worker and execution-worker no longer require Docker-node affinity. They can scale horizontally on any schedulable node and use the runtime endpoint returned by jobs-service for image resolution, execution, logs, termination, deletion, and expired-lease recovery. Runtime capacity is controlled by runtime_nodes.max_concurrency, not worker pod placement.
During an upgrade from plaintext 2375, deploy matching runtime-agent, workflow-worker, and execution-worker images. With proxy mTLS configured, runtime-agent normalizes its configured health endpoint, while both workers preserve the stored runtime host and normalize only a legacy tcp://<host>:2375 endpoint before cache lookup. This covers historical job and idempotency snapshots on both single-node and multi-node clusters without rewriting PostgreSQL. Endpoints remain unchanged when proxy TLS is disabled.
Workers must be able to reach runtime node IPs on TCP 2376. The base NetworkPolicy allows that egress port, but hostPort bypasses NetworkPolicy across CNIs — production must restrict 2376 at the infrastructure layer (node firewall / security group / CNI host policy) in addition to mTLS verify required + token + allowlist. Never expose 2376 publicly.
Multi-node kind and similar Docker-container-based Kubernetes emulators may not route one emulator node's hostPort from pods on another emulator node. If you choose that topology, use a pod-IP runtime endpoint override as an emulator-only workaround. Real single-node and multi-node Kubernetes clusters should use node-stable runtime endpoints.
Single-node and multi-node Kubernetes are supported only when every labeled runtime node exposes Docker Engine at /var/run/docker.sock and workers can route to each labeled node IP on 2376. containerd-only clusters and Docker Desktop's built-in Kubernetes do not provide that Docker Engine contract. The execution-worker identity remains node-root-equivalent if its key and token are compromised because container creation against a host daemon is privileged. The workflow role cannot create, but can read logs and stop/delete a known container ID. Role ACLs reduce service privilege; they do not make Docker a tenant isolation boundary.
Rotate the proxy PKI during a maintenance window:
scripts/k8s/setup.sh --mode production --context <context> \
--rotate-docker-proxy-certsThe workflow first publishes an old+new CA bundle, then rotates the three clients, the server, and finally removes the old trust root with rollouts at each safe boundary. Use scripts/k8s/rotate-docker-proxy-certs.sh directly if no manifest apply is needed. A single-node hostPort DaemonSet has a brief interruption while its pod restarts.
Validation
make k8s/render/local
make k8s/render/production
make k8s/dry-run/local
make k8s/dry-run/production
scripts/k8s/setup.sh --mode local --dry-run
scripts/k8s/setup.sh --mode production --context <context> --dry-runThe render targets work offline. Both kubectl apply --dry-run=client and the setup-script dry runs still use API discovery and therefore require a reachable selected cluster. Full OpenAPI schema validation requires that live cluster or an external validator such as kubeconform.
Direct Kustomize apply remains available after required Secrets, certificates, node labels, storage, public URLs, and ingress values have been prepared:
kubectl apply -k infra/k8s/overlays/local
kubectl apply -k infra/k8s/overlays/productionLocal production smoke access
The production overlay is configured for the Ingress host in SERVER_HOST_URL.
If that value is https://chronoverse.example.com, browser cookies are scoped to
that host and marked secure. A plain port-forward to http://localhost:8080
will load the UI, but signup/login cookies will be rejected by the browser.
For normal local browser validation, use the local strategy. For a temporary
production-overlay smoke test in kind, you may override the server origin to
localhost and restart only the server deployment:
kubectl --context kind-chronoverse -n chronoverse patch configmap server-config \
--type merge \
-p '{"data":{"SERVER_HOST_URL":"http://localhost:8080","SERVER_FRONTEND_URL":"http://localhost:8080","SERVER_ALLOWED_ORIGINS":"http://localhost:8080,"}}'
kubectl --context kind-chronoverse -n chronoverse rollout restart deployment/server
kubectl --context kind-chronoverse -n chronoverse rollout status deployment/server
kubectl --context kind-chronoverse -n chronoverse port-forward svc/nginx 8080:80This is a local-only smoke-test override. Reapply the production overlay, or set the production values to your real HTTPS domain, before using the cluster as a production environment.