Kafka partitions
Consumer throughput is bounded by partition count. Production defaults use more partitions for jobs and job_logs than for lower-volume workflow and analytics traffic.
Execution workers
Scale concurrency only when registered runtime capacity, Docker CPU, memory, image bandwidth, Kafka partitions, jobs-service latency, and lease-renewal headroom support it.
Execution worker container limits protect the worker process. EXECUTION_WORKER_WORKLOAD_CONTAINER_* limits protect each Docker workload container launched by the worker, and do not apply to image pulls. EXECUTION_WORKER_WORKLOAD_NETWORK names the dedicated Docker network every workload attaches to (default chronoverse-workloads, created on demand with inter-container communication disabled). Custom names must match [A-Za-z0-9][A-Za-z0-9_.-]*; the runtime revalidates the network before every container creation and recreates it after a safe prune.
Execution workers do not need to run on the same node as the Docker daemon. They receive runtime_endpoint from jobs-service and use the direct Docker data plane for image ensure, container start, logs, and cleanup.
Runtime agents
Run one runtime agent beside each Docker-capable daemon. New container claims only use fresh READY runtime rows. Missing heartbeats pause new claims for that runtime, while expired-lease recovery uses the stored job owner to decide whether to clean up or retry.
Kubernetes runtime endpoints are node IPs on mTLS hostPort:2376, not a load-balanced Service. Every labeled node must expose Docker Engine and be routable from every worker. Client caches are bounded to 256 endpoints with 30-minute idle and LRU eviction, so node churn does not grow worker memory without limit. containerd-only clusters are not Docker-workflow runtimes; multi-node kind-like emulators require an explicit cross-node hostPort reachability test.
Role certificates scale with replicas: workflow-worker replicas share the workflow role and execution-worker replicas share the execution role, while each role is mounted separately. This separates privilege classes, not individual pod identities. A compromised execution role remains node-root-equivalent because it can create containers; use isolated/rootless runtime pools when that shared role blast radius is unacceptable.
Workflow workers
Replicas that share a runtime node coordinate identical image pulls through Redis. Different runtime nodes still pull independently. Docker host is used as a fallback when runtime scope is omitted. Build completion records the resolved workflow image digest; execution workers ensure that digest on the assigned runtime node before creating the container.
Outbox relay
Multiple replicas use processing leases. Tune batch size and polling against PostgreSQL and Kafka latency, not in isolation.
Logs
High log volume requires sufficient job_logs partitions, processor replicas, ClickHouse insertion capacity, Meilisearch indexing capacity, and network throughput.
Databases
Review every service's connection pool when increasing replicas. Replica multiplication can exhaust PostgreSQL, ClickHouse, Redis, or Meilisearch before CPU becomes the limiting resource.
Safe sequence
Measure backlog and latency, increase the constraining partition or store capacity, adjust worker replicas and concurrency, then confirm lease and retry behavior under failure.