How long does a worker hold a job?

An execution worker claims a lease and renews it every 30s by default. That lease is what stops a stale worker from finishing work after ownership has moved, which matters when a process is paused, partitioned or restarted mid job.

How long are failed commands remembered?

Command idempotency records are kept for 336h, and the setting cannot go below 168h. It has to cover the longest Kafka, published outbox or manual redrive window, or a retried command could run twice.

What does it need to run?

Docker for local work and Kubernetes for production. Go services and workers sit behind an HTTP and gRPC API, with Kafka for events, PostgreSQL for state, Redis for sessions and live logs, and ClickHouse with Meilisearch for retained search.

MIT. You run it on your own machines, and you keep the data it produces. There is no hosted tier and no account to create.

Self-hosted orchestration

Reliable scheduled work, on infrastructure you control.

Chronoverse is a self-hosted scheduler for heartbeat checks and container workloads. Go services and Kafka workers sit behind it, state lives in PostgreSQL, and every run leaves a trail you can inspect.

Read the docs View source

scripts/k8s/setup.sh --mode production --context <context>

Chronoverse astronaut with an hourglass visor

workflow.buildcompleted

job.leaserenewed · 30s

outbox.publishkafka/jobs

trace.contextpropagated

5gRPC domains

6worker roles

4Kafka topics

1+runtime nodes

Product

One lifecycle from schedule to evidence.

Define work once, run it on a schedule or on demand, and keep the trail you need to explain what happened.

Scheduled workflows

Run interval workloads with generation guards. Build prep runs in the background.

Manual runs

Run jobs now through the same replay-safe path as a scheduled run.

Container execution

Run Docker commands on the owning node, with health checks and a constrained socket proxy.

Heartbeat checks

Run lightweight service checks. They stay apart from container jobs that keep logs.

Live logs

Stream running stdout and stderr to the dashboard over Server-Sent Events.

Retained search

Store logs in ClickHouse, search with Meilisearch, and download safely filtered output.

Analytics

Track workflow and job totals, log volume, and time spent running.

Notifications

Alert when workflow or job state changes. Each alert has a replay-safe ID.

Engineering

Synchronous ownership. Asynchronous progress.

The API stays responsive across gRPC domain boundaries. Kafka workers route slow, distributed, and failure-prone work to the Docker nodes they own.

HTTP gatewaySessions · CSRF · REST · SSE

gRPC domains

usersworkflowsjobsnotificationsanalytics

Kafka workers

schedulerworkflowexecutionjob logsanalyticsoutbox

Runtime data plane

runtime-agentDocker proxyruntime_nodes

infrastructure

PostgreSQLClickHouseRedisKafkaMeilisearchLGTM

Synchronous domain ownership with asynchronous, replay-safe orchestration.

Event flow

Who owns each of the 5 stages, and where does recovery start?

01
Command
The gateway validates session, CSRF state, input, and idempotency.
02
Commit
The service writes state and an outbox event in one transaction.
03
Dispatch
The relay publishes to Kafka and partition workers take ownership.
04
Execute
A worker claims a lease and takes a runtime endpoint, then runs the container and renews the lease.
05
Observe
Logs, notifications, analytics, traces, and terminal state converge.

Reliability model

Failure is represented in state, not hidden behind retries.

Chronoverse assumes messages repeat, processes restart, and ownership expires. Explicit invariants keep it correct at every boundary, and idempotency records outlive the longest redrive window.

Explore replay safety

Idempotency keys

Retry the same command without duplicating a workflow or manual job.

Transactional outbox

Commit domain state and publication intent in one PostgreSQL transaction.

Workflow generations

Reject build, schedule, terminate, and delete work from an old definition.

Durable job leases

Stop stale workers from finishing work after ownership has moved.

Deterministic events

Deduplicate notifications, analytics, and retained logs.

Partition commit policy

Advance Kafka offsets only after the final result is known.

PostgreSQL

Transactional state, idempotency, outbox rows, leases, retries, and analytics.

ClickHouse + Meilisearch

Ordered retained output with low-latency text search and safe highlights.

Redis

Sessions, cached reads, live log delivery, and image pulls scoped to each runtime node.

TLS + OpenTelemetry

mTLS between services, and trace propagation across HTTP, gRPC and Kafka.

Operations

Can you inspect it while it runs?

Startup order comes from health checks. LGTM receives traces, metrics and logs. Recovery loops make abandoned work visible and bounded.

$ kubectl -n chronoverse get deploy,dsoutbox-relay 2/2 availableexecution-worker 2/2 availabledocker-proxy ds runtime-agent sidecars readyjoblogs-processor 2/2 availableserver 2/2 available

MonitoringHealth signals and first response ScalingPartitions, replicas, and capacity TroubleshootingTLS, readiness, logs, and SSE

Documentation

Where does the engineering reference live?

The guides, the OpenAPI contract, the navigation, the link checks and this site are all authored in this repository and deployed together.

Open documentation Browse the API

FAQ

Questions readers ask first.

Short answers, taken from the configuration reference and the source.

Two workflow kinds. A HEARTBEAT workflow is a lightweight service check that produces no logs. A CONTAINER workflow runs a Docker workload and can retain its stdout and stderr. Both can run on an interval or be dispatched by hand, and both go through the same replay-safe lifecycle.