Durable job leases

Job leases provide exclusive, time-bounded execution ownership.

Claim

An execution worker presents the expected workflow generation and dispatch attempt. The jobs service checks current state and returns a lease token only when the job is eligible. For CONTAINER jobs, claim also selects a fresh READY runtime node and returns its Docker endpoint. HEARTBEAT jobs do not receive runtime ownership.

Protected mutations

The token is required to:

• renew the lease;
• attach a container ID with the expected runtime node;
• complete or fail the job;
• cancel a claimed job;
• release it for retry.

Renewal

Workers prove authority with an immediate conditional renewal before starting work from a claim or recovery response, then renew while the workload runs. Renewal never revives an already-expired lease. EXECUTION_WORKER_LEASE_DURATION must remain comfortably longer than EXECUTION_WORKER_LEASE_RENEW_INTERVAL to tolerate transient latency.

Recovery

Execution workers periodically request expired-lease recovery in bounded batches. A replay returns only leases whose token, worker, process owner, status, and database-evaluated expiry are still valid. Recovery includes the job's stored runtime owner and endpoint, so cleanup targets the Docker daemon that created the container. If that runtime is unavailable, recovery releases or fails eligible work through the normal retry policy instead of moving the container lifecycle to a different node.

If execution ends without a confirmed terminal or release response, the worker retains its bounded handoff permit and reconciles the original claim against PostgreSQL. This reconciliation is read-only: it never extends the lease. A live exact authority remains suppressed, while an expired or changed authority releases the permit and leaves the job eligible for normal recovery.

Split-brain prevention

After expiration and recovery, the old token is invalid. A delayed worker cannot report success for work that another worker now owns.