chronoverse
DocsOperationsObservability

Observability

OpenTelemetry signals and LGTM dashboards.

Chronoverse emits OpenTelemetry traces, metrics, and structured logs to the bundled Grafana OTEL LGTM stack.

Collection

Services and workers export over OTLP gRPC to lgtm:4317. The Grafana UI is available inside the lgtm container on port 3000, but is not host-published by default. Anonymous access is disabled (GF_AUTH_ANONYMOUS_ENABLED=false); authenticate with GF_SECURITY_ADMIN_USER / GF_SECURITY_ADMIN_PASSWORD.

Grafana access

For Kubernetes, use kubectl -n chronoverse port-forward svc/lgtm 3000:3000. For Compose, use the opt-in loopback override:

docker compose -f compose.dev.yaml -f compose.grafana.yaml up -d lgtm

compose.grafana.yaml publishes 127.0.0.1:${GRAFANA_HOST_PORT:-3000}:3000; it does not expose Grafana to the LAN.

For an existing Compose lgtm:/data volume, recreate lgtm from the updated Compose file and reset the database credential from the configured container secret:

docker compose -f compose.dev.yaml up -d --force-recreate --no-deps lgtm
scripts/grafana/reset-admin-password.sh

The helper refuses containers that still mount the obsolete /otel-lgtm application volume. It uses Grafana's supported grafana cli command with the correct image home and data paths, passes the password through standard input, and migrates the stored admin login to GF_SECURITY_ADMIN_USER when needed. If the stored login is not admin, run the helper with GRAFANA_CURRENT_ADMIN_USER=old-login. It then verifies anonymous requests return 401 and the configured credentials return 200. Do not delete the lgtm volume for this migration: /data contains the Grafana database and dashboards, Prometheus metrics, Loki logs, Tempo traces, and Pyroscope profiles.

Traces

Trace context crosses HTTP, gRPC, and Kafka. Use a single trace to follow an API mutation into outbox publication, worker processing, domain RPCs, and follow-up events.

Metrics

Runtime and host metrics complement application spans. HTTP, gRPC, and Redis instrumentation uses the same global OpenTelemetry meter provider and exports through the configured OTLP endpoint.

Endpoint metrics

The HTTP server emits the standard OpenTelemetry histograms:

  • http.server.request.duration
  • http.server.request.body.size
  • http.server.response.body.size

Outbound heartbeat requests emit the corresponding http.client.* metrics. HTTP server metrics use route templates such as /workflows/{workflow_id}; workflow IDs, job IDs, users, URLs, and authentication data are not metric attributes.

All gRPC servers and clients emit:

  • rpc.server.duration and rpc.client.duration
  • rpc.server.request.size and rpc.client.request.size
  • rpc.server.response.size and rpc.client.response.size
  • per-RPC request and response message counts

Useful dimensions include HTTP method, route, and response status, plus gRPC service, method, and status code. Unary and streaming RPCs are both recorded.

Request volume is derived from histogram sample counts rather than a separate request counter. In Grafana's Prometheus data source, the OTEL metrics are normalized to Prometheus names. Example queries include:

# REST requests per second by route and method
sum by (http_route, http_request_method) (
  rate(http_server_request_duration_seconds_count[5m])
)
 
# REST 5xx ratio
sum(rate(http_server_request_duration_seconds_count{http_response_status_code=~"5.."}[5m]))
/
sum(rate(http_server_request_duration_seconds_count[5m]))
 
# REST p95 latency by route
histogram_quantile(
  0.95,
  sum by (le, http_route) (rate(http_server_request_duration_seconds_bucket[5m]))
)
 
# gRPC requests per second by service and method
sum by (rpc_service, rpc_method) (
  rate(rpc_server_duration_milliseconds_count[5m])
)

Watch latency distributions, error ratios, and saturation rather than relying only on process uptime.

Logs

Structured application logs include service context and trace correlation where available. Container job stdout and stderr are a separate product data stream and follow the live/retained log pipeline.

Investigation order

Start from the user-visible request or job ID, locate its trace, inspect failed spans and correlated logs, then confirm durable state in PostgreSQL and transport progress in Kafka.