Avvyland exports OpenTelemetry traces over OTLP to an OTLP-compatible endpoint. The default production target is Grafana Tempo (direct or via Grafana Alloy / an OpenTelemetry Collector).
avvy-worker propagates trace context through:
When tracing is enabled, structured log lines include trace_id and span_id for the major task stages, making it possible to pivot from a trace to a log search and back.
state-server emits OTLP traces for HTTP, WebSocket, and database operations. (Cross-check the state-server's runtime config in internal/infrastructure/tracing/ against the active deployment.)
worker:
observability:
tracing:
enabled: true
service_name: "avvy-worker"
service_version: "dev"
environment: "production"
otlp_traces_endpoint: "https://grafana.avvyland.com/api/otlp/v1/traces"
protocol: "http/protobuf"
sampling_ratio: 1.0
headers:
- "Authorization=Bearer <GRAFANA_OTLP_TOKEN>"
| Variable | Purpose |
|---|---|
TRACING_ENABLED |
Master switch. |
TRACING_SERVICE_NAME, TRACING_SERVICE_VERSION, TRACING_ENVIRONMENT |
Service identity. |
OTLP_ENDPOINT, OTLP_TRACES_ENDPOINT, OTLP_PROTOCOL, OTLP_HEADERS, OTLP_INSECURE |
OTLP target. |
TRACING_SAMPLING_RATIO |
Float [0.0, 1.0]. |
GRAFANA_OTLP_ENABLED, GRAFANA_OTLP_ENDPOINT, GRAFANA_OTLP_TRACES_ENDPOINT, GRAFANA_OTLP_TOKEN |
Convenience vars for Grafana Tempo. If GRAFANA_OTLP_TOKEN is set and no OTLP endpoint is configured, the worker defaults to https://grafana.avvyland.com/api/otlp/v1/traces with Authorization: Bearer <token>. |
Both state-server and avvy-worker emit structured logs. Recommended sinks:
Enrich log lines with trace_id and span_id whenever a trace is active.
Use the standard runtime metrics (Go process, GC, goroutines) plus app-level counters:
Prometheus scraping is the default; Grafana dashboards live alongside the Tempo/Loki stack.
avvy-worker architecture for the trace context plumbing.
Redis task queue for queue depth and retry counters.
Deployed services for the monitoring stack manifest set.
gitlab.avvyland.com/avvy/avvy-worker — README.md (observability section, env vars).
gitlab.avvyland.com/avvy/monitoring — observability stack definitions.