Skip to main content

Observability

View Markdown

Temporal records every Workflow Execution as a durable Event History: the inputs, every Activity result, every timer, and every message, in order. Logs and metrics tell you what a process reported while it was running. The Event History lets you replay what a Workflow actually did, including one that finished last week.

Around that history, Temporal gives you three things you would otherwise assemble yourself.

Query Executions by your own identifiers

Visibility indexes running and closed Workflow Executions so you can list, filter, and count them with a SQL-like List Filter. Custom Search Attributes put your own identifiers into that index like, an order Id, customer, region, or tier. An operator can then answer "which orders are stuck in payment retry" from the Web UI or the CLI, without querying your application database.

Visibility is a search index that Temporal updates asynchronously, so a List or Count query can return results that are a few seconds stale. It's built for finding Executions across many, not for reading the authoritative state of one.

Metrics from both sides of the connection

Temporal emits two independent, Prometheus-compatible metric sets, and you need both:

  • SDK metrics come from your Workers and Clients: task slot availability, Schedule-To-Start latency, Activity failures. They tell you whether your Workers are keeping up with the work.
  • Temporal Service metrics come from the Service: request rates, latencies, and errors. On Temporal Cloud these stream from the OpenMetrics endpoint into Datadog, Grafana Cloud, New Relic, or your own Prometheus. Self-hosted deployments scrape Temporal Service metrics directly.

Watching one side alone hides the most common failure mode: the Service is accepting and serving work normally while Tasks pile up in a Task Queue because the Workers are under-provisioned.

Tracing and logs that survive process boundaries

A Workflow Execution can span days, several Activities, and multiple Worker restarts, which is where hand-rolled tracing usually loses the thread. Temporal SDKs ship OpenTelemetry interceptors that propagate trace context from the Client into the Workflow and out to each Activity, across Worker processes, so one trace covers the whole Execution.

SDKs also expose a logger for Workflow and Activity code that suppresses duplicate output during replay, so a retried or resumed Workflow doesn't repeat every log line it already emitted.

What you operate

What you run yourself depends on where the Temporal Service runs:

  • Temporal Cloud. Visibility is enabled by default, and metrics come from the OpenMetrics endpoint. There's no Visibility store to size or upgrade. Both the number of custom Search Attributes and the Visibility API rate limit are scoped per Namespace.
  • Self-hosted. You choose and operate the Visibility store: MySQL v8.0.17+ or PostgreSQL v12+ on Temporal Server v1.20 and later, or Elasticsearch and OpenSearch. Dual Visibility covers migrating between them without downtime.

On both, the Namespace Retention Period bounds how long closed Workflow Executions stay queryable. To keep Event Histories past that window, configure Archival.

Resources

Or jump straight to the SDK feature guide for implementation details: