How to correlate Mastra metrics, logs, and traces in Grafana

I had a Mastra agent whose Grafana error tile held at 0% while a tool inside it failed on every call. The metric wasn’t lying: the agent caught each thrown exception and answered anyway, so at the agent level nothing had failed. The tool had failed, and the logs showed it; the number everyone watches never did.

No single signal catches that. The failure only surfaces once you correlate the metric with the logs, and a trace ties them to the same request: one turn, two ERROR lines, both under a single trace ID.

This post wires that correlation for a store-free Mastra agent. Traces and logs go through one OpenTelemetry Collector to Tempo and Loki, the Prometheus metrics from the earlier posts stay exactly as they were, and three lines of Grafana datasource config link all three, so a failed turn is a few clicks from the log line that explains it. It ships the correlated dashboard JSON so you can drop it into your own Grafana. It follows two earlier posts, on scraping Mastra into Prometheus and building the metrics dashboard . Both ended by saying traces and logs deserve their own article. This is that article.

Everything runs locally against a demo agent and the open-source exporter : three Docker backends, one host process, no cloud account.

The short version #

You add one exporter and one Collector. The @mastra/otel-exporter emits traces and logs over OTLP to a single OpenTelemetry Collector, which fans them out to Tempo (traces) and Loki (logs). The metrics pillar from the earlier posts stays exactly as it was, feeding Prometheus. Then three lines of Grafana datasource config wire the signals together: a metric exemplar carries a trace_id to Tempo, a Tempo span links to its Loki logs, and a Loki log links back to its trace.

The result is one board where a failed request, or a latency spike, is three clicks from the log line that explains it. The whole stack provisions from config and comes up with docker compose up.

Mastra collects traces and logs with no trace store #

Mastra collects traces and structured logs with only a LibSQLStore configured, no DuckDB or ClickHouse. The metrics posts left one question open: metrics reach Prometheus without an analytics store because the exporter listens on Mastra’s observability bus, so does the same hold for traces and logs? It does.

The reason is the same mechanism. The @mastra/otel-exporter is an exporter on that observability bus, the same interface the store-free metrics exporter uses. Receiving span and log events on the bus is independent of storage. An OLAP store (DuckDB or ClickHouse) is what backs querying stored observability data through Studio or the getMetric* and listLogsVNext APIs; the OTLP export path to Tempo and Loki doesn’t go through it, which is why this ran with a LibSQLStore as the only store.

I verified this the direct way. With new LibSQLStore(...) as the only store, a grep for duckdb or clickhouse across package.json, the observability config, and node_modules returned nothing, and spans still landed in Tempo while log records landed in Loki. “Store-free” here means no observability or OLAP store. A primary LibSQLStore is still configured, as it always is; the point is that the trace and log export path does not depend on it.

The stack: one exporter, one Collector, three backends #

The correlation needs three backends and one hub. Traces go to Tempo, logs to Loki, metrics to Prometheus, and Grafana reads all three. The OpenTelemetry Collector sits in front of Tempo and Loki as the single OTLP endpoint your agent posts to.

The package versions the config below assumes: @mastra/core 1.50.1, @mastra/otel-exporter 1.3.3, @mastra/observability 1.16.0, and both @opentelemetry/exporter-trace-otlp-proto and @opentelemetry/exporter-logs-otlp-proto at 0.220.0. Backend image tags are standard; Tempo 3.0’s config break is called out in the gotchas.

Here is the shape, with the correlation links drawn in:

weather-agent (store-free: LibSQLStore only)
  ├─ metric events ─ mastra-prometheus-exporter ─▶ Prometheus (exemplar-storage on)
  └─ trace + log events ─ @mastra/otel-exporter (http/protobuf)
                            POST /v1/traces  and  /v1/logs
                                     ▼
                          OpenTelemetry Collector
                            ├─ traces ─▶ Tempo
                            └─ logs   ─▶ Loki

Grafana:  metric exemplar trace_id ─▶ Tempo ─▶ Loki logs ─▶ back to Tempo

Wire the agent to emit both signals #

One OtelExporter emits both traces and logs. You configure it with a single endpoint, and Mastra derives the log endpoint from the trace endpoint by swapping /v1/traces for /v1/logs internally. Both signals default on; the signals block is where you would switch one off:

export const otel = new OtelExporter({
  provider: {
    custom: {
      endpoint: process.env.OTEL_ENDPOINT ?? 'http://localhost:4318/v1/traces',
      protocol: 'http/protobuf',       // logs auto-post to /v1/logs
    },
  },
  signals: { traces: true, logs: true },
})

export const mastra = new Mastra({
  storage: new LibSQLStore({ id: 'app', url: 'file:./mastra.db' }),  // no OLAP store
  logger: new PinoLogger({ name: 'weather-agent', level: 'debug' }),
  observability: new Observability({
    configs: { default: {
      serviceName: 'weather-agent',
      exporters: [prom, otel],                          // metrics + OTLP, side by side
      logging: { enabled: true, level: 'info' },        // forwards logger calls onto the bus
    } },
  }),
  agents: { weatherAgent },
})

Two lines carry the weight. logging: { enabled: true } forwards logger calls, at or above logging.level, onto the observability pipeline, where the OTLP exporter receives them and ships them to Loki. (That level is a separate gate from the Pino console logger’s own level.) And the metrics exporter (prom) sits alongside otel in the same exporters array, so keeping the Prometheus pillar means leaving that entry in place.

Logs correlate to traces because a logger call made inside a tool shares that tool’s span. A tool gets its logger from context.mastra.getLogger(), the second argument to execute, not from a module-level closure:

execute: async (inputData, context) => {
  const city = inputData.city ?? 'San Francisco'
  context?.mastra?.getLogger()?.info('weather lookup', { city, tool: 'get-weather' })
  return { city, tempC: 15 + (city.length % 10), conditions: 'Foggy' }
},

That logger call fires inside the tool’s span, so the log record carries the active trace_id and span_id. That shared trace_id is what every cross-link in Grafana uses.

Route both signals through the Collector #

The Collector receives OTLP on one port and fans out to two backends. It runs two pipelines that share the receiver: traces exports to Tempo over gRPC, logs exports to Loki over HTTP. Loki 3.x ingests OTLP natively at /otlp , so the generic otlphttp exporter is all you need, no Loki-specific exporter:

receivers:
  otlp:
    protocols:
      http: { endpoint: 0.0.0.0:4318 }    # Mastra posts /v1/traces + /v1/logs here
      grpc: { endpoint: 0.0.0.0:4317 }
processors:
  batch: { timeout: 2s }
exporters:
  otlp/tempo:    { endpoint: tempo:4317, tls: { insecure: true } }
  otlphttp/loki: { endpoint: http://loki:3100/otlp, tls: { insecure: true } }
service:
  pipelines:
    traces: { receivers: [otlp], processors: [batch], exporters: [otlp/tempo] }
    logs:   { receivers: [otlp], processors: [batch], exporters: [otlphttp/loki] }

Sending straight to Tempo is possible, since Tempo has its own OTLP receiver, but the Collector is the right shape for a system you plan to grow. One endpoint fans out to both backends and gives you the place to add sampling, redaction, or a second destination later.

The Tempo and Loki backends run on near-default configs, with one exception on Tempo covered under what broke below. Prometheus scrapes the exporter exactly as in the metrics dashboard post , with --enable-feature=exemplar-storage added so the exemplar diamonds retain their trace_id.

Connect the three pillars in Grafana #

Three datasource settings turn three separate signals into one correlated view. Each is a jsonData block on a provisioned Grafana datasource, and each wires one direction of the correlation. Prometheus points its exemplars at Tempo; Tempo points its spans at Loki; Loki points its logs back at Tempo:

# Prometheus: a metric exemplar's trace_id opens the trace in Tempo
exemplarTraceIdDestinations:
  - { name: trace_id, datasourceUid: tempo }

# Tempo: a span links to its logs in Loki, filtered to that trace
tracesToLogsV2:
  { datasourceUid: loki, filterByTraceID: true, spanStartTimeShift: '-1h', spanEndTimeShift: '1h' }

# Loki: a log line's trace_id links back to Tempo
derivedFields:
  - { name: trace_id, matcherType: label, matcherRegex: trace_id, datasourceUid: tempo, url: '${__value.raw}' }

One detail is worth knowing before you wire the Loki side. Trace context appears in Loki under two names: the OTLP-native trace_id and span_id, and Mastra’s own mastra_traceId and mastra_spanId attributes. They hold the same value. Key the derived field on trace_id, and use matcherType: label, which matches structured metadata as well as indexed labels.

From a failed turn to the log line that explains it #

Start where an operator starts: something looks wrong. The correlated dashboard has three rows: golden signals (request rate, p95 latency, agent error percentage, target up), a metrics-to-traces row where the latency panel renders exemplar diamonds, and a logs-to-traces row with a recent-traces table and a Loki logs panel. The three datasource links from the previous section are the whole mechanism, and they share one key, the trace_id that appears in every signal.

Take the failed turn, trace 6cf6411e. Its trace ID opens the waterfall in Tempo: a GenAI-semantic-convention tree of invoke_agent weather-agent, then chat qwen/..., then the model spans and the two execute_tool spans flagged red, where the tool threw.

Every span has a Logs button, the Tempo-to-Loki link, that filters Loki to that one trace. That is where the failure becomes legible: the single turn produced two correlated ERROR lines under one trace ID, incident: payments probe failed and incident check failed, both from the throwing tool. You reach them by trace ID, without searching the whole log stream or working out which lines belong to which request.

The links work from the metrics side too, which is where a latency investigation begins. A p95 exemplar diamond carries its own trace_id and jumps straight to the trace. Prometheus stored dozens of exemplars on the agent duration histogram in my run, which needs --enable-feature=exemplar-storage to retain them. Whichever signal you start from, you land on the same request.

Mastra’s spans follow the OpenTelemetry GenAI semantic conventions (v1.38.0) out of the box: span names like invoke_agent weather-agent, chat qwen/..., and execute_tool weatherTool, with attributes like gen_ai.operation.name, gen_ai.request.model, and gen_ai.usage.input_tokens (the traces I captured also included gen_ai.tool.* on the tool spans). Any OTLP-compliant backend accepts them without custom mapping, because the schema is the open GenAI semantic-convention standard rather than anything Tempo-specific. I only exercised Tempo and Loki here.

What broke, and the fixes #

Four things cost me time. The first three would cost any reader the same, so they are worth naming.

Tempo 3.0 dropped the top-level ingester and compactor config blocks. The quickstart config still all over the web fails to start on grafana/tempo:3.0.0 with field ingester not found in type app.Config. Tempo 3.0 restructured its config. A minimal server, distributor (the OTLP receivers), and storage is all a lab needs; sensible defaults cover the rest.

Log export needs both the trace and the log OTLP protocol packages, or logs vanish with no error. For http/protobuf you install @opentelemetry/exporter-trace-otlp-proto and @opentelemetry/exporter-logs-otlp-proto. From the Mastra docs , verbatim: “If the matching log exporter package isn’t installed, log export is silently disabled and traces continue to work.” Install only the trace package and you get a healthy trace pipeline and zero logs, with nothing to tell you why. If your traces show up in Tempo but Loki stays empty, check this first.

The OTLP severity arrives uppercased, so filter on severity_text="ERROR", not lowercase. Loki stores the OTLP severity_text verbatim as structured metadata, preserving its case exactly as the exporter sent it (confirmed live against this stack). This setup’s OTel exporter emits severities uppercase (the common instrumentation convention), so {service_name="weather-agent"} | severity_text="ERROR" returns the error lines, while the lowercase error silently returns zero rows and a reader filtering by level concludes no errors exist. The casing comes from the emitter, not from Loki.

The fourth is not a bug, but it surprised me the first time: OTLP log attributes show up in Loki’s raw result view as a dozen fields on the stream, which looks like a cardinality problem. It isn’t. GET /loki/api/v1/labels returns only service_name for this agent. Loki promotes a small curated set of resource attributes to index labels (service, deployment, cloud, and Kubernetes identifiers), and this agent sets only service.name; everything else is structured metadata, which is queryable and low-cost, not an index label. Loki 3.x does the right thing by default.

What the correlation shows that a single pillar hides #

This is the failure from the top of the post, now explained. When a tool throws on every call, the agent catches it and answers, so the agent’s span and metric read status="ok". The tool span carries status="error", and an error-level log fires. On the board, the “agent error %” tile sits at 0% while ERROR lines stream in the logs panel below it. No single pillar shows both halves; the correlation does.

The metric alone tells you the agent is healthy. The metric plus the logs, correlated, tell you a tool is failing every call and the agent is papering over it. Watch tool status and logs, not the top-line agent metric alone.

Common questions #

What I would do Monday morning #

If you already run Mastra agents with the Prometheus exporter, add the OTLP exporter and one Collector, and you have traces and logs correlated to your existing metrics. It is one docker compose up and two npm packages away. Turn on signals: { traces: true, logs: true }, install both OTLP protocol packages so logs don’t vanish, and provision the three Grafana datasource links so a failing request is a click from its trace and its logs.

Then remember what the correlation is for: the agent error tile can read 0% while a tool fails every call, and the logs panel next to it is where you see the truth. Download the correlated dashboard JSON, change the service name, and you are watching your own agent across all three pillars. The exporter is at github.com/charlesgwilson/mastra-prometheus-exporter .

Practical AI #

I co-contribute to Practical AI , Damian Galarza’s newsletter for builders working with AI.

It covers the patterns, tradeoffs, and lessons that show up when AI moves from prototype to real work.