[{"content":"Experienced CTO and CIO, and a founding engineer on FDA 510(k)-cleared medical devices. I write about the engineering work that makes agents reliable enough for real environments.\n","date":null,"permalink":"https://imawilson.com/","section":"Greg Wilson","summary":"\u003cp\u003eExperienced CTO and CIO, and a founding engineer on FDA 510(k)-cleared medical devices. I write about the engineering work that makes agents reliable enough for real environments.\u003c/p\u003e","title":"Greg Wilson"},{"content":"When a Grafana alert fires, a Mastra agent reads the metric, pulls the failing trace, follows the stack to the bug at orders.js:23, and posts a cited file-or-close recommendation to Slack. Then it stops. It changes nothing, and a human makes the call. This is the first step of an SRE agent I\u0026rsquo;m building, read-only to start. The post walks through the whole loop and what I learned wiring it up. The code is public, so you can clone it and build it yourself .\nWhat read-only buys you #The slow part of an incident is assembling the evidence, and that is what the agent collapses. The metric says something is wrong, the trace says which request, the stack trace says which line, and the logs say how often. On a bad night you gather those by hand across four tools while the channel fills up. The agent gathers them in one pass and writes down what they mean, with links, before you have finished reading the page. Nothing is suppressed silently. The raw alerts still exist, and every recommendation is a hypothesis with receipts I can audit.\nStarting read-only is already teaching me to separate two things we usually collapse into one, how freely the agent may investigate and what it may act on. This step grants the first and withholds the second. My instinct going in was to reach for action first, and watching the agent investigate is showing me why that is worth resisting.\nThe demo stack #Everything here runs on a self-contained demo stack, so the whole loop is reproducible and no real system is involved. One docker-compose stack holds the telemetry and the alerting. The demo webapp (the subject) and the triage agent both run as host processes beside it. It is all in the companion repo . Clone it, follow the README, and you get the same loop firing on your machine. Every code block below is an excerpt linking that repo at the v0.1.0 tag. The observability half (Grafana, Tempo, Loki, Prometheus) is the same stack the earlier posts stood up across exporting Mastra metrics to Prometheus , the metrics dashboard , and correlating metrics, logs, and traces .\nAn instrumented demo webapp, a small Express service with OpenTelemetry traces and logs and a prom-client metrics endpoint, plus a traffic-and-fault generator. It has a planted bug. computeTotal in orders.js sums item.price * item.qty with no null-check, so an undefined line item throws TypeError: Cannot read properties of undefined (reading 'price'). The handler records the exception on the span, so a failing trace names orders.js:23. That is the thread the agent pulls. Its own Grafana, Tempo, Loki, and Prometheus (plus Alloy as the OTLP hub), with datasource correlation provisioned so the alert-to-trace-to-logs join resolves. Grafana-managed alerting, provisioned as YAML, a rule on the error ratio, a webhook contact point, and a notification policy. A Grafana-managed alert rule routes through Grafana\u0026rsquo;s own built-in Alertmanager to its contact point, so the demo needs no separately-deployed external Alertmanager (confirmed live, with the rule firing and Grafana POSTing straight to the webhook and no external Alertmanager running in the stack). The agent\u0026rsquo;s read surface is the Grafana MCP , a Model Context Protocol (MCP) server that exposes Grafana\u0026rsquo;s datasources as agent tools (mcp-grafana, over stdio through Mastra\u0026rsquo;s MCPClient, on a read-only Viewer service account), for Prometheus metrics and Loki logs. Tempo traces and the source read go direct, because mcp-grafana exposes no Tempo trace-query tool. The load-bearing versions are Node 22.23.1, @mastra/core 1.51.0, @mastra/mcp 1.14.0 (the MCP client), mcp-grafana v0.17.0, Grafana 13.1.0, Tempo 3.0.2, Loki 3.7.3, Prometheus 3.13.0, Alloy 1.17.1.\nflowchart LR A[\"demo webapp(orders.js:23 bug)\"] --\u003e|OTLP| Al[\"Alloy(OTLP hub)\"] Al --\u003e T[\"Tempo(traces)\"] Al --\u003e Lo[\"Loki(logs)\"] A --\u003e|/metrics| C[\"Prometheus\"] C --\u003e G[\"Grafanaalert: WebappErrorRatioHigh\"] G --\u003e|\"webhook POST (Bearer)\"| M[\"Mastra server/webhooks/grafana\"] M --\u003e S[\"seed: metric + error trace to file:line\"] S --\u003e L[\"agent ReAct loop4 read-only tools\"] L --\u003e Gate[\"citation gate(deterministic)\"] Gate --\u003e Slack[\"Slack card\"] L -.-\u003e|\"metrics, logs\"| MCP[\"Grafana MCP(read-only SA)\"] MCP -.-\u003e|\"metrics\"| C MCP -.-\u003e|\"logs\"| Lo L -.-\u003e|\"traces, direct HTTP\"| T L -.-\u003e|\"source\"| A The seed the agent starts from #The agent does not start from a blank slate. A deterministic seed runs first, because 2 facts are cheap to establish without a model and expensive to get wrong. It queries Prometheus for the error ratio, confirms the breach, searches Tempo for a recent error trace, and parses the exception out of it:\nSEED (deterministic, before the model runs): metric: error_ratio_5xx = 0.6000000000000001 (threshold 0.1) trace: 2986e91dfa8644013024391c8a0400 { resource.service.name = \u0026#34;demo-webapp\u0026#34; \u0026amp;\u0026amp; status = error } exception: TypeError: Cannot read properties of undefined (reading \u0026#39;price\u0026#39;) -\u0026gt; orders.js:23 The seed already names the failing frame (the exact file and line the exception\u0026rsquo;s stack trace blames), so the model is not finding the root cause from scratch. It corroborates that starting point, judges whether the alert is noise or actionable, and gathers the context a human would want attached. Handing the model a grounded start is what makes this step reliable. It is also why the model you pick matters less than you would expect, and I come back to it in a later post.\nFrom there the agent drives its own investigation with 4 tools, and every one is a read. The full implementations, including the guard that bounds them, are in tools.ts and backends.ts .\nTool Reads What the agent gets get_baseline(metric, window) Prometheus, via the Grafana MCP the metric\u0026rsquo;s value one window back, to compare against now get_more_logs(window, filter) Loki, via the Grafana MCP error and context log lines, for a window and filter it chooses get_neighboring_traces(onlyErrors, limit) Tempo, direct HTTP recent traces around the incident, to judge blast radius read_source(file, line) the webapp checkout, scoped the source snippet the failing trace points at The Grafana MCP runs on a read-only Viewer service account, so read-only is enforced by the account\u0026rsquo;s role, not by my code. A write call returns 403 no matter what the agent asks. The one tool that touches the filesystem, read_source , resolves real paths and refuses anything outside the webapp\u0026rsquo;s checkout.\nRoute the alert to the agent #The Mastra server is the webhook receiver. registerApiRoute adds a POST route, and a middleware checks a shared-secret bearer token before the handler runs, so an unauthenticated POST gets a 401. The handler acks immediately, then runs the triage off the request path in a detached task (the full route , with persistence and logging elided below):\nhandler: async (c) =\u0026gt; { let payload: any try { payload = await c.req.json() } catch { return c.json({ error: \u0026#39;invalid payload\u0026#39; }, 400) } if (payload == null || typeof payload !== \u0026#39;object\u0026#39; || Array.isArray(payload)) return c.json({ error: \u0026#39;invalid payload\u0026#39; }, 400) const service = payload?.commonLabels?.service || payload?.alerts?.[0]?.labels?.service || \u0026#39;demo-webapp\u0026#39; if (typeof service !== \u0026#39;string\u0026#39; || service.length \u0026gt; 200 || !SERVICE_RE.test(service)) return c.json({ error: \u0026#39;invalid service\u0026#39; }, 400) if (payload?.status !== \u0026#39;firing\u0026#39;) return c.json({ received: true, skipped: `status=${payload?.status}` }) void (async () =\u0026gt; { // ack now, investigate async (try/catch + persistence writes elided) const s = await seed(service) if (s.metric.value == null || !s.trigger.traceId) return // don\u0026#39;t triage against nothing const inv = await runInvestigation(s) const triage = await structure(inv.text) const gate = citationGate(inv.ledger, triage) const card = renderCard(inv.ledger, triage, gate) await maybePostSlack(card) // posts the card; never touches prod })() return c.json({ received: true, status: \u0026#39;processing\u0026#39; }) }, renderCard formats the Slack card and maybePostSlack posts it, logging to stdout when no Slack token is set.\nTwo boundary checks matter here. The bearer check uses crypto.timingSafeEqual, so a wrong token can\u0026rsquo;t be recovered a byte at a time from the response timing. The service label is charset-restricted at the boundary before it reaches a TraceQL (Tempo\u0026rsquo;s trace query language) or LogQL (Loki\u0026rsquo;s log query language) query.\nThat bearer check guards the webhook route only, not the rest of the Mastra API on the same port, which includes the Studio playground and its other routes. So keep the server bound locally.\nAcking before you investigate matters, and I did not do it at first. Grafana\u0026rsquo;s webhook notifier has a short timeout, and the gather-plus-model pass outran it. Grafana marked the notification failed and retried on every evaluation, which re-triggered the agent. The fix is to return a 200 immediately, then run the triage in a detached async task.\nOn the Grafana side, the contact point is provisioned as YAML , a webhook with a Bearer scheme :\n# observability/grafana/provisioning/alerting/contactpoints.yaml apiVersion: 1 contactPoints: - orgId: 1 name: mastra-webhook receivers: - uid: mastra-webhook type: webhook disableResolveMessage: false settings: url: http://host.docker.internal:14111/webhooks/grafana httpMethod: POST authorization_scheme: Bearer authorization_credentials: $GRAFANA_WEBHOOK_SECRET Grafana interpolates $GRAFANA_WEBHOOK_SECRET from the container\u0026rsquo;s environment when it loads the provisioning file, so the secret stays out of the YAML.\nLet the agent drive the investigation #runInvestigation is a bounded ReAct loop (the agent reasons, then acts, one step at a time), not a fixed script. The agent gets one turn of user content, the seed rendered as a prompt. Its investigation strategy lives in the system INSTRUCTIONS , and within that it decides which tools to call and when to stop. It calls agent.generate with a step cap and a per-step callback that records each step\u0026rsquo;s tools:\nconst result = await agent.generate(seedPrompt, { maxSteps, onStepFinish: ({ toolCalls, finishReason }: StepFinish) =\u0026gt; { stepLog.push({ step: stepLog.length + 1, tools: (toolCalls ?? []).map((call) =\u0026gt; call.toolName ?? call.payload?.toolName ?? \u0026#39;unknown\u0026#39;), finishReason: finishReason ?? \u0026#39;unknown\u0026#39;, }) }, }) The step log is the receipt that the path is the agent\u0026rsquo;s, not mine. On the shown run the frontier model took 3 steps and 5 tool calls, and it did what a fixed evidence bundle never does. In one turn it fetched a baseline from a window back, read the source at the exception\u0026rsquo;s line, and pulled a first round of neighboring traces, then went back for a second pass to size the blast radius, 16 in all.\nstep 1: get_baseline, read_source, get_more_logs, get_neighboring_traces (finish: tool-calls) step 2: get_neighboring_traces (finish: tool-calls) step 3: (none) (finish: stop) -\u0026gt; 5 tool calls; gathered 2 metric points, 12 logs, 16 traces, 1 source read Freedom to explore needs a floor, so each tool guards itself in code , with a per-tool call cap and a stop on the third identical query. Mastra also ships Code Mode , where the agent writes one program that calls the tools instead of calling them one at a time. For a read-only step I want the narrowest blast radius, so I kept four constrained tools behind a deterministic guard. Letting the model write code that runs in a sandbox is a later step, once that sandbox and its safety story are worked out.\nGate every citation, then render the card #The agent\u0026rsquo;s freedom to investigate is balanced by a deterministic gate on its output. Before a card renders, every citation is checked against the evidence ledger the tools wrote. A source citation clears two checks. It must name a file the agent read (existence), and it must be the frame the exception points at (support):\n// existence: the ref must name a source a tool actually read, by path const source = ledger.sources.find((s) =\u0026gt; ref.includes(norm(s.path))) if (!source) { existenceIssues.push(`source \u0026#34;${citation.ref}\u0026#34; was never read by a tool (fabricated or unread)`) continue } // support: the cited source must be the failing frame (basename + line), not just any read file if (!citesFailingFrame(ledger, ref)) { semanticIssues.push( `source \u0026#34;${citation.ref}\u0026#34; was read but is not the failing frame ` + `(exception points at ${failingFile}:${failingLine})`, ) } // further down, an actionable disposition needs a source citation that survives BOTH checks Two negative controls, run against the gate directly, show it earns its place. Feed it a citation to billing.js:99, a file no tool read, and existence rejects it. Feed it server.js:30, a real source line seeded into the ledger as read but not the frame the exception points at, and it clears existence and is rejected on support.\nThe card labels that second check \u0026ldquo;semantic support,\u0026rdquo; which oversells it. It is not the agent re-reading the span and judging relevance. It is a deterministic match against the failing frame the seed already parsed from the trace, orders.js:23. That makes it a strong guard when the seed is right, and no guard at all when the seed is wrong. A later post takes that limit apart. Here it is enough that the gate is deterministic and conditional on the seed, not a model grading its own homework.\nThe output is the Slack card. It carries the seed\u0026rsquo;s breach metric, 0.60, now surrounded by the baseline, logs, traces, and source read the agent gathered to back the call. Here is the shown run, gate passed:\n*Alert triage — demo-webapp* 🚨 *Disposition:* actionable | *Recommendation:* file_ticket | *Confidence:* high *Root cause hypothesis:* orders.js:23 attempts to read the \u0026#39;price\u0026#39; property from items in an order array without null-checking. When the array contains undefined elements, the code throws \u0026#34;TypeError: Cannot read properties of undefined (reading \u0026#39;price\u0026#39;)\u0026#34;, causing checkout failures at a 60% error rate. *Investigation:* 5 tool calls (get_baseline, read_source, get_more_logs, get_neighboring_traces, get_neighboring_traces) *Evidence gathered:* 2 metric points · 12 logs · 16 traces · 1 source reads *Citations:* `metric`: error_ratio_5xx = 0.6000000000000001 (threshold 0.1) · `source`: orders.js:23 · `log`: checkout failed: Cannot read properties of undefined (reading \u0026#39;price\u0026#39;) · `trace`: 2986e91dfa8644013024391c8a0400 *Citation gate:* ✅ passed (existence + semantic support) _Read-only recommendation. A human decides._ An operator reading that at 3 a.m. does not start from zero. They start from a named line, a cited trace, a blast-radius estimate, and a call they can audit or overrule. Then they decide what to do, which was never the agent\u0026rsquo;s to make.\nWhat broke, and the fixes #Two Mastra-specific things cost me time, and they will cost you the same.\nA structured answer took two tries. I wanted the model to return a filled-in card, not free text. My first try used the output option. It came back empty, and the code that expected data crashed. The one that works is structuredOutput . There is one more catch. onStepFinish (the hook that lets me watch each step the agent takes) only fires in plain-text mode, not with structured output. So the agent investigates in text, and a cheap second pass turns it into the card.\nThe agent couldn\u0026rsquo;t find the code to read. The source-reading tool is scoped to one folder, set by WEBAPP_SRC, and its default (./demo-webapp) is resolved from the process\u0026rsquo;s working directory. The dev server runs somewhere other than the project root, so that resolved to nowhere and the tool read nothing. With no source to cite, the gate blocked the verdict. It won\u0026rsquo;t pass an \u0026ldquo;act on this\u0026rdquo; call with no source behind it. So the bug surfaced as a blocked answer, not a wrong one. The fix is to set WEBAPP_SRC to an absolute path in .env, then restart, since the server reads it only at startup.\nCommon questions # How do you send a Grafana alert to an AI agent? Use a Grafana-managed alert rule with a webhook contact point pointed at an HTTP endpoint your agent exposes. Grafana\u0026rsquo;s alerting ships its own built-in Alertmanager, so you don\u0026rsquo;t stand up a separate external one, and when the rule fires it POSTs the alert payload (status: \u0026quot;firing\u0026quot;) to the URL. Secure it with a bearer token on the contact point and check that token in the receiver before doing any work. Should an AI triage agent be allowed to act on alerts? Eventually, behind the right guardrails, but not yet in this build. I don\u0026rsquo;t yet know which alerts are reliably safe to auto-resolve or what a wrong call would cost, so read-only lets me watch the agent triage and build that judgment first. Acting behind an explicit approval gate is a later step in the series. What I would do Monday morning #If you run on-call with Grafana alerting, stand up the investigate-only step before anything that acts. Point a firing alert at a webhook, seed the investigation from the metric and the failing trace, give the agent a few tools that only read, and gate every citation before it posts. You get a grounded, auditable hypothesis attached to the page, and every action stays in human hands.\nThe reason to start read-only is what it teaches you. You cannot write the runbook for an SRE agent until you have watched one triage without touching anything. So clone the repo , fire the demo alert, and try it on your own cases, a noisy alert, a wrong seed, a different model, and watch where the gate holds and where it slips. The next post in the series works through what the agent has to read before its recommendations deserve trust.\nPractical AI #I co-contribute to Practical AI , Damian Galarza\u0026rsquo;s newsletter for builders working with AI.\nIt covers the patterns, tradeoffs, and lessons that show up when AI moves from prototype to real work.\n","date":"July 24, 2026","permalink":"https://imawilson.com/posts/route-a-grafana-alert-to-a-mastra-triage-agent/","section":"Writing","summary":"\u003cp\u003eWhen a \u003ca href=\"https://grafana.com/oss/grafana/\" target=\"_blank\" rel=\"nofollow noopener noreferrer\"\u003eGrafana\u003c/a\u003e\n alert fires, a \u003ca href=\"https://mastra.ai\" target=\"_blank\" rel=\"nofollow noopener noreferrer\"\u003eMastra\u003c/a\u003e\n agent reads the metric, pulls the failing trace, follows the stack to the bug at \u003ccode\u003eorders.js:23\u003c/code\u003e, and posts a cited file-or-close recommendation to Slack. Then it stops. It changes nothing, and a human makes the call. This is the first step of an SRE agent I\u0026rsquo;m building, read-only to start. The post walks through the whole loop and what I learned wiring it up. The code is public, so you can \u003ca href=\"https://github.com/charlesgwilson/mastra-alert-triage-agent\" target=\"_blank\" rel=\"nofollow noopener noreferrer\"\u003eclone it and build it yourself\u003c/a\u003e\n.\u003c/p\u003e","title":"Route a Grafana alert to a read-only Mastra triage agent"},{"content":"","date":null,"permalink":"https://imawilson.com/posts/","section":"Writing","summary":"","title":"Writing"},{"content":"I had a Mastra agent whose Grafana error tile held at 0% while a tool inside it failed on every call. The metric wasn\u0026rsquo;t lying: the agent caught each thrown exception and answered anyway, so at the agent level nothing had failed. The tool had failed, and the logs showed it; the number everyone watches never did.\nNo single signal catches that. The failure only surfaces once you correlate the metric with the logs, and a trace ties them to the same request: one turn, two ERROR lines, both under a single trace ID.\nThis post wires that correlation for a store-free Mastra agent. Traces and logs go through one OpenTelemetry Collector to Tempo and Loki, the Prometheus metrics from the earlier posts stay exactly as they were, and three lines of Grafana datasource config link all three, so a failed turn is a few clicks from the log line that explains it. It ships the correlated dashboard JSON so you can drop it into your own Grafana. It follows two earlier posts, on scraping Mastra into Prometheus and building the metrics dashboard . Both ended by saying traces and logs deserve their own article. This is that article.\nEverything runs locally against a demo agent and the open-source exporter : three Docker backends, one host process, no cloud account.\nThe short version #You add one exporter and one Collector. The @mastra/otel-exporter emits traces and logs over OTLP to a single OpenTelemetry Collector, which fans them out to Tempo (traces) and Loki (logs). The metrics pillar from the earlier posts stays exactly as it was, feeding Prometheus. Then three lines of Grafana datasource config wire the signals together: a metric exemplar carries a trace_id to Tempo, a Tempo span links to its Loki logs, and a Loki log links back to its trace.\nThe result is one board where a failed request, or a latency spike, is three clicks from the log line that explains it. The whole stack provisions from config and comes up with docker compose up.\nMastra collects traces and logs with no trace store #Mastra collects traces and structured logs with only a LibSQLStore configured, no DuckDB or ClickHouse. The metrics posts left one question open: metrics reach Prometheus without an analytics store because the exporter listens on Mastra\u0026rsquo;s observability bus, so does the same hold for traces and logs? It does.\nThe reason is the same mechanism. The @mastra/otel-exporter is an exporter on that observability bus, the same interface the store-free metrics exporter uses. Receiving span and log events on the bus is independent of storage. An OLAP store (DuckDB or ClickHouse) is what backs querying stored observability data through Studio or the getMetric* and listLogsVNext APIs; the OTLP export path to Tempo and Loki doesn\u0026rsquo;t go through it, which is why this ran with a LibSQLStore as the only store.\nI verified this the direct way. With new LibSQLStore(...) as the only store, a grep for duckdb or clickhouse across package.json, the observability config, and node_modules returned nothing, and spans still landed in Tempo while log records landed in Loki. \u0026ldquo;Store-free\u0026rdquo; here means no observability or OLAP store. A primary LibSQLStore is still configured, as it always is; the point is that the trace and log export path does not depend on it.\nThe stack: one exporter, one Collector, three backends #The correlation needs three backends and one hub. Traces go to Tempo, logs to Loki, metrics to Prometheus, and Grafana reads all three. The OpenTelemetry Collector sits in front of Tempo and Loki as the single OTLP endpoint your agent posts to.\nThe package versions the config below assumes: @mastra/core 1.50.1, @mastra/otel-exporter 1.3.3, @mastra/observability 1.16.0, and both @opentelemetry/exporter-trace-otlp-proto and @opentelemetry/exporter-logs-otlp-proto at 0.220.0. Backend image tags are standard; Tempo 3.0\u0026rsquo;s config break is called out in the gotchas.\nHere is the shape, with the correlation links drawn in:\nweather-agent (store-free: LibSQLStore only) ├─ metric events ─ mastra-prometheus-exporter ─▶ Prometheus (exemplar-storage on) └─ trace + log events ─ @mastra/otel-exporter (http/protobuf) POST /v1/traces and /v1/logs ▼ OpenTelemetry Collector ├─ traces ─▶ Tempo └─ logs ─▶ Loki Grafana: metric exemplar trace_id ─▶ Tempo ─▶ Loki logs ─▶ back to Tempo Wire the agent to emit both signals #One OtelExporter emits both traces and logs. You configure it with a single endpoint, and Mastra derives the log endpoint from the trace endpoint by swapping /v1/traces for /v1/logs internally. Both signals default on; the signals block is where you would switch one off:\nexport const otel = new OtelExporter({ provider: { custom: { endpoint: process.env.OTEL_ENDPOINT ?? \u0026#39;http://localhost:4318/v1/traces\u0026#39;, protocol: \u0026#39;http/protobuf\u0026#39;, // logs auto-post to /v1/logs }, }, signals: { traces: true, logs: true }, }) export const mastra = new Mastra({ storage: new LibSQLStore({ id: \u0026#39;app\u0026#39;, url: \u0026#39;file:./mastra.db\u0026#39; }), // no OLAP store logger: new PinoLogger({ name: \u0026#39;weather-agent\u0026#39;, level: \u0026#39;debug\u0026#39; }), observability: new Observability({ configs: { default: { serviceName: \u0026#39;weather-agent\u0026#39;, exporters: [prom, otel], // metrics + OTLP, side by side logging: { enabled: true, level: \u0026#39;info\u0026#39; }, // forwards logger calls onto the bus } }, }), agents: { weatherAgent }, }) Two lines carry the weight. logging: { enabled: true } forwards logger calls, at or above logging.level, onto the observability pipeline, where the OTLP exporter receives them and ships them to Loki. (That level is a separate gate from the Pino console logger\u0026rsquo;s own level.) And the metrics exporter (prom) sits alongside otel in the same exporters array, so keeping the Prometheus pillar means leaving that entry in place.\nLogs correlate to traces because a logger call made inside a tool shares that tool\u0026rsquo;s span. A tool gets its logger from context.mastra.getLogger(), the second argument to execute, not from a module-level closure:\nexecute: async (inputData, context) =\u0026gt; { const city = inputData.city ?? \u0026#39;San Francisco\u0026#39; context?.mastra?.getLogger()?.info(\u0026#39;weather lookup\u0026#39;, { city, tool: \u0026#39;get-weather\u0026#39; }) return { city, tempC: 15 + (city.length % 10), conditions: \u0026#39;Foggy\u0026#39; } }, That logger call fires inside the tool\u0026rsquo;s span, so the log record carries the active trace_id and span_id. That shared trace_id is what every cross-link in Grafana uses.\nRoute both signals through the Collector #The Collector receives OTLP on one port and fans out to two backends. It runs two pipelines that share the receiver: traces exports to Tempo over gRPC, logs exports to Loki over HTTP. Loki 3.x ingests OTLP natively at /otlp , so the generic otlphttp exporter is all you need, no Loki-specific exporter:\nreceivers: otlp: protocols: http: { endpoint: 0.0.0.0:4318 } # Mastra posts /v1/traces + /v1/logs here grpc: { endpoint: 0.0.0.0:4317 } processors: batch: { timeout: 2s } exporters: otlp/tempo: { endpoint: tempo:4317, tls: { insecure: true } } otlphttp/loki: { endpoint: http://loki:3100/otlp, tls: { insecure: true } } service: pipelines: traces: { receivers: [otlp], processors: [batch], exporters: [otlp/tempo] } logs: { receivers: [otlp], processors: [batch], exporters: [otlphttp/loki] } Sending straight to Tempo is possible, since Tempo has its own OTLP receiver, but the Collector is the right shape for a system you plan to grow. One endpoint fans out to both backends and gives you the place to add sampling, redaction, or a second destination later.\nThe Tempo and Loki backends run on near-default configs, with one exception on Tempo covered under what broke below. Prometheus scrapes the exporter exactly as in the metrics dashboard post , with --enable-feature=exemplar-storage added so the exemplar diamonds retain their trace_id.\nConnect the three pillars in Grafana #Three datasource settings turn three separate signals into one correlated view. Each is a jsonData block on a provisioned Grafana datasource, and each wires one direction of the correlation. Prometheus points its exemplars at Tempo; Tempo points its spans at Loki; Loki points its logs back at Tempo:\n# Prometheus: a metric exemplar\u0026#39;s trace_id opens the trace in Tempo exemplarTraceIdDestinations: - { name: trace_id, datasourceUid: tempo } # Tempo: a span links to its logs in Loki, filtered to that trace tracesToLogsV2: { datasourceUid: loki, filterByTraceID: true, spanStartTimeShift: \u0026#39;-1h\u0026#39;, spanEndTimeShift: \u0026#39;1h\u0026#39; } # Loki: a log line\u0026#39;s trace_id links back to Tempo derivedFields: - { name: trace_id, matcherType: label, matcherRegex: trace_id, datasourceUid: tempo, url: \u0026#39;${__value.raw}\u0026#39; } One detail is worth knowing before you wire the Loki side. Trace context appears in Loki under two names: the OTLP-native trace_id and span_id, and Mastra\u0026rsquo;s own mastra_traceId and mastra_spanId attributes. They hold the same value. Key the derived field on trace_id, and use matcherType: label, which matches structured metadata as well as indexed labels.\nFrom a failed turn to the log line that explains it #Start where an operator starts: something looks wrong. The correlated dashboard has three rows: golden signals (request rate, p95 latency, agent error percentage, target up), a metrics-to-traces row where the latency panel renders exemplar diamonds, and a logs-to-traces row with a recent-traces table and a Loki logs panel. The three datasource links from the previous section are the whole mechanism, and they share one key, the trace_id that appears in every signal.\nThe correlated board: golden signals on top, latency percentiles with exemplar diamonds and throughput, and the recent-traces table below (the logs panel continues past the fold). Take the failed turn, trace 6cf6411e. Its trace ID opens the waterfall in Tempo: a GenAI-semantic-convention tree of invoke_agent weather-agent, then chat qwen/..., then the model spans and the two execute_tool spans flagged red, where the tool threw.\nThe failed turn\u0026rsquo;s waterfall: the agent span reads healthy while its two execute_tool spans are flagged red. Each span has a Logs button that jumps to that trace\u0026rsquo;s log lines in Loki. Every span has a Logs button, the Tempo-to-Loki link, that filters Loki to that one trace. That is where the failure becomes legible: the single turn produced two correlated ERROR lines under one trace ID, incident: payments probe failed and incident check failed, both from the throwing tool. You reach them by trace ID, without searching the whole log stream or working out which lines belong to which request.\nLoki filtered to the same trace_id as the waterfall: the two correlated ERROR lines for that failed turn, with the trace context carried as structured metadata. The links work from the metrics side too, which is where a latency investigation begins. A p95 exemplar diamond carries its own trace_id and jumps straight to the trace. Prometheus stored dozens of exemplars on the agent duration histogram in my run, which needs --enable-feature=exemplar-storage to retain them. Whichever signal you start from, you land on the same request.\nMastra\u0026rsquo;s spans follow the OpenTelemetry GenAI semantic conventions (v1.38.0) out of the box: span names like invoke_agent weather-agent, chat qwen/..., and execute_tool weatherTool, with attributes like gen_ai.operation.name, gen_ai.request.model, and gen_ai.usage.input_tokens (the traces I captured also included gen_ai.tool.* on the tool spans). Any OTLP-compliant backend accepts them without custom mapping, because the schema is the open GenAI semantic-convention standard rather than anything Tempo-specific. I only exercised Tempo and Loki here.\nWhat broke, and the fixes #Four things cost me time. The first three would cost any reader the same, so they are worth naming.\nTempo 3.0 dropped the top-level ingester and compactor config blocks. The quickstart config still all over the web fails to start on grafana/tempo:3.0.0 with field ingester not found in type app.Config. Tempo 3.0 restructured its config. A minimal server, distributor (the OTLP receivers), and storage is all a lab needs; sensible defaults cover the rest.\nLog export needs both the trace and the log OTLP protocol packages, or logs vanish with no error. For http/protobuf you install @opentelemetry/exporter-trace-otlp-proto and @opentelemetry/exporter-logs-otlp-proto. From the Mastra docs , verbatim: \u0026ldquo;If the matching log exporter package isn\u0026rsquo;t installed, log export is silently disabled and traces continue to work.\u0026rdquo; Install only the trace package and you get a healthy trace pipeline and zero logs, with nothing to tell you why. If your traces show up in Tempo but Loki stays empty, check this first.\nThe OTLP severity arrives uppercased, so filter on severity_text=\u0026quot;ERROR\u0026quot;, not lowercase. Loki stores the OTLP severity_text verbatim as structured metadata, preserving its case exactly as the exporter sent it (confirmed live against this stack). This setup\u0026rsquo;s OTel exporter emits severities uppercase (the common instrumentation convention), so {service_name=\u0026quot;weather-agent\u0026quot;} | severity_text=\u0026quot;ERROR\u0026quot; returns the error lines, while the lowercase error silently returns zero rows and a reader filtering by level concludes no errors exist. The casing comes from the emitter, not from Loki.\nThe fourth is not a bug, but it surprised me the first time: OTLP log attributes show up in Loki\u0026rsquo;s raw result view as a dozen fields on the stream, which looks like a cardinality problem. It isn\u0026rsquo;t. GET /loki/api/v1/labels returns only service_name for this agent. Loki promotes a small curated set of resource attributes to index labels (service, deployment, cloud, and Kubernetes identifiers), and this agent sets only service.name; everything else is structured metadata, which is queryable and low-cost, not an index label. Loki 3.x does the right thing by default.\nWhat the correlation shows that a single pillar hides #This is the failure from the top of the post, now explained. When a tool throws on every call, the agent catches it and answers, so the agent\u0026rsquo;s span and metric read status=\u0026quot;ok\u0026quot;. The tool span carries status=\u0026quot;error\u0026quot;, and an error-level log fires. On the board, the \u0026ldquo;agent error %\u0026rdquo; tile sits at 0% while ERROR lines stream in the logs panel below it. No single pillar shows both halves; the correlation does.\nThe metric alone tells you the agent is healthy. The metric plus the logs, correlated, tell you a tool is failing every call and the agent is papering over it. Watch tool status and logs, not the top-line agent metric alone.\nCommon questions # Can Mastra collect traces and logs without a trace store? Yes. The @mastra/otel-exporter is an exporter on Mastra\u0026rsquo;s observability bus, the same mechanism the store-free metrics exporter uses, so it receives span and log events with only a LibSQLStore configured and no DuckDB or ClickHouse. Querying stored observability data through Studio or the getMetric* and listLogsVNext APIs needs an OLAP store (DuckDB or ClickHouse); the OTLP export path to Tempo and Loki does not depend on one, which is why it runs with a LibSQLStore alone. Why are my Mastra logs missing from Loki when traces work? The most common cause is a missing OTLP protocol package. For http/protobuf you need both @opentelemetry/exporter-trace-otlp-proto and @opentelemetry/exporter-logs-otlp-proto. From the Mastra docs : \u0026ldquo;If the matching log exporter package isn\u0026rsquo;t installed, log export is silently disabled and traces continue to work.\u0026rdquo; So traces keep flowing while Loki stays empty, with nothing in the logs to explain it. Install both packages. How do I filter Mastra logs by level in Loki? Query the uppercased value: {service_name=\u0026quot;weather-agent\u0026quot;} | severity_text=\u0026quot;ERROR\u0026quot;. This setup\u0026rsquo;s OTel exporter emits severities uppercase and Loki stores severity_text verbatim (as structured metadata), so the lowercase error returns zero rows. How do I link a Grafana metric spike to its trace and logs? Three datasource settings, one per direction. Set exemplarTraceIdDestinations on Prometheus so a metric exemplar\u0026rsquo;s trace_id opens Tempo, tracesToLogsV2 on Tempo so a span links to its Loki logs, and derivedFields on Loki so a log\u0026rsquo;s trace_id links back to Tempo. Prometheus also needs --enable-feature=exemplar-storage or the exemplar diamonds never render. Do I need Grafana Cloud, or can this run locally? It runs fully local. Tempo, Loki, Prometheus, Grafana, and the OpenTelemetry Collector all run in Docker with no account. Grafana Cloud is a reasonable next step if you don\u0026rsquo;t want to operate the backends yourself. I didn\u0026rsquo;t test that path, but the shape should be a config swap: point the Collector\u0026rsquo;s OTLP exporters at Grafana Cloud instead of local Tempo and Loki, with an auth header. Treat it as unverified until you run it. What I would do Monday morning #If you already run Mastra agents with the Prometheus exporter, add the OTLP exporter and one Collector, and you have traces and logs correlated to your existing metrics. It is one docker compose up and two npm packages away. Turn on signals: { traces: true, logs: true }, install both OTLP protocol packages so logs don\u0026rsquo;t vanish, and provision the three Grafana datasource links so a failing request is a click from its trace and its logs.\nThen remember what the correlation is for: the agent error tile can read 0% while a tool fails every call, and the logs panel next to it is where you see the truth. Download the correlated dashboard JSON, change the service name, and you are watching your own agent across all three pillars. The exporter is at github.com/charlesgwilson/mastra-prometheus-exporter .\nPractical AI #I co-contribute to Practical AI , Damian Galarza\u0026rsquo;s newsletter for builders working with AI.\nIt covers the patterns, tradeoffs, and lessons that show up when AI moves from prototype to real work.\n","date":"July 16, 2026","permalink":"https://imawilson.com/posts/correlate-mastra-metrics-logs-traces-grafana/","section":"Writing","summary":"\u003cp\u003eI had a Mastra agent whose Grafana error tile held at 0% while a tool inside it failed on every call. The metric wasn\u0026rsquo;t lying: the agent caught each thrown exception and answered anyway, so at the agent level nothing had failed. The tool had failed, and the logs showed it; the number everyone watches never did.\u003c/p\u003e","title":"How to correlate Mastra metrics, logs, and traces in Grafana"},{"content":"A single Grafana dashboard is enough to watch a Mastra agent during an incident: five golden-signal numbers, a p95 latency, and the throughput and spend underneath. Because the exporter emits Prometheus histogram buckets, the p95 is a true windowed percentile, and every spike is tagged with its trace_id as an exemplar. This post builds that board, panel by panel, and ships the provisioned JSON so you can drop it into your own Grafana.\nIt follows an earlier post on scraping Mastra into Prometheus. That post ended with metrics and nowhere to look at them. This one is the place. Everything runs locally against a demo agent and the open-source exporter : no OLAP store, no cloud account.\nThe short version #To watch a Mastra agent in Grafana, you need one board and five numbers: requests per second, p95 latency, error percentage, tokens per second, and whether the scrape target is up. Put status colors on the two that have a threshold, add one latency panel with exemplars underneath, and you have a screen you can read at a glance during an incident.\nThe whole thing provisions from JSON and datasource YAML, so it comes up with docker compose up and reads the metrics directly. Build it panel by panel below, or copy the finished JSON from the end.\nThe stack: two containers #The dashboard needs Prometheus to scrape the exporter and Grafana to read Prometheus. That is the whole stack for a metrics dashboard. Traces and logs are worth wiring up too, but each one is an article in its own right, so I keep this post to metrics. Everything here is pinned: @mastra/core 1.49.0, @mastra/observability 1.15.2, mastra-prometheus-exporter 0.1.0 (installed from the v0.1.0 tarball), Node 22.23.1, Prometheus v3.13.0, and Grafana 13.1.0.\n# docker-compose.yml services: prometheus: image: prom/prometheus:v3.13.0 command: - \u0026#39;--config.file=/etc/prometheus/prometheus.yml\u0026#39; - \u0026#39;--enable-feature=exemplar-storage\u0026#39; # required for Grafana exemplars volumes: - ./observability/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro - ./observability/prometheus/rules.yml:/etc/prometheus/rules.yml:ro ports: [\u0026#39;127.0.0.1:9091:9090\u0026#39;] extra_hosts: [\u0026#39;host.docker.internal:host-gateway\u0026#39;] # Linux: reach the host exporter grafana: image: grafana/grafana:13.1.0 volumes: - ./observability/grafana/provisioning:/etc/grafana/provisioning:ro - ./observability/grafana/dashboards:/var/lib/grafana/dashboards:ro ports: [\u0026#39;127.0.0.1:3001:3000\u0026#39;] depends_on: [prometheus] Two details that will save you time. Prometheus needs --enable-feature=exemplar-storage or Grafana shows no exemplars. And on Linux, a container reaches an exporter running on the host through host.docker.internal only if you add the host-gateway mapping.\nPoint Prometheus at the exporter with a plain scrape config. The exporter already stamps a service label on every series, so the scrape config adds nothing:\n# prometheus.yml scrape_configs: - job_name: \u0026#39;weather-agent\u0026#39; static_configs: - targets: [\u0026#39;host.docker.internal:9465\u0026#39;] Grafana reads one datasource, provisioned with a fixed uid so the dashboard can reference it:\n# provisioning/datasources/datasources.yml datasources: - name: Prometheus uid: prometheus type: prometheus url: http://prometheus:9090 isDefault: true The metric surface you are drawing from #The exporter turns Mastra\u0026rsquo;s metric events into a handful of mastra_* series. The Overview draws from three of them:\nThe agent duration histogram mastra_agent_duration_seconds, with service, entity, entity_type, and status labels. Its _bucket, _count, and _sum give you latency, throughput, and errors from one series. A token counter mastra_model_tokens_total, with direction, type, provider, and model. A best-effort cost counter, mastra_model_cost_usd_total. One rule prevents most confusion: scope every query to your service. The type label on the token counter is also used by Node\u0026rsquo;s runtime metrics, so an unscoped label_values(type) mixes token types with things like TCPSocketWrap. Every query below is scoped to {service=\u0026quot;weather-agent\u0026quot;}.\nThe five numbers #The golden-signal row is the reason to keep this board on a screen. Five stat tiles, each one a rate or quantile, with status colors on only the two that have a threshold. Requests per second and tokens per second are plain rates:\n# requests per second sum(rate(mastra_agent_duration_seconds_count{service=\u0026#34;weather-agent\u0026#34;}[5m])) # tokens per second, scoped to type=\u0026#34;total\u0026#34; so overlapping token types don\u0026#39;t double-count sum(rate(mastra_model_tokens_total{service=\u0026#34;weather-agent\u0026#34;,type=\u0026#34;total\u0026#34;}[5m])) Target up is the scrape health, straight off Prometheus, mapped so 0 shows DOWN and 1 shows UP:\nup{job=\u0026#34;weather-agent\u0026#34;} The p95 and the error percentage are the two with a threshold, and each has a catch worth its own section below. In my run, the board below read about 0.2 requests per second, an agent p95 of 2.86s, a 0 error rate, roughly 168 input and 9 output tokens per second, and the target up. These are windowed rates, so the exact figures drift between captures.\nThe Overview board: the golden-signal stat row above the latency and throughput panels. The error tile reads 0, and a tool was failing on every call when this was captured. Latency: a windowed p95, not a per-scrape average #Latency is the headline number, and this is where emitting histograms pays off. Because the exporter emits histogram buckets, the p95 is a windowed percentile across the scrape window, not a per-scrape average:\nhistogram_quantile(0.95, sum by (le) (rate(mastra_agent_duration_seconds_bucket{service=\u0026#34;weather-agent\u0026#34;}[5m]))) Put p50, p90, p95, and p99 on one timeseries and turn on exemplars for the p95 target. In my run the p95 sat near 2.7s over a 5-minute window. The exemplar diamonds mark individual runs, each tagged with its trace_id. Grafana can link that id to a trace backend so a spike becomes one click from the trace; wiring that backend is a separate job, but the markers show up here as soon as exemplar storage is on.\nWhy the error tile can read zero when things are broken #The error percentage on this board can read 0 while your agent is failing. An agent that calls a tool can recover when that tool throws, so the agent turn reports status=\u0026quot;ok\u0026quot; and the agent error ratio stays flat. Build the tile so an all-healthy window reads 0 rather than \u0026ldquo;No Data\u0026rdquo;:\n(sum(rate(mastra_agent_duration_seconds_count{service=\u0026#34;weather-agent\u0026#34;,status=\u0026#34;error\u0026#34;}[5m])) / clamp_min(sum(rate(mastra_agent_duration_seconds_count{service=\u0026#34;weather-agent\u0026#34;}[5m])),1e-9)) or vector(0) The failure is real; it surfaces one level down, on the tool metric. This Overview shows the agent view, so add a tool error ratio next to it, because this is where a thrown tool shows up:\n# tool error ratio, by tool: this is where a thrown tool shows up sum by (entity) (rate(mastra_tool_duration_seconds_count{service=\u0026#34;weather-agent\u0026#34;,status=\u0026#34;error\u0026#34;}[5m])) / clamp_min(sum by (entity)(rate(mastra_tool_duration_seconds_count{service=\u0026#34;weather-agent\u0026#34;}[5m])),1e-9) In my run, a tool that threw on every call left the agent error tile at 0 across all 27 turns while the tool error ratio ran to 100%. The screenshot above is exactly that moment: the error tile reads 0, and a tool was failing every call. So the tile is honest about the agent and blind to the tool. Watch tool status too, and alert on the metric that records the failure. This mirrors the exporter post, where the model span reports ok on an upstream provider failure and the error surfaces on the agent span. Pick the right span for the alert.\nThroughput and spend, underneath the numbers #Two panels sit under the stat row. Throughput by status is a stacked rate, with green reserved for ok and red for error, so a rising red band is visible at a glance:\nsum by (status) (rate(mastra_agent_duration_seconds_count{service=\u0026#34;weather-agent\u0026#34;}[5m])) Token burn is the spend signal you can trust. Rate it by direction, scoped to type=\u0026quot;total\u0026quot;:\nsum by (direction) (rate(mastra_model_tokens_total{service=\u0026#34;weather-agent\u0026#34;,type=\u0026#34;total\u0026#34;}[5m])) Cost is best-effort. The exporter emits mastra_model_cost_usd_total from Mastra\u0026rsquo;s estimated cost when the provider supplies it. OpenRouter does, so the cost tile was populated in my run, climbing to about $0.005 over a 15-minute window. Some providers do not attach an estimate, and prices drift regardless, so treat the token counters as the source of truth and derive cost with a recording rule over tokens times a price table.\nProvision it as code, and copy the JSON #The whole board lives in one dashboard JSON file, mounted into Grafana through a file provider alongside the datasource YAML. No click-ops, no dashboard that exists only in one person\u0026rsquo;s browser: it comes up the same way every time and it diffs in review. You can download the Overview dashboard JSON and drop it into your own provisioning/dashboards directory; change the service name to match your agent and it renders against your metrics.\nThe recording rules and example alerts ship in the exporter repo (recording-rules.yml and alerts.yml ) and load through Prometheus rule_files. In my run all four recording rules and three alerts loaded healthy, and the alerts sat inactive because nothing was wrong: p95 under the 10s threshold, a 0 agent error ratio, and no dropped events.\nThe gotchas on this board #A few things cost me time so they do not have to cost you any:\nEmpty panels are honest; design for \u0026ldquo;No Data.\u0026rdquo; A panel bound to a metric your agent never emits stays empty, because a metric does not exist until its first event. The demo agent runs no workflow and no custom processors, so those panels never fill. On a board you watch, that should read as \u0026ldquo;nothing has happened yet,\u0026rdquo; not a broken panel, so build panels and alerts that treat cold start and absent series as 0 or \u0026ldquo;No Data,\u0026rdquo; not a false alarm. Do not stamp service in both places. The exporter emits a service label. If the scrape config adds one too, Prometheus keeps its own and renames the exporter\u0026rsquo;s to exported_service, and a query scoped to {service=...} quietly matches a split set of series. Let one side own the label. histogram_quantile needs le in the sum by, or the p95 panel returns nothing. Exemplars need the storage flag. --enable-feature=exemplar-storage on Prometheus, or the diamonds never render no matter what the panel says. Common questions # How do I get p95 latency for a Mastra agent in Grafana? Use histogram_quantile over the exporter\u0026rsquo;s _bucket series: histogram_quantile(0.95, sum by (le) (rate(mastra_agent_duration_seconds_bucket{service=\u0026quot;your-service\u0026quot;}[5m]))). Because the exporter emits histograms, this is a windowed percentile rather than a per-scrape average. Put p50 through p99 on one timeseries panel. Why does my agent error ratio stay at zero when a tool is failing? An agent can catch a tool failure and still return a successful turn, so the agent duration metric reports status=\u0026quot;ok\u0026quot; while the tool duration metric reports status=\u0026quot;error\u0026quot;. Add the tool error ratio to your board as well as the agent error tile, and alert on the metric that records the failure. Why is my Mastra cost panel empty? mastra_model_cost_usd_total is best-effort. It only populates when the provider attaches an estimated cost to the token event; some providers do not. Track tokens as the source of truth and derive cost with a recording rule over mastra_model_tokens_total times a price table. What should a good AI agent dashboard show? Golden signals first: request rate, p95 latency, error ratio, token burn, and whether the scrape target is up. Keep the board to about five numbers so it reads at a glance during an incident, with a latency panel and throughput underneath. Add drill-down boards for per-tool timing and runtime health once the top board tells you something is wrong. Do I need a database to build this? No. The exporter listens on Mastra\u0026rsquo;s observability bus and turns each event into an in-memory prom-client metric it serves at /metrics, so the exporter itself keeps no store: Prometheus and Grafana are the only moving parts you add, and Prometheus stores the history. If you want Mastra\u0026rsquo;s Studio metrics view and its query API, that is a different path (DuckDB or ClickHouse) I did not cover here. What I would do Monday morning #If you run Mastra agents and already have Prometheus, add the exporter, drop in this one board, and put it on a screen. It is one docker compose up and a running agent away. Watch the five numbers, and remember the error tile can read 0 while a tool is failing, so keep an eye on tool status too.\nDownload the dashboard JSON, change the service name, and you are watching your own agent. You can find the exporter at github.com/charlesgwilson/mastra-prometheus-exporter .\nPractical AI #I co-contribute to Practical AI , Damian Galarza\u0026rsquo;s newsletter for builders working with AI.\nIt covers the patterns, tradeoffs, and lessons that show up when AI moves from prototype to real work.\n","date":"July 10, 2026","permalink":"https://imawilson.com/posts/mastra-agent-grafana-dashboard/","section":"Writing","summary":"\u003cp\u003eA single Grafana dashboard is enough to watch a Mastra agent during an incident: five golden-signal numbers, a p95 latency, and the throughput and spend underneath. Because the exporter emits Prometheus histogram buckets, the p95 is a true windowed percentile, and every spike is tagged with its \u003ccode\u003etrace_id\u003c/code\u003e as an exemplar. This post builds that board, panel by panel, and ships the \u003ca href=\"/posts/mastra-agent-grafana-dashboard/overview-dashboard.json\" download\u003eprovisioned JSON\u003c/a\u003e so you can drop it into your own Grafana.\u003c/p\u003e","title":"A Grafana dashboard for your Mastra agent metrics"},{"content":"Mastra derives per-agent metrics (latency, tokens, cost) from its traces, and shows them well in Studio. When you want those metrics in Prometheus so your production monitoring and alerting can use them, you need a scrape endpoint that Mastra doesn\u0026rsquo;t ship. You can add one without running an analytics database. Mastra puts every derived metric onto an observability event bus, so a small exporter can listen, feed prom-client, and serve /metrics from memory, with no analytics database (DuckDB or ClickHouse) to run. Here is how I wired it up, and the open-source exporter that does it.\nEvery code block below links to the exact line in the exporter\u0026rsquo;s v0.1.0 tag so you can follow along.\nWhat Mastra gives you, and the one piece to add #Mastra already has strong observability. For Prometheus-based monitoring and alerting, you add one piece: a metrics endpoint to scrape.\nMastra extracts metrics from spans automatically when a span ends, so you get per-agent latency, token, and cost numbers with no manual instrumentation. Studio renders them in a dashboard that\u0026rsquo;s genuinely useful for watching an agent\u0026rsquo;s behavior. And the built-in exporters send trace data to systems like Datadog LLM Observability and Langfuse, which are good tools for trace-based analysis. I reach for all of these.\nWhat Mastra doesn\u0026rsquo;t ship is a Prometheus exporter: a /metrics endpoint that a Prometheus server scrapes, so you can compute a windowed p95 with histogram_quantile, wire it into Alertmanager, and page someone when an agent slows down or starts failing. If Prometheus is already how your team monitors production, that endpoint is the piece to add. This article adds it.\nWhere Mastra\u0026rsquo;s metrics live #Mastra\u0026rsquo;s metrics query API reads from an analytics store : DuckDB in development, ClickHouse in production. Run one and you get the Studio metrics view and a queryable metric history.\nThis is a reasonable design. OLAP stores are built for aggregating measurements over time, which is what a metrics history is for. If you already operate ClickHouse, that path gives you the Studio dashboard and the getMetric* query API for free.\nThe query API is one way to reach the metrics, but it isn\u0026rsquo;t the only one. Alongside storage, Mastra pushes each derived metric onto its observability bus as the metric is produced. That path is live, and it doesn\u0026rsquo;t involve the analytics store at all. So if you don\u0026rsquo;t run DuckDB or ClickHouse, the metrics are still reachable on the bus.\nThe exporter: listen on the bus, serve /metrics #The ObservabilityExporter interface has an onMetricEvent callback that fires for every derived metric, independent of storage. An exporter that implements it turns those events into prom-client metrics.\nHere is the dispatch at the center of it (src/index.ts ):\nonMetricEvent(event: unknown): void { const metric = this.#readMetric(event) if (typeof metric.name !== \u0026#39;string\u0026#39; || !isMeasurement(metric.value)) { return this.#drop(\u0026#39;invalid_payload\u0026#39;) } if (DURATION_METRIC.test(metric.name)) return this.#recordDuration(metric.name, metric) const token = parseTokenName(metric.name) if (token) return this.#recordTokens(token, metric) this.#drop(\u0026#39;unmapped_metric\u0026#39;) } A duration metric becomes a histogram observation, and a token metric becomes a counter increment. Anything it can\u0026rsquo;t map increments a dropped-events counter, so a miss shows up as a number instead of silence.\nRegistering it takes a few lines, and the storage stays a plain LibSQL file (this is the README usage ; a complete runnable version lives in examples/basic.ts ):\nimport { Mastra } from \u0026#39;@mastra/core\u0026#39; import { LibSQLStore } from \u0026#39;@mastra/libsql\u0026#39; import { Observability } from \u0026#39;@mastra/observability\u0026#39; import { PrometheusExporter } from \u0026#39;mastra-prometheus-exporter\u0026#39; const prom = new PrometheusExporter({ version: \u0026#39;1.0.0\u0026#39; }) export const mastra = new Mastra({ storage: new LibSQLStore({ id: \u0026#39;app\u0026#39;, url: \u0026#39;file:./app.db\u0026#39; }), // no OLAP store observability: new Observability({ configs: { default: { serviceName: \u0026#39;my-service\u0026#39;, exporters: [prom] } }, }), }) prom.serve(9464) // http://0.0.0.0:9464/metrics There\u0026rsquo;s no DuckDB and no ClickHouse in the process. prom.serve(9464) starts an HTTP server that exposes the metrics in the format Prometheus scrapes. The path from an agent run to a firing alert looks like this:\nflowchart LR A[Agent run] --\u003e S[Spans] S --\u003e M[Derived metrics on the bus] M --\u003e|onMetricEvent| E[Exporter /metrics] E --\u003e P[Prometheus] P --\u003e AM[Alertmanager] S -.-\u003e O[(OLAP store: DuckDB / ClickHouse)] O -.-\u003e|query API| ST[Studio dashboards] The solid path is the exporter. The dashed path is the OLAP query route that powers Studio. They run off the same spans, so you can use both.\nThe details that make it production-grade #A working exporter is short. A correct one follows a few Prometheus conventions that a first draft tends to miss.\nThe shipped version handles them:\nDurations are converted from milliseconds to seconds, the Prometheus base unit, so histogram_quantile returns a number you can read. The buckets are tuned for model latency. prom-client\u0026rsquo;s default histogram tops out at 10 seconds, and agent calls run longer, so the defaults extend to 120 seconds and stay configurable. Counters end in _total, following Prometheus naming, so tooling recognizes them. High-cardinality fields ride as exemplars, not labels. trace_id and run identifiers would blow up label cardinality, so they attach as OpenMetrics exemplars, and a latency spike links straight to the exact trace. The exporter keeps its own registry and reports its own dropped-event counter, so it never touches prom-client\u0026rsquo;s global default and you can see when it fails to map something. The full metric table is in the README , and Mastra documents the upstream metric names and cost fields in its automatic metrics reference .\nWiring Prometheus: scrape, p95, alert #Prometheus scrapes the exporter\u0026rsquo;s /metrics, computes a real p95 with histogram_quantile, and fires an alert, all on LibSQL.\nPoint Prometheus at the exporter:\nscrape_configs: - job_name: mastra static_configs: - targets: [\u0026#39;host.docker.internal:9464\u0026#39;] Record the p95 as a rule, then alert on it (from the shipped recording-rules.yml and alerts.yml ):\ngroups: - name: mastra rules: - record: mastra:agent_latency_p95_seconds:5m expr: histogram_quantile(0.95, sum by (le, entity) (rate(mastra_agent_duration_seconds_bucket[5m]))) - alert: MastraAgentLatencyP95High expr: mastra:agent_latency_p95_seconds:5m \u0026gt; 10 for: 5m labels: { severity: warning } annotations: summary: \u0026#34;Agent p95 latency high\u0026#34; Because the exporter emits real histogram buckets, this is a windowed percentile, not a per-scrape average. That\u0026rsquo;s the reason to build histograms rather than read a stored average out of a table.\nHere\u0026rsquo;s the scrape from my run, trimmed to the interesting lines:\nmastra_agent_duration_seconds_bucket{le=\u0026#34;5\u0026#34;,entity=\u0026#34;Weather Agent\u0026#34;,status=\u0026#34;ok\u0026#34;} 2 # {trace_id=\u0026#34;f919...\u0026#34;} 3.069 mastra_model_tokens_total{direction=\u0026#34;input\u0026#34;,type=\u0026#34;total\u0026#34;,provider=\u0026#34;openrouter\u0026#34;,model=\u0026#34;gpt-4o-mini\u0026#34;} 1859 mastra_model_cost_usd_total{provider=\u0026#34;openrouter\u0026#34;,model=\u0026#34;gpt-4o-mini\u0026#34;} 0.00027885 The # {trace_id=\u0026quot;f919...\u0026quot;} on the bucket line is the exemplar. Point Grafana at the same Prometheus for dashboards, where the exemplar turns a latency spike into a one-click jump to that trace. Prometheus stored the exemplars and returned them from query_exemplars, and the alert fired against a demo threshold. The whole pipeline ran on a single LibSQL file, with no analytics database. Those counts are from one lab run, so treat them as an example rather than a benchmark.\nThe gotchas that cost me time #Provider, model, and cost are nested in costContext. On Mastra 1.49, a token event carries them in a costContext object, not at the top level (automatic metrics reference ):\n// the shape Mastra attaches to a token metric metric.costContext = { provider: \u0026#39;openrouter\u0026#39;, model: \u0026#39;gpt-4o-mini\u0026#39;, estimatedCost, costUnit: \u0026#39;USD\u0026#39; } Read the wrong field (metric.provider or metric.estimatedCost) and token metrics come back with provider=\u0026quot;unknown\u0026quot; and no cost, even on a successful call. The exporter reads costContext for you; if you write your own, read there.\nThe rest:\nA metric doesn\u0026rsquo;t exist until its first event. Token and cost series appear after the first successful model call. On error-only traffic you correctly see only the duration histograms, with status=\u0026quot;error\u0026quot;. Alert on the agent status. Mastra reports the model span status=\u0026quot;ok\u0026quot; when a provider call fails upstream (an OpenRouter 429, for example); the agent span carries the error. The shipped error-ratio rule keys off the agent metric for that reason. When you also want Studio and the OLAP store #The exporter gives you Prometheus alerting. It doesn\u0026rsquo;t replace Studio or the stored-metric history, and it isn\u0026rsquo;t meant to. DuckDB and ClickHouse buy you the Studio metrics view and a queryable history; the exporter buys you live scrape-based alerting and Grafana. They read from the same spans, so run both when you want exploration in Studio and paging in Prometheus. If all you need is a monitor and a pager, the exporter on its own is the shorter path.\nCommon questions # Do I need ClickHouse or DuckDB to get Mastra metrics into Prometheus? No. You can get Mastra metrics without ClickHouse or DuckDB. Mastra pushes every derived metric onto its observability bus via onMetricEvent, independent of storage, so an exporter that listens there turns each event into an in-memory prom-client metric it serves at /metrics. The analytics store (DuckDB or ClickHouse) is only for Mastra\u0026rsquo;s own metrics query API and Studio history; the exporter path never touches it. Does Mastra ship a Prometheus exporter? No. Mastra\u0026rsquo;s built-in exporters send trace data to systems like Datadog and Langfuse. To get a Prometheus metrics endpoint, you add a small exporter against the ObservabilityExporter interface, or use the open-source mastra-prometheus-exporter. How do I get p95 latency? Use histogram_quantile over the _bucket series: histogram_quantile(0.95, sum by (le, entity) (rate(mastra_agent_duration_seconds_bucket[5m]))). Because the exporter emits real histograms, this is a windowed percentile rather than a per-scrape average. How do I track cost? Track tokens as the source of truth and derive cost with a recording rule over tokens times price. The exporter also emits a best-effort mastra_model_cost_usd_total from Mastra\u0026rsquo;s estimated cost, but prices drift, so the recording rule is the authoritative path. What about Datadog? Datadog is a different path. Mastra\u0026rsquo;s Datadog exporter sends traces to Datadog LLM Observability, which is a strong fit for trace-based analysis. The Prometheus exporter here is for scrape-based metrics and alerting. Use whichever matches how your team already monitors. How do I visualize Mastra metrics in Grafana? Point Grafana at the Prometheus that scrapes the exporter, then build panels over the mastra_* series: latency percentiles with histogram_quantile, token usage, and cost. The trace_id exemplars let you click from a latency spike to the exact Mastra trace. How does this relate to OpenTelemetry? Mastra ships an OpenTelemetry exporter that sends traces to an OTel collector. This is a different, direct path: it turns Mastra\u0026rsquo;s metric events into a Prometheus /metrics endpoint you scrape. Reach for OTel when you\u0026rsquo;re standardizing on a collector, and for this when Prometheus is already your metrics stack. Can I use a SQL exporter over Mastra\u0026#39;s tables instead? Yes, if you already run ClickHouse. The trade-off is coupling: a SQL exporter depends on Mastra\u0026rsquo;s internal table schema, which can change between releases. The onMetricEvent exporter depends on the public interface instead. What I\u0026rsquo;d do Monday morning #If you run Mastra agents in production and page on metrics, add a Prometheus scrape endpoint off the observability bus and alert on the agent metric. You don\u0026rsquo;t need a new database to do it.\nYou can find the exporter at github.com/charlesgwilson/mastra-prometheus-exporter . Point it at your agents, and your existing Prometheus setup can monitor and alert on them.\nPractical AI #I co-contribute to Practical AI , Damian Galarza\u0026rsquo;s newsletter for builders working with AI.\nIt covers the patterns, tradeoffs, and lessons that show up when AI moves from prototype to real work.\n","date":"July 3, 2026","permalink":"https://imawilson.com/posts/export-mastra-agent-metrics-to-prometheus/","section":"Writing","summary":"\u003cp\u003eMastra derives per-agent metrics (latency, tokens, cost) from its traces, and shows them well in Studio. When you want those metrics in Prometheus so your production monitoring and alerting can use them, you need a scrape endpoint that Mastra doesn\u0026rsquo;t ship. You can add one without running an analytics database. Mastra puts every derived metric onto an observability event bus, so a small exporter can listen, feed \u003ccode\u003eprom-client\u003c/code\u003e, and serve \u003ccode\u003e/metrics\u003c/code\u003e from memory, with no analytics database (DuckDB or ClickHouse) to run. Here is how I wired it up, and the \u003ca href=\"https://github.com/charlesgwilson/mastra-prometheus-exporter\" target=\"_blank\" rel=\"nofollow noopener noreferrer\"\u003eopen-source exporter\u003c/a\u003e\n that does it.\u003c/p\u003e","title":"How to export Mastra agent metrics to Prometheus (no database required)"},{"content":"I’m Greg Wilson, an experienced CTO and CIO who still likes being close to the code. I write about production AI and the engineering work around it: agents, infrastructure, DevOps, internal tools, and the operating details that make software reliable enough for real work. That bias toward useful software started early.\nMy first useful program was job bidding and estimation software I built with my dad in QBasic as a young man. He used it in his sheet metal shop. That made software real to me before it felt abstract. It was not a toy project or a syntax exercise; it was a tool someone used to run a business. Later, I rebuilt versions of the same tool while learning other languages because I understood the problem well enough to focus on the craft. That set the pattern: build tools that change how real work gets done.\nThat pattern has held for more than 20 years. I’ve built and run production software across product engineering, identity, integrations, regulated workflows, cloud infrastructure, DevOps, security, compliance, data, and delivery. The common thread is practical: build the thing, understand the workflow, and stay responsible for how it behaves in production.\nI’m now the Chief Information and Technology Officer at Buoy Software, where I started as a founding engineer. I helped build the first product, then helped turn it into a set of services for regulated healthcare workflows used by blood and plasma centers. The work includes FDA 510(k)-cleared medical devices, so I think a lot about where new technology can remove friction, automate the right work, and still leave people confident in what happened.\nMy AI work starts from that same idea: remove friction without losing confidence in the system. I build and operate agents that can help with real work, including Slack-first assistants, coding-agent orchestration, Claude Code skills and plugins, Model Context Protocol (MCP) integrations, browser and research automation, content workflows, and internal AI enablement for engineering teams.\nThat is why I write about agents the way I do. Once an agent can write data, call tools, notify people, or change a workflow, it stops being a demo. It becomes part of the system people rely on to get work done.\nWhen software becomes part of real work, evidence matters. You need to know what changed, who approved it, what the logs say, how rollback works, and who owns the failure. Regulated software taught me that lesson, and AI agents need the same discipline.\nWhat I write about #AI agents are production systems when they become part of real work. I’m interested in the practical version of that problem: how teams use AI to remove friction, automate the right work, and still understand what happened.\nThat leads to the operating details I write about:\nagent observability and evals tool boundaries and blast radius deployment and approval gates secrets, cost controls, and rollback agent-assisted software delivery governance inside the workflow agent readiness and operating reviews the platform work that lets teams use AI without guessing I want each post to give you a sharper question, a pattern to try, or a failure mode to watch for in your own systems.\nPractical AI #I co-contribute to Practical AI , Damian Galarza\u0026rsquo;s newsletter for builders working with AI.\nIt covers the patterns, tradeoffs, and lessons that show up when AI moves from prototype to real work.\nContact #Email me at hello@imawilson.com .\n","date":null,"permalink":"https://imawilson.com/about/","section":"Greg Wilson","summary":"\u003cp\u003eI’m Greg Wilson, an experienced CTO and CIO who still likes being close to the code. I write about production AI and the engineering work around it: agents, infrastructure, DevOps, internal tools, and the operating details that make software reliable enough for real work. That bias toward useful software started early.\u003c/p\u003e","title":"About"}]