Route a Grafana alert to a read-only Mastra triage agent

When a Grafana alert fires, a Mastra agent reads the metric, pulls the failing trace, follows the stack to the bug at orders.js:23, and posts a cited file-or-close recommendation to Slack. Then it stops. It changes nothing, and a human makes the call. This is the first step of an SRE agent I’m building, read-only to start. The post walks through the whole loop and what I learned wiring it up. The code is public, so you can clone it and build it yourself .

What read-only buys you #

The slow part of an incident is assembling the evidence, and that is what the agent collapses. The metric says something is wrong, the trace says which request, the stack trace says which line, and the logs say how often. On a bad night you gather those by hand across four tools while the channel fills up. The agent gathers them in one pass and writes down what they mean, with links, before you have finished reading the page. Nothing is suppressed silently. The raw alerts still exist, and every recommendation is a hypothesis with receipts I can audit.

Starting read-only is already teaching me to separate two things we usually collapse into one, how freely the agent may investigate and what it may act on. This step grants the first and withholds the second. My instinct going in was to reach for action first, and watching the agent investigate is showing me why that is worth resisting.

The demo stack #

Everything here runs on a self-contained demo stack, so the whole loop is reproducible and no real system is involved. One docker-compose stack holds the telemetry and the alerting. The demo webapp (the subject) and the triage agent both run as host processes beside it. It is all in the companion repo . Clone it, follow the README, and you get the same loop firing on your machine. Every code block below is an excerpt linking that repo at the v0.1.0 tag. The observability half (Grafana, Tempo, Loki, Prometheus) is the same stack the earlier posts stood up across exporting Mastra metrics to Prometheus , the metrics dashboard , and correlating metrics, logs, and traces .

• An instrumented demo webapp, a small Express service with OpenTelemetry traces and logs and a prom-client metrics endpoint, plus a traffic-and-fault generator. It has a planted bug. computeTotal in orders.js sums item.price * item.qty with no null-check, so an undefined line item throws TypeError: Cannot read properties of undefined (reading 'price'). The handler records the exception on the span, so a failing trace names orders.js:23. That is the thread the agent pulls.
• Its own Grafana, Tempo, Loki, and Prometheus (plus Alloy as the OTLP hub), with datasource correlation provisioned so the alert-to-trace-to-logs join resolves.
• Grafana-managed alerting, provisioned as YAML, a rule on the error ratio, a webhook contact point, and a notification policy. A Grafana-managed alert rule routes through Grafana’s own built-in Alertmanager to its contact point, so the demo needs no separately-deployed external Alertmanager (confirmed live, with the rule firing and Grafana POSTing straight to the webhook and no external Alertmanager running in the stack).
• The agent’s read surface is the Grafana MCP , a Model Context Protocol (MCP) server that exposes Grafana’s datasources as agent tools (mcp-grafana, over stdio through Mastra’s MCPClient, on a read-only Viewer service account), for Prometheus metrics and Loki logs. Tempo traces and the source read go direct, because mcp-grafana exposes no Tempo trace-query tool.

The load-bearing versions are Node 22.23.1, @mastra/core 1.51.0, @mastra/mcp 1.14.0 (the MCP client), mcp-grafana v0.17.0, Grafana 13.1.0, Tempo 3.0.2, Loki 3.7.3, Prometheus 3.13.0, Alloy 1.17.1.

The seed the agent starts from #

The agent does not start from a blank slate. A deterministic seed runs first, because 2 facts are cheap to establish without a model and expensive to get wrong. It queries Prometheus for the error ratio, confirms the breach, searches Tempo for a recent error trace, and parses the exception out of it:

SEED (deterministic, before the model runs):
  metric: error_ratio_5xx = 0.6000000000000001  (threshold 0.1)
  trace:  2986e91dfa8644013024391c8a0400  { resource.service.name = "demo-webapp" && status = error }
  exception: TypeError: Cannot read properties of undefined (reading 'price')  ->  orders.js:23

The seed already names the failing frame (the exact file and line the exception’s stack trace blames), so the model is not finding the root cause from scratch. It corroborates that starting point, judges whether the alert is noise or actionable, and gathers the context a human would want attached. Handing the model a grounded start is what makes this step reliable. It is also why the model you pick matters less than you would expect, and I come back to it in a later post.

From there the agent drives its own investigation with 4 tools, and every one is a read. The full implementations, including the guard that bounds them, are in tools.ts and backends.ts .

The Grafana MCP runs on a read-only Viewer service account, so read-only is enforced by the account’s role, not by my code. A write call returns 403 no matter what the agent asks. The one tool that touches the filesystem, read_source , resolves real paths and refuses anything outside the webapp’s checkout.

Route the alert to the agent #

The Mastra server is the webhook receiver. registerApiRoute adds a POST route, and a middleware checks a shared-secret bearer token before the handler runs, so an unauthenticated POST gets a 401. The handler acks immediately, then runs the triage off the request path in a detached task (the full route , with persistence and logging elided below):

handler: async (c) => {
  let payload: any
  try { payload = await c.req.json() } catch { return c.json({ error: 'invalid payload' }, 400) }
  if (payload == null || typeof payload !== 'object' || Array.isArray(payload)) return c.json({ error: 'invalid payload' }, 400)
  const service = payload?.commonLabels?.service || payload?.alerts?.[0]?.labels?.service || 'demo-webapp'
  if (typeof service !== 'string' || service.length > 200 || !SERVICE_RE.test(service)) return c.json({ error: 'invalid service' }, 400)
  if (payload?.status !== 'firing') return c.json({ received: true, skipped: `status=${payload?.status}` })
  void (async () => {                       // ack now, investigate async (try/catch + persistence writes elided)
    const s = await seed(service)
    if (s.metric.value == null || !s.trigger.traceId) return  // don't triage against nothing
    const inv = await runInvestigation(s)
    const triage = await structure(inv.text)
    const gate = citationGate(inv.ledger, triage)
    const card = renderCard(inv.ledger, triage, gate)
    await maybePostSlack(card)              // posts the card; never touches prod
  })()
  return c.json({ received: true, status: 'processing' })
},

renderCard formats the Slack card and maybePostSlack posts it, logging to stdout when no Slack token is set.

Two boundary checks matter here. The bearer check uses crypto.timingSafeEqual, so a wrong token can’t be recovered a byte at a time from the response timing. The service label is charset-restricted at the boundary before it reaches a TraceQL (Tempo’s trace query language) or LogQL (Loki’s log query language) query.

That bearer check guards the webhook route only, not the rest of the Mastra API on the same port, which includes the Studio playground and its other routes. So keep the server bound locally.

Acking before you investigate matters, and I did not do it at first. Grafana’s webhook notifier has a short timeout, and the gather-plus-model pass outran it. Grafana marked the notification failed and retried on every evaluation, which re-triggered the agent. The fix is to return a 200 immediately, then run the triage in a detached async task.

On the Grafana side, the contact point is provisioned as YAML , a webhook with a Bearer scheme :

# observability/grafana/provisioning/alerting/contactpoints.yaml
apiVersion: 1

contactPoints:
  - orgId: 1
    name: mastra-webhook
    receivers:
      - uid: mastra-webhook
        type: webhook
        disableResolveMessage: false
        settings:
          url: http://host.docker.internal:14111/webhooks/grafana
          httpMethod: POST
          authorization_scheme: Bearer
          authorization_credentials: $GRAFANA_WEBHOOK_SECRET

Grafana interpolates $GRAFANA_WEBHOOK_SECRET from the container’s environment when it loads the provisioning file, so the secret stays out of the YAML.

Let the agent drive the investigation #

runInvestigation is a bounded ReAct loop (the agent reasons, then acts, one step at a time), not a fixed script. The agent gets one turn of user content, the seed rendered as a prompt. Its investigation strategy lives in the system INSTRUCTIONS , and within that it decides which tools to call and when to stop. It calls agent.generate with a step cap and a per-step callback that records each step’s tools:

const result = await agent.generate(seedPrompt, {
  maxSteps,
  onStepFinish: ({ toolCalls, finishReason }: StepFinish) => {
    stepLog.push({
      step: stepLog.length + 1,
      tools: (toolCalls ?? []).map((call) => call.toolName ?? call.payload?.toolName ?? 'unknown'),
      finishReason: finishReason ?? 'unknown',
    })
  },
})

The step log is the receipt that the path is the agent’s, not mine. On the shown run the frontier model took 3 steps and 5 tool calls, and it did what a fixed evidence bundle never does. In one turn it fetched a baseline from a window back, read the source at the exception’s line, and pulled a first round of neighboring traces, then went back for a second pass to size the blast radius, 16 in all.

step 1: get_baseline, read_source, get_more_logs, get_neighboring_traces   (finish: tool-calls)
step 2: get_neighboring_traces                                             (finish: tool-calls)
step 3: (none)                                                             (finish: stop)
-> 5 tool calls; gathered 2 metric points, 12 logs, 16 traces, 1 source read

Freedom to explore needs a floor, so each tool guards itself in code , with a per-tool call cap and a stop on the third identical query. Mastra also ships Code Mode , where the agent writes one program that calls the tools instead of calling them one at a time. For a read-only step I want the narrowest blast radius, so I kept four constrained tools behind a deterministic guard. Letting the model write code that runs in a sandbox is a later step, once that sandbox and its safety story are worked out.

Gate every citation, then render the card #

The agent’s freedom to investigate is balanced by a deterministic gate on its output. Before a card renders, every citation is checked against the evidence ledger the tools wrote. A source citation clears two checks. It must name a file the agent read (existence), and it must be the frame the exception points at (support):

// existence: the ref must name a source a tool actually read, by path
const source = ledger.sources.find((s) => ref.includes(norm(s.path)))
if (!source) {
  existenceIssues.push(`source "${citation.ref}" was never read by a tool (fabricated or unread)`)
  continue
}
// support: the cited source must be the failing frame (basename + line), not just any read file
if (!citesFailingFrame(ledger, ref)) {
  semanticIssues.push(
    `source "${citation.ref}" was read but is not the failing frame ` +
      `(exception points at ${failingFile}:${failingLine})`,
  )
}
// further down, an actionable disposition needs a source citation that survives BOTH checks

Two negative controls, run against the gate directly, show it earns its place. Feed it a citation to billing.js:99, a file no tool read, and existence rejects it. Feed it server.js:30, a real source line seeded into the ledger as read but not the frame the exception points at, and it clears existence and is rejected on support.

The card labels that second check “semantic support,” which oversells it. It is not the agent re-reading the span and judging relevance. It is a deterministic match against the failing frame the seed already parsed from the trace, orders.js:23. That makes it a strong guard when the seed is right, and no guard at all when the seed is wrong. A later post takes that limit apart. Here it is enough that the gate is deterministic and conditional on the seed, not a model grading its own homework.

The output is the Slack card. It carries the seed’s breach metric, 0.60, now surrounded by the baseline, logs, traces, and source read the agent gathered to back the call. Here is the shown run, gate passed:

*Alert triage — demo-webapp* 🚨
*Disposition:* actionable  |  *Recommendation:* file_ticket  |  *Confidence:* high

*Root cause hypothesis:* orders.js:23 attempts to read the 'price' property from items in
an order array without null-checking. When the array contains undefined elements, the code
throws "TypeError: Cannot read properties of undefined (reading 'price')", causing checkout
failures at a 60% error rate.

*Investigation:* 5 tool calls (get_baseline, read_source, get_more_logs, get_neighboring_traces,
                 get_neighboring_traces)
*Evidence gathered:* 2 metric points · 12 logs · 16 traces · 1 source reads

*Citations:* `metric`: error_ratio_5xx = 0.6000000000000001 (threshold 0.1) · `source`: orders.js:23 ·
             `log`: checkout failed: Cannot read properties of undefined (reading 'price') ·
             `trace`: 2986e91dfa8644013024391c8a0400
*Citation gate:* ✅ passed (existence + semantic support)
_Read-only recommendation. A human decides._

An operator reading that at 3 a.m. does not start from zero. They start from a named line, a cited trace, a blast-radius estimate, and a call they can audit or overrule. Then they decide what to do, which was never the agent’s to make.

What broke, and the fixes #

Two Mastra-specific things cost me time, and they will cost you the same.

A structured answer took two tries. I wanted the model to return a filled-in card, not free text. My first try used the output option. It came back empty, and the code that expected data crashed. The one that works is structuredOutput . There is one more catch. onStepFinish (the hook that lets me watch each step the agent takes) only fires in plain-text mode, not with structured output. So the agent investigates in text, and a cheap second pass turns it into the card.

The agent couldn’t find the code to read. The source-reading tool is scoped to one folder, set by WEBAPP_SRC, and its default (./demo-webapp) is resolved from the process’s working directory. The dev server runs somewhere other than the project root, so that resolved to nowhere and the tool read nothing. With no source to cite, the gate blocked the verdict. It won’t pass an “act on this” call with no source behind it. So the bug surfaced as a blocked answer, not a wrong one. The fix is to set WEBAPP_SRC to an absolute path in .env, then restart, since the server reads it only at startup.

Common questions #

What I would do Monday morning #

If you run on-call with Grafana alerting, stand up the investigate-only step before anything that acts. Point a firing alert at a webhook, seed the investigation from the metric and the failing trace, give the agent a few tools that only read, and gate every citation before it posts. You get a grounded, auditable hypothesis attached to the page, and every action stays in human hands.

The reason to start read-only is what it teaches you. You cannot write the runbook for an SRE agent until you have watched one triage without touching anything. So clone the repo , fire the demo alert, and try it on your own cases, a noisy alert, a wrong seed, a different model, and watch where the gate holds and where it slips. The next post in the series works through what the agent has to read before its recommendations deserve trust.

Practical AI #

I co-contribute to Practical AI , Damian Galarza’s newsletter for builders working with AI.

It covers the patterns, tradeoffs, and lessons that show up when AI moves from prototype to real work.