A Grafana dashboard for your Mastra agent metrics

Q: How do I get p95 latency for a Mastra agent in Grafana?

Use histogram_quantile over the exporter’s _bucket series: histogram_quantile(0.95, sum by (le) (rate(mastra_agent_duration_seconds_bucket{service="your-service"}[5m]))). Because the exporter emits histograms, this is a windowed percentile rather than a per-scrape average. Put p50 through p99 on one timeseries panel.

Q: Why is my Mastra cost panel empty?

mastra_model_cost_usd_total is best-effort. It only populates when the provider attaches an estimated cost to the token event; some providers do not. Track tokens as the source of truth and derive cost with a recording rule over mastra_model_tokens_total times a price table.

A single Grafana dashboard is enough to watch a Mastra agent during an incident: five golden-signal numbers, a p95 latency, and the throughput and spend underneath. Because the exporter emits Prometheus histogram buckets, the p95 is a true windowed percentile, and every spike is tagged with its trace_id as an exemplar. This post builds that board, panel by panel, and ships the provisioned JSON so you can drop it into your own Grafana.

It follows an earlier post on scraping Mastra into Prometheus. That post ended with metrics and nowhere to look at them. This one is the place. Everything runs locally against a demo agent and the open-source exporter : no OLAP store, no cloud account.

The short version #

To watch a Mastra agent in Grafana, you need one board and five numbers: requests per second, p95 latency, error percentage, tokens per second, and whether the scrape target is up. Put status colors on the two that have a threshold, add one latency panel with exemplars underneath, and you have a screen you can read at a glance during an incident.

The whole thing provisions from JSON and datasource YAML, so it comes up with docker compose up and reads the metrics directly. Build it panel by panel below, or copy the finished JSON from the end.

The stack: two containers #

The dashboard needs Prometheus to scrape the exporter and Grafana to read Prometheus. That is the whole stack for a metrics dashboard. Traces and logs are worth wiring up too, but each one is an article in its own right, so I keep this post to metrics. Everything here is pinned: @mastra/core 1.49.0, @mastra/observability 1.15.2, mastra-prometheus-exporter 0.1.0 (installed from the v0.1.0 tarball), Node 22.23.1, Prometheus v3.13.0, and Grafana 13.1.0.

# docker-compose.yml
services:
  prometheus:
    image: prom/prometheus:v3.13.0
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--enable-feature=exemplar-storage'   # required for Grafana exemplars
    volumes:
      - ./observability/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./observability/prometheus/rules.yml:/etc/prometheus/rules.yml:ro
    ports: ['127.0.0.1:9091:9090']
    extra_hosts: ['host.docker.internal:host-gateway']  # Linux: reach the host exporter
  grafana:
    image: grafana/grafana:13.1.0
    volumes:
      - ./observability/grafana/provisioning:/etc/grafana/provisioning:ro
      - ./observability/grafana/dashboards:/var/lib/grafana/dashboards:ro
    ports: ['127.0.0.1:3001:3000']
    depends_on: [prometheus]

Two details that will save you time. Prometheus needs --enable-feature=exemplar-storage or Grafana shows no exemplars. And on Linux, a container reaches an exporter running on the host through host.docker.internal only if you add the host-gateway mapping.

Point Prometheus at the exporter with a plain scrape config. The exporter already stamps a service label on every series, so the scrape config adds nothing:

# prometheus.yml
scrape_configs:
  - job_name: 'weather-agent'
    static_configs:
      - targets: ['host.docker.internal:9465']

Grafana reads one datasource, provisioned with a fixed uid so the dashboard can reference it:

# provisioning/datasources/datasources.yml
datasources:
  - name: Prometheus
    uid: prometheus
    type: prometheus
    url: http://prometheus:9090
    isDefault: true

The metric surface you are drawing from #

The exporter turns Mastra’s metric events into a handful of mastra_* series. The Overview draws from three of them:

The agent duration histogram mastra_agent_duration_seconds, with service, entity, entity_type, and status labels. Its _bucket, _count, and _sum give you latency, throughput, and errors from one series.
A token counter mastra_model_tokens_total, with direction, type, provider, and model.
A best-effort cost counter, mastra_model_cost_usd_total.

One rule prevents most confusion: scope every query to your service. The type label on the token counter is also used by Node’s runtime metrics, so an unscoped label_values(type) mixes token types with things like TCPSocketWrap. Every query below is scoped to {service="weather-agent"}.

The five numbers #

The golden-signal row is the reason to keep this board on a screen. Five stat tiles, each one a rate or quantile, with status colors on only the two that have a threshold. Requests per second and tokens per second are plain rates:

# requests per second
sum(rate(mastra_agent_duration_seconds_count{service="weather-agent"}[5m]))

# tokens per second, scoped to type="total" so overlapping token types don't double-count
sum(rate(mastra_model_tokens_total{service="weather-agent",type="total"}[5m]))

Target up is the scrape health, straight off Prometheus, mapped so 0 shows DOWN and 1 shows UP:

up{job="weather-agent"}

The p95 and the error percentage are the two with a threshold, and each has a catch worth its own section below. In my run, the board below read about 0.2 requests per second, an agent p95 of 2.86s, a 0 error rate, roughly 168 input and 9 output tokens per second, and the target up. These are windowed rates, so the exact figures drift between captures.

Grafana Overview dashboard for a Mastra agent: five golden-signal stat tiles (requests/s, p95, error %, tokens/s, target up) above latency-percentile and throughput panels — The Overview board: the golden-signal stat row above the latency and throughput panels. The error tile reads 0, and a tool was failing on every call when this was captured.

Latency: a windowed p95, not a per-scrape average #

Latency is the headline number, and this is where emitting histograms pays off. Because the exporter emits histogram buckets, the p95 is a windowed percentile across the scrape window, not a per-scrape average:

histogram_quantile(0.95, sum by (le) (rate(mastra_agent_duration_seconds_bucket{service="weather-agent"}[5m])))

Put p50, p90, p95, and p99 on one timeseries and turn on exemplars for the p95 target. In my run the p95 sat near 2.7s over a 5-minute window. The exemplar diamonds mark individual runs, each tagged with its trace_id. Grafana can link that id to a trace backend so a spike becomes one click from the trace; wiring that backend is a separate job, but the markers show up here as soon as exemplar storage is on.

Why the error tile can read zero when things are broken #

The error percentage on this board can read 0 while your agent is failing. An agent that calls a tool can recover when that tool throws, so the agent turn reports status="ok" and the agent error ratio stays flat. Build the tile so an all-healthy window reads 0 rather than “No Data”:

(sum(rate(mastra_agent_duration_seconds_count{service="weather-agent",status="error"}[5m]))
  / clamp_min(sum(rate(mastra_agent_duration_seconds_count{service="weather-agent"}[5m])),1e-9)) or vector(0)

The failure is real; it surfaces one level down, on the tool metric. This Overview shows the agent view, so add a tool error ratio next to it, because this is where a thrown tool shows up:

# tool error ratio, by tool: this is where a thrown tool shows up
sum by (entity) (rate(mastra_tool_duration_seconds_count{service="weather-agent",status="error"}[5m]))
  / clamp_min(sum by (entity)(rate(mastra_tool_duration_seconds_count{service="weather-agent"}[5m])),1e-9)

In my run, a tool that threw on every call left the agent error tile at 0 across all 27 turns while the tool error ratio ran to 100%. The screenshot above is exactly that moment: the error tile reads 0, and a tool was failing every call. So the tile is honest about the agent and blind to the tool. Watch tool status too, and alert on the metric that records the failure. This mirrors the exporter post, where the model span reports ok on an upstream provider failure and the error surfaces on the agent span. Pick the right span for the alert.

Throughput and spend, underneath the numbers #

Two panels sit under the stat row. Throughput by status is a stacked rate, with green reserved for ok and red for error, so a rising red band is visible at a glance:

sum by (status) (rate(mastra_agent_duration_seconds_count{service="weather-agent"}[5m]))

Token burn is the spend signal you can trust. Rate it by direction, scoped to type="total":

sum by (direction) (rate(mastra_model_tokens_total{service="weather-agent",type="total"}[5m]))

Cost is best-effort. The exporter emits mastra_model_cost_usd_total from Mastra’s estimated cost when the provider supplies it. OpenRouter does, so the cost tile was populated in my run, climbing to about $0.005 over a 15-minute window. Some providers do not attach an estimate, and prices drift regardless, so treat the token counters as the source of truth and derive cost with a recording rule over tokens times a price table.

Provision it as code, and copy the JSON #

The whole board lives in one dashboard JSON file, mounted into Grafana through a file provider alongside the datasource YAML. No click-ops, no dashboard that exists only in one person’s browser: it comes up the same way every time and it diffs in review. You can download the Overview dashboard JSON and drop it into your own provisioning/dashboards directory; change the service name to match your agent and it renders against your metrics.

The recording rules and example alerts ship in the exporter repo (recording-rules.yml and alerts.yml ) and load through Prometheus rule_files. In my run all four recording rules and three alerts loaded healthy, and the alerts sat inactive because nothing was wrong: p95 under the 10s threshold, a 0 agent error ratio, and no dropped events.

The gotchas on this board #

A few things cost me time so they do not have to cost you any:

Empty panels are honest; design for “No Data.” A panel bound to a metric your agent never emits stays empty, because a metric does not exist until its first event. The demo agent runs no workflow and no custom processors, so those panels never fill. On a board you watch, that should read as “nothing has happened yet,” not a broken panel, so build panels and alerts that treat cold start and absent series as 0 or “No Data,” not a false alarm.
Do not stamp service in both places. The exporter emits a service label. If the scrape config adds one too, Prometheus keeps its own and renames the exporter’s to exported_service, and a query scoped to {service=...} quietly matches a split set of series. Let one side own the label.
histogram_quantile needs le in the sum by, or the p95 panel returns nothing.
Exemplars need the storage flag. --enable-feature=exemplar-storage on Prometheus, or the diamonds never render no matter what the panel says.

Common questions #

How do I get p95 latency for a Mastra agent in Grafana?

Use histogram_quantile over the exporter’s _bucket series: histogram_quantile(0.95, sum by (le) (rate(mastra_agent_duration_seconds_bucket{service="your-service"}[5m]))). Because the exporter emits histograms, this is a windowed percentile rather than a per-scrape average. Put p50 through p99 on one timeseries panel.

Why does my agent error ratio stay at zero when a tool is failing?

An agent can catch a tool failure and still return a successful turn, so the agent duration metric reports status="ok" while the tool duration metric reports status="error". Add the tool error ratio to your board as well as the agent error tile, and alert on the metric that records the failure.

Why is my Mastra cost panel empty?

mastra_model_cost_usd_total is best-effort. It only populates when the provider attaches an estimated cost to the token event; some providers do not. Track tokens as the source of truth and derive cost with a recording rule over mastra_model_tokens_total times a price table.

What should a good AI agent dashboard show?

Golden signals first: request rate, p95 latency, error ratio, token burn, and whether the scrape target is up. Keep the board to about five numbers so it reads at a glance during an incident, with a latency panel and throughput underneath. Add drill-down boards for per-tool timing and runtime health once the top board tells you something is wrong.

Do I need a database to build this?

No. The exporter listens on Mastra’s observability bus and turns each event into an in-memory prom-client metric it serves at /metrics, so the exporter itself keeps no store: Prometheus and Grafana are the only moving parts you add, and Prometheus stores the history. If you want Mastra’s Studio metrics view and its query API, that is a different path (DuckDB or ClickHouse) I did not cover here.

What I would do Monday morning #

If you run Mastra agents and already have Prometheus, add the exporter, drop in this one board, and put it on a screen. It is one docker compose up and a running agent away. Watch the five numbers, and remember the error tile can read 0 while a tool is failing, so keep an eye on tool status too.

Download the dashboard JSON, change the service name, and you are watching your own agent. You can find the exporter at github.com/charlesgwilson/mastra-prometheus-exporter .

Practical AI #

I co-contribute to Practical AI , Damian Galarza’s newsletter for builders working with AI.

It covers the patterns, tradeoffs, and lessons that show up when AI moves from prototype to real work.