Monitoring an AI agent in production with Arize: traces to alerts

Start with one agent, one owner, one threshold

Arize AX is a platform for tracing, evaluating and monitoring AI agents. A conventional application either errors or it does not. An agent can return a fluent answer after choosing the wrong tool, so you need to see each step and score it. That is the job Arize describes for AX: end-to-end traces, evals at span, trace and session level, and production signals that point to problems.

My advice is to pilot on one agent and settle three questions before the first monitor goes live: who is alerted, at what quality threshold, and who fixes it. The PagerDuty example is the clearest model because it addresses all three.

123

Traces: what gets captured and how much fits in a plan

The integrations documentation splits tracing from evaluation. Tracing covers 16 LLM providers (including OpenAI, Anthropic, Google GenAI and AWS Bedrock), agent frameworks in Python, TypeScript or JavaScript, Java and .NET, ten coding agents such as GitHub Copilot, Cursor and Devin, and platforms such as Dify, Flowise and LangFlow. OpenTelemetry routes are listed too: Arize OTEL, OpenLLMetry and OpenLIT.

Volume sets the plan. The pricing grid lists AX Free at 25,000 trace spans a month with 15 days of retention, and AX Pro at 50 dollars a month with 50,000 spans and 30 days. Enterprise has custom volume and retention and can be self-hosted. Evals, experiments and human annotations are listed as unlimited on every tier. Because one agent run produces many spans, estimate spans per run from a test before choosing a plan.

45

Evals: judge, ground truth and human feedback

PagerDuty, as Arize describes it, runs several kinds of evaluation across its agents in PagerDuty Advance. Deterministic questions, such as incidents from yesterday, are checked against ground truth. An LLM judge scores individual steps: tool selection, parameter accuracy and response relevance. Users also rate answers with a thumbs up or down in Slack, and each rating is tied to a span ID so the team can rebuild the trace behind it.

TheFork evaluates specific components rather than treating the system as a black box: LLM call quality, guardrail correctness, retrieval quality, latency, token use and errors. It combines LLM judges with code-based checks and targets the slices of traffic with the most signal, such as spans crossing latency or guardrail thresholds. Online evals run continuously with hourly uploads and targeted backfills. Tracing exposed duplicated embedding calls on a critical path, and removing them reduced p95 latency, according to Arize's write-up.

16

Alerts: from monitor to page to ticket

At PagerDuty, when an output-quality metric crosses a threshold, Arize sends an alert to PagerDuty. The agent team's on-call engineer receives a low-urgency page, and an incident workflow creates a Jira ticket for visibility and ownership. PagerDuty's own SRE Agent then triages AI incidents: it classifies whether human escalation is needed and brings Arize failure-pattern context into the incident.

On the question of who fixes it, the case describes a spread-out model. Specialized agent engineers act as internal experts and build shared tooling, software engineers across the company build agents on top of it, and product managers increasingly review agent behavior to tighten feedback loops. Decide early which of those roles owns the Jira ticket in your organization.

1

Booking.com: collect once, redact, route

Booking.com runs two paths. Agent traces follow OpenTelemetry and OpenInference conventions and pass through a central AI telemetry collector that samples them, removes sensitive passages and sends them to Arize and analytics systems. Traditional ML models feed prediction logs from the data warehouse, with labels added when they become available. If your agents handle personal data, a collector in front of the observability tool is the design to copy.

Arize reports two episodes. In one, monitoring caught an agent whose latency jumped several standard deviations above baseline; the cause was a new experiment using a different model without the right service tier. In the other, while prototyping an itinerary-planning agent, P90 token counts spiked and later turns scored lower in multi-turn evals; long attraction links fed back into the conversation history were bloating the context. According to the write-up, the team intervened before that configuration reached production. These are vendor-reported cases from one company, not a benchmark.

7

Quality score is not business result, and what the sources leave open

Every outcome in these cases is a technical signal: latency, tool choice, retrieval quality, eval scores. The sources read here do not link an eval score to revenue, conversion or support cost, so do that join yourself, for example by comparing eval results with a business metric your product team already owns. That is an editorial recommendation, not an Arize feature.

One gap is worth closing before purchase. The integrations page lists tracing and evaluation integrations but no alerting or notification ones, while PagerDuty describes pages and Jira tickets. Confirm in a trial how a monitor reaches your on-call and ticket tools. Customer counts on usedby show who uses Arize, not how well it works.

1467