Traditional application monitoring answers: 'Is the service responding to requests?' Agent observability answers: 'Is the agent making good decisions?' An agent can be 'up' (responding to every request) while being 'wrong' (making poor decisions at every step). This requires a fundamentally different observability approach — one that tracks the agent's reasoning, not just its response time.
What to Track
Agent observability tracks six signal types: (1) Decision traces — the complete chain of reasoning from input to output, including every tool call, every intermediate decision, and the final output. (2) Tool metrics — which tools are called, how often, with what parameters, and what the success/failure rate is for each. (3) Quality scores — automated quality evaluation of agent outputs against expected outcomes. (4) Latency breakdown — time spent in each phase: planning, tool execution, LLM inference, output validation. (5) Cost tracking — LLM tokens consumed, tool API calls, compute resources. (6) Error taxonomy — categorising failures by type (tool failure, planning error, guardrail violation, quality degradation).
The Observability Stack
A production agent observability stack has three layers: (1) Trace storage — an append-only store of every agent decision trace, queryable by task, agent, tool, outcome, and time range. (2) Dashboard — real-time visibility into agent health: success rate, average latency, cost per task, error rate by type. (3) Alerting — automated alerts on quality degradation, latency spikes, error rate increases, and cost anomalies. The stack should answer: 'What did the agent do? Why did it do it? Was the outcome good? How much did it cost?'
Note
If your observability stack cannot show you the complete reasoning chain for a specific agent execution, you cannot debug agent failures.
Drift Detection
Agent quality degrades over time as the data distribution changes. Implement drift detection: compare the distribution of recent agent outputs against the baseline distribution, flag statistically significant changes, and trigger investigation. Common drift signals: changing tool call patterns (the agent is using different tools than it used to), changing output quality (scores are declining), and changing cost patterns (the agent is consuming more tokens per task). Drift detection catches quality degradation weeks before it becomes a business problem.
Conclusion
Agent observability is not application monitoring with extra fields. It is a different discipline that tracks reasoning quality, decision chains, and outcome distribution. Build the trace storage, dashboard, and alerting from day one — debugging agent failures without observability is nearly impossible.
Key Takeaways
- Track six signals: decision traces, tool metrics, quality scores, latency breakdown, cost, error taxonomy
- The stack must answer: what did the agent do, why, was it good, and how much did it cost?
- Drift detection catches quality degradation weeks before it becomes a business problem
- Without complete decision traces, debugging agent failures is nearly impossible
- Agent observability is a different discipline from application monitoring