What is telemetry?

Definition

Services can report metrics, events or traces about requests, errors and elapsed time. Agent telemetry may show when a tool was called and whether it succeeded, subject to privacy and access controls.

Telemetry helps teams find changes in behavior, but it may not explain why an outcome matters to a user. Reviews and outcome data can add important context.

ELI5

For example, an assistant might record how often document searches return errors and how long they take.

If failures increase after an update, the team can investigate the affected step without needing to save every user's document.

Does telemetry need to include full user messages?

No. Operational counts, timing and outcome flags can help diagnose many problems without storing raw inputs.

Can telemetry catch regressions?

Changes in error rates, completion signals or response time can reveal regressions, especially when compared with a consistent baseline.

  1. Nishant Gupta beside the words Agent Evaluations on black.
Definition card for telemetry: What is telemetry?

Telemetry is operational data a running system records so its behavior can be measured and investigated.