Ship and Observe

Cost and token telemetry

Know which feature spent the money, and when

Scroll

Know which feature spent the money, and when

The provider's invoice tells you what you spent last month. It does not tell you which feature spent it, which customer, or why Tuesday cost three times as much as Monday. For that you need to record usage per request, at the moment it happens.

Every model response reports how many input and output tokens it used, and many also report cached input tokens, the part of the prompt served from the provider's cache, separately. Log those with each request, along with the model name, the feature that made the call and an ID for the user or team, then multiply by a price table you keep up to date to get the cost. For agents, also log the number of model calls and tool calls in each run, because one user action can turn into dozens of requests. Attach the request ID your tracing already uses, so an expensive request can be opened and inspected step by step.

With that in place, alert on the patterns behind surprise bills: a feature's spend well above its usual daily level; a jump in input tokens per request, which often means a prompt or retrieval change started sending more context; replies hitting the output limit again and again; and agent runs that take far more steps than usual. Track cost per successful task alongside cost per request, because a cheaper prompt that fails more often can cost more overall. And keep prompts and replies out of the cost logs unless you need them. Token counts and IDs are enough, and they carry far less private data.

Observability
SPEND BY FEATURE, PER DAY daily spend · illustrative retrieval change alert: 3× usual summarizer support search day → one logged request req 8f2a · summarizer · team-legal · in 7,950 · out 410 · $0.030 Log tokens, feature and user on every request. The invoice alone cannot say what changed.
Daily spend by feature. One line jumps after a retrieval change, and an alert fires.