Ship and Observe
Know which feature spent the money, and when
Scroll
Know which feature spent the money, and when
The provider's invoice tells you what you spent last month. It does not tell you which feature spent it, which customer, or why Tuesday cost three times as much as Monday. For that you need to record usage per request, at the moment it happens.
Every model response reports how many input and output tokens it used, and many also report cached input tokens, the part of the prompt served from the provider's cache, separately. Log those with each request, along with the model name, the feature that made the call and an ID for the user or team, then multiply by a price table you keep up to date to get the cost. For agents, also log the number of model calls and tool calls in each run, because one user action can turn into dozens of requests. Attach the request ID your tracing already uses, so an expensive request can be opened and inspected step by step.
With that in place, alert on the patterns behind surprise bills: a feature's spend well above its usual daily level; a jump in input tokens per request, which often means a prompt or retrieval change started sending more context; replies hitting the output limit again and again; and agent runs that take far more steps than usual. Track cost per successful task alongside cost per request, because a cheaper prompt that fails more often can cost more overall. And keep prompts and replies out of the cost logs unless you need them. Token counts and IDs are enough, and they carry far less private data.
Observability