Cost and Choices
You pay for what you send and what comes back
Scroll
You pay for what you send and what comes back
Hosted models charge per token, and they count what you send separately from what comes back. Output tokens usually cost several times more than input tokens, partly because the model writes its reply one token at a time but reads your input in one pass.
Take a support assistant. Each request sends about 2,000 tokens: the system prompt, a few retrieved passages and the question. The reply is about 300 tokens. At an illustrative $3 per million input tokens and $15 per million output tokens, 10,000 requests a day comes to $60 of input and $45 of output. That is about $105 a day, or $3,150 a month.
Input is the bigger share here, even though each output token costs five times as much. The system prompt and the passages are resent with every request, and in a chat the whole history is resent every turn, so a long conversation costs more with each message. That points to the levers: trim what you resend, retrieve fewer passages, cap the reply length, cache the part of the prompt that repeats, and send easy requests to a cheaper model. The pages on prompt caching and model routing cover the last two.
Cost