Inside the Model
Paying for thinking, by the token
Scroll
Paying for thinking, by the token
Chain-of-thought prompting asks a model to show its working. Reasoning models are trained to do it unprompted: before answering, they produce a long run of reasoning tokens, often trying an approach, checking it and backing out. The training usually relies on reinforcement learning against problems with checkable answers, such as math and code.
This moves effort from training time to answer time, which is why it is called test-time compute. On hard problems, accuracy keeps rising as the model is allowed to think for longer, though each further gain costs more tokens than the last. Many APIs expose this as a setting, such as a thinking budget or an effort level.
Reasoning tokens are usually billed as output tokens, so a request that thinks for 20,000 tokens costs about as much, and takes about as long, as one that writes 20,000 tokens, even if the final answer is one line. On easy questions the extra thinking adds cost and delay without improving the answer, so set the budget per task. And the reasoning a model shows is not guaranteed to be the computation that produced its answer. It helps with debugging, but it is a weak audit trail.
Generation