How AI Works
Expensive once, cheap per request
Scroll
Expensive once, cheap per request
A model's life has two phases. Training is when its weights are set: the model is shown vast amounts of data and adjusted until its predictions are good. Inference is everything after: the frozen model being used to answer requests.
The costs are lopsided. Training a frontier model occupies large clusters of accelerators for weeks and happens once per model version. Inference costs a tiny fraction of that per request — but it happens on every request, from every user, for as long as the model is in service, so it is where the ongoing spend goes.
The split explains a common misunderstanding. A conversation does not train the model: your messages are input to inference, and the weights do not change. When an assistant seems to remember you, something outside the model stored a note and put it back into the input. And a model knows nothing after its training cutoff unless someone puts it in the prompt — the problem retrieval solves.
Foundations