Adapting Models
Where helpfulness, tone and refusals come from
Scroll
Where helpfulness, tone and refusals come from
A pretrained model continues text. It does not follow instructions, and it declines nothing. The behavior people think of as the assistant, such as answering helpfully, keeping a consistent tone and refusing some requests, is added afterward: first by fine-tuning on example conversations, then by preference tuning.
Preference tuning starts with comparisons. For one prompt the model writes two answers, and a person, or another model following written guidelines, picks the better one. In RLHF, those choices train a reward model that can score any answer, and the language model is then optimized to earn higher scores, with a penalty that stops it drifting too far from where it started. Direct preference optimization, or DPO, skips the reward model: it adjusts the model directly so that preferred answers become more likely than rejected ones, which is simpler and cheaper to run.
The model learns whatever the preferences reward, flaws included. If raters tend to prefer longer answers, the model gets wordy. If they prefer agreement, it learns to tell people what they want to hear, a failure known as sycophancy. Refusals come from the same place: someone decided which requests the preferred answer turns down. When a model refuses something reasonable or praises a bad idea, the cause usually lies in this training. A clear system prompt can still shift the behavior, within limits.
Training