Adapting Models

RLHF and preference tuning

Where helpfulness, tone and refusals come from

Scroll

Where helpfulness, tone and refusals come from

A pretrained model continues text. It does not follow instructions, and it declines nothing. The behavior people think of as the assistant, such as answering helpfully, keeping a consistent tone and refusing some requests, is added afterward: first by fine-tuning on example conversations, then by preference tuning.

Preference tuning starts with comparisons. For one prompt the model writes two answers, and a person, or another model following written guidelines, picks the better one. In RLHF, those choices train a reward model that can score any answer, and the language model is then optimized to earn higher scores, with a penalty that stops it drifting too far from where it started. Direct preference optimization, or DPO, skips the reward model: it adjusts the model directly so that preferred answers become more likely than rejected ones, which is simpler and cheaper to run.

The model learns whatever the preferences reward, flaws included. If raters tend to prefer longer answers, the model gets wordy. If they prefer agreement, it learns to tell people what they want to hear, a failure known as sycophancy. Refusals come from the same place: someone decided which requests the preferred answer turns down. When a model refuses something reasonable or praises a bad idea, the cause usually lies in this training. A clear system prompt can still shift the behavior, within limits.

Training
ONE COMPARISON, MANY TIMES OVER prompt: summarize this email in one line A: Thanks for your email. To summarize, there are several points worth noting… B: The launch moves to Friday. preferred RLHF reward model, then RL DPO adjusts the model directly after tuning, for prompts like this one · illustrative answers like A less likely answers like B more likely Whatever the raters prefer, the model learns, including their habits.
Two answers to one prompt, a preference between them, and the model shifting toward the preferred kind.