Ship and Observe
Measure whether real users are better off
Scroll
Measure whether real users are better off
A golden set tells you whether a change is better on the cases you wrote down. It cannot tell you whether real users, with questions you never anticipated, are better off. For that you run an online experiment: send a random share of users to the new version, keep the rest on the current one, and compare how the two groups do.
The hard part is choosing what to compare. Thumbs-up and thumbs-down buttons are useful, but few people press them and those who do tend to have strong reactions. Behavior says more: whether users accept, copy or edit the answer, whether they ask the same question again in different words, whether they give up and contact support, and whether the task they came for got done. Track cost and latency for each group as well, because a version that is slightly better and twice as slow may not be worth shipping. Decide the main metric, and the smallest change that would justify shipping, before the test starts, so the result cannot be argued into whatever you hoped for.
LLM features make these tests noisier than most. Outputs vary from run to run, effects are often small, and the same person can end up seeing both versions unless each user is assigned to one group for the whole test. Run long enough to collect a sample that can detect the change you care about, and check results by segment, because a change that helps new users can hurt experienced ones. Ship in stages. First a shadow run, where the new version handles real traffic but its answers are not shown, to catch errors. Then a canary, a small share of users who do see it, to check safety. Then the experiment, to measure value. The canary and shadow page covers the first two in detail.
Evaluation