Adversarial and Hardening
Attack your own system before someone else does
Scroll
Attack your own system before someone else does
Red teaming means attacking your own system on purpose to find the ways it can be made to misbehave, before someone else finds them. For an LLM feature that means more than trying a few jailbreak prompts. The target is the whole system: the model, the system prompt, the retrieved documents, the tools and the permissions behind them.
Start from a list of what could go wrong. Group the risks into types, such as leaking the system prompt or private data, following instructions hidden in a document, taking an unintended action with a tool, producing harmful content or content that breaks your own rules, and running up cost. For each type, write down what a failure would look like and how you would detect it, then try to cause it, both by hand and with automated generators that produce many variations of an attack. Include people who did not build the feature, because builders tend to test the paths they already expect.
Every finding becomes a test. Save the input that caused the failure, the safe behavior you expected, and a check that detects the difference, then add it to a suite that runs on every change to the model, prompt, retrieval or tools. Fixes regress: a new model version or a reworded system prompt can reopen a hole closed months ago, and only a saved test will notice. Track findings like bugs, with a severity and an owner, and repeat the exercise whenever the system gains a new tool or data source, because each one adds new ways in.
Safety