Adversarial and Hardening

Red teaming an LLM system

Attack your own system before someone else does

Scroll

Attack your own system before someone else does

Red teaming means attacking your own system on purpose to find the ways it can be made to misbehave, before someone else finds them. For an LLM feature that means more than trying a few jailbreak prompts. The target is the whole system: the model, the system prompt, the retrieved documents, the tools and the permissions behind them.

Start from a list of what could go wrong. Group the risks into types, such as leaking the system prompt or private data, following instructions hidden in a document, taking an unintended action with a tool, producing harmful content or content that breaks your own rules, and running up cost. For each type, write down what a failure would look like and how you would detect it, then try to cause it, both by hand and with automated generators that produce many variations of an attack. Include people who did not build the feature, because builders tend to test the paths they already expect.

Every finding becomes a test. Save the input that caused the failure, the safe behavior you expected, and a check that detects the difference, then add it to a suite that runs on every change to the model, prompt, retrieval or tools. Fixes regress: a new model version or a reworded system prompt can reopen a hole closed months ago, and only a saved test will notice. Track findings like bugs, with a severity and an owner, and repeat the exercise whenever the system gains a new tool or data source, because each one adds new ways in.

Safety
FROM FAILURE TYPES TO A TEST SUITE prompt or data leak hidden instructions unintended tool action harmful content runaway cost attacks by hand + generated findings 3 this round test suite every change rerun on every model, prompt or tool change each finding is saved as: the input · the expected safe behavior · a check Start from a list of failure types, attack each one, and turn every finding into a test so a fixed hole cannot reopen unnoticed.
Failure types feed a round of attacks, and every finding becomes a test that reruns on each change.