Adversarial and Hardening

Data exfiltration and tool permissions

How an agent leaks data, and how to give it less to leak

Scroll

How an agent leaks data, and how to give it less to leak

Exfiltration means getting data out of a system to someone who should not have it. An agent that can read private data and also takes in untrusted content can be turned against its own user this way. Instructions hidden in a web page or an email tell it to collect something sensitive and then send it somewhere. The sending can be surprisingly indirect. If the chat interface displays images automatically, the model can be led to write an image link whose address contains the data. The user's own app then requests that address, which delivers the data to whoever runs the server, without any tool call at all.

Three things together make this possible: access to private data, exposure to untrusted content, and a way to send information out. Remove any one and this kind of attack fails, so start by asking which of the three each feature really needs. A summarizer for public web pages does not need the user's inbox. An assistant that reads the inbox may not need to browse the web or display remote images.

Then give every tool the smallest set of permissions it can work with, a principle known as least privilege. Prefer read-only access, narrow scopes such as one folder or one project, and the user's own credentials over a shared admin key, so the agent can never reach more than the person using it. Restrict where data can go: keep a list of domains the agent is allowed to contact, and display images only from trusted hosts. Require confirmation for anything that sends, shares or deletes, and log every tool call with its arguments so a leak can be traced afterward.

Safety
A LEAK WITH NO TOOL CALL web page hidden instruction agent inbox private reply includes an image link: unknown-site.example/pic.png?d=… the user's app loads it data reaches the site the attack needs all three: access to private data untrusted content a way to send data out cut one: show images only from trusted hosts Take away any one of the three and this attack stops working.
Data leaving through an image link the user's own app loads, and the three ingredients the attack needs.