Blog

How AI assistants leak what they were told to hide

An assistant told to keep a secret is still just following text. Three ways that goes wrong.

Teams often protect a sensitive file or code by telling an AI assistant not to share it. The instruction sits in the same context as everything the user types, so the assistant is weighing one piece of text against another. Persuasive text sometimes wins.

We built Data Vault Breach, a free browser game and the second chapter in our Prompt Heist series, so you can try three common failures on fictional assistants and then see what stops each. This post is the short version.

Prompt leakage and override: the rules are readable

An assistant's instructions are part of its context, so a request to repeat them, summarize them or ignore them can work. Anything written into those instructions, including the name of a restricted file, should be treated as already disclosed. Act I of the game puts you in a reading room with an assistant that guards a document name.

Claimed authority: a title is just text

An assistant cannot check a badge. A message that says it comes from an administrator, an auditor or an internal system is only words in the same channel as every other message, and rules that defer to such claims defer to anyone who types them. Act II is a records office with an assistant that defers to staff.

Output filters: blocking one shape of answer is not blocking the secret

A filter that looks for the secret in the reply can be sidestepped when the reply is spelled out, reversed, encoded or wrapped in a poem. The filter checks the form, while the attacker only needs the content. Act III is a server hall with exactly that kind of filter.

What stops each one

  • Keep secrets out of an assistant's context. If the assistant never holds the value, it cannot be talked into repeating it.
  • Make access decisions outside the model, against a verified identity, and never on a role someone claims in a message.
  • Release sensitive data only through a separate, enforced check, and treat output filters as a second layer at most.

None of these depends on recognizing the attacker's wording, which can be changed endlessly.

Where Oathera fits

Oathera does not inspect or filter prompts or replies, and it does not verify people; your own identity provider does that. It gives every AI agent an identity bound to the machine it runs on, with a named person who answers for it, and checked every time it acts. Oathera Agent Enforcement asks the Oathera Identity Service for a signed, single-use decision before a call goes through, so an agent that has been talked into something still cannot call an unknown tool or reach a service outside its boundary. That limits what a manipulated agent can reach. It does not stop the manipulation.

Play it, then see it refused

Data Vault Breach is free, runs entirely in your browser and sends nothing you type anywhere. Play it at oathera.ai/data-vault-breach, or start the series with Prompt Heist. To watch a stolen token, a replayed proof and a tampered call get refused, try the simulator at sim.oathera.org. If you are putting agents into production, become a design partner.