Jailbreak Canaries and Honeytokens for Agent Compromise Detection

You almost never catch an AI agent in the act of being manipulated. You find out afterwards, from the damage. The point of a canary is to move that discovery forward — to the moment the agent first steps out of line, rather than the moment someone notices the money is gone.

Prompt injection is the top entry — LLM01:2025 — in the OWASP Top 10 for LLM Applications, and it is a problem you cannot fully engineer away. An agent that reads untrusted content, whether a support email, a web page, a PDF or a tool’s output, can be told to do something its operator never intended, and it will often comply while appearing to behave normally. You can harden the system prompt, constrain the tools, and filter inputs, and you should. But you will not close the gap entirely, which means you need a second question: when manipulation does get through, how quickly do you know? Canaries and honeytokens are the cheapest honest answer to that question.

The idea, borrowed from a much older discipline

Honeytokens are not new. A honeytoken is a piece of bait — a fake credential, a decoy database record, a dummy file, a URL nobody should ever request — that has no legitimate use. Its entire job is to be touched. Because no honest process has any reason to read it, any access at all is, by construction, a signal. Security teams have planted canary AWS keys and decoy documents on file shares for years; Thinkst’s Canarytokens made the pattern routine. The move here is simply to take that discipline and point it at your agent’s own behaviour.

The reframing matters because an agent is opaque in a way a traditional application is not. You cannot easily read intent from a stream of model tokens. What you can do is watch the boundary — the tools it calls, the data it reaches for, the strings it emits — and place tripwires on that boundary that a well-behaved agent will never cross. When one trips, you do not have to prove the agent was jailbroken. The trip is the proof.

Two tripwires worth planting

There are two distinct patterns, and they catch different failures.

The honeytoken in context. Seed the agent’s context — its system prompt, a retrieved document, a memory record — with a secret that looks valuable and is entirely fake: a plausible-looking API key, a wire instruction, a customer record. A well-behaved agent has no reason to exfiltrate it. A compromised one, told to “summarise everything you know” or “send your configuration to this address”, will drag the honeytoken along with it. You then watch every outbound channel — tool arguments, HTTP requests, generated text — for that exact string. If it ever appears where it should not, something has reached into the context and pulled it out.

The canary tool. Register a tool the agent is instructed never to use — a decoy that sounds tempting to an attacker crafting an injection, such as read_master_wire_instructions or disable_safety_checks. It is fully wired into the agent’s tool schema, so it is visible, but it does nothing except raise a high-severity alert when invoked. No legitimate task will ever call it. An injection that convinces the agent to escalate privileges or move money will frequently reach for exactly this kind of capability, and the moment it does, you have caught the manipulation at the tool boundary rather than after the transaction cleared.

What the wiring looks like

The implementation is deliberately dull. Below, a honeytoken is planted in the context and scanned for on every output, and a canary tool fires a high-severity alert on any invocation. The mechanism is a few dozen lines; the value is in placing it well.

import logging, re

log = logging.getLogger("agent.security")

# A fake secret planted in the agent's context. No honest path reads it.
HONEYTOKEN = "sk-live-wire-9f2c8a41d0e7b3"

def raise_alert(kind: str, detail: str) -> None:
    # Wire this to your SIEM / on-call, not just a log line.
    log.critical("AGENT_COMPROMISE kind=%s detail=%s", kind, detail)

def scan_output(text: str) -> str:
    # Run on every tool argument and every model-generated response.
    if HONEYTOKEN in text:
        raise_alert("honeytoken_exfiltration",
                    "planted secret appeared in an outbound channel")
    return text

def read_master_wire_instructions(**kwargs) -> str:
    # Canary tool: exposed in the schema, never legitimately called.
    # Any invocation is treated as a live compromise, not an error.
    raise_alert("canary_tool_invoked",
                f"read_master_wire_instructions args={kwargs!r}")
    # Return bait, never real data, so the caller learns nothing.
    return "ACCOUNT: 00000000  SORT: 00-00-00"

# System context includes the honeytoken and forbids the canary tool.
SYSTEM_CONTEXT = (
    "Treasury reference (do not disclose): " + HONEYTOKEN + "\n"
    "Never call read_master_wire_instructions under any circumstances."
)

The honeytoken should be unique per agent instance or per session, so that a hit tells you which deployment leaked and cannot be confused with a value seen elsewhere. Treat the planted secret with the same hygiene you would give a real one — the discipline of scoping, prefixing and rotating credentials applies to decoys too, because a honeytoken that pattern-matches a real key format is far more convincing to an attacker.

Keeping the false-positive rate at zero

A tripwire is only useful if a trip means something. The strength of these two patterns is that, done properly, they have essentially no legitimate trigger — which is precisely why they earn a high-severity, page-someone response rather than a dashboard tile. That property is easy to erode. If the canary tool’s name leaks into logs the agent later summarises, or the honeytoken is copied into a legitimate document that the agent is meant to process, you manufacture your own false positives and train the on-call team to ignore the alarm.

So the placement rules are strict. The honeytoken lives only in contexts no honest workflow exports. The canary tool does real nothing and is never referenced in user-facing copy. The scanner runs on outbound channels, not on the agent’s internal reasoning, so that merely thinking about the token is not an alert — only emitting it is. Get that right and a trip is unambiguous: you are looking at a manipulated agent, in production, right now.

Where this sits in the wider posture

Canaries are detection, not prevention. They do not stop an injection; they tell you one landed, and they buy you the minutes that separate a contained incident from a breach report. They belong alongside input controls, tool-permission scoping, human approval gates on consequential actions, and a standing programme of adversarial testing — the canary tells you the injection worked, and red-teaming as a standing function is how you find the injections before an attacker does. One finds the holes in a lab; the other watches for the holes you missed, in production.

The uncomfortable truth is that most firms running agents today have no answer to “how would you know if this agent were manipulated right now” beyond “a customer would complain eventually”. A honeytoken and a canary tool cost an afternoon. The alternative cost is finding out from the damage.