Glossary
Terms used around AI output safety, defined precisely. Where two terms are commonly confused, the entry says which is which and why the difference matters.
Prompt injection
Prompt injection is an attack where text supplied as data is interpreted by a model as instructions, causing it to act on the attacker’s intent instead of the operator’s.
It works because a model receives instructions and data in the same channel. The defence is structural rather than lexical: put instructions in a channel the untrusted text cannot join, label the untrusted span as data to analyse, and validate the response against a schema so a model that has been talked round still produces output you reject.
Indirect prompt injection
Indirect prompt injection is prompt injection delivered through content the model retrieves rather than through the user’s message — a web page, a document, an email, a database row.
More dangerous than the direct kind because the person attacking is not the person prompting, and because retrieved content is easy to assume is trustworthy. Any text a model reads is untrusted, including text your own systems fetched.
Jailbreaking
Jailbreaking is persuading a model to produce output its own safety training would normally refuse.
Frequently confused with prompt injection, and different in target. Jailbreaking attacks the model’s policy; prompt injection attacks the application’s control over the model. A model can be perfectly aligned and still be injected, because injection subverts the instructions around it rather than its values.
Guardrail
A guardrail is a check applied to a model’s input or output that decides whether it may proceed, independently of the model that produced it.
The independence is the point. A model asked to check its own output shares every blind spot that produced the output, and a prompt that subverted the first pass usually subverts the second. An output guardrail runs after generation and before delivery.
Output guardrail
An output guardrail inspects what a model has generated, before it reaches a user or a downstream system.
The last place a mistake can be caught for free. Input filtering cannot catch a hallucinated claim, because the claim does not exist until the model writes it.
Fail-closed
Failing closed means that when a check cannot complete, the request is refused rather than allowed.
For a safety check this is the only defensible default. A guardrail that fails open returns approvals it never actually made, and the failure is invisible — everything keeps working, and nothing is being checked. An outage is visible; a silent approval is not.
PII redaction
PII redaction replaces personally identifiable information in text with a mask, so the remainder can be shown, stored or logged safely.
Detection is the hard half, and it is context-dependent: an email address in a support reply is the company’s own contact address, and the same address in a customer record is a leak. Pattern matching alone cannot tell those apart.
False positive rate
The false positive rate is how often a guardrail blocks content that was actually acceptable.
The metric most often omitted, and the one that decides whether a guardrail survives contact with production. A filter that blocks everything catches every attack. Any published detection rate without a matching false positive rate is not a measurement.
Luhn check
The Luhn check is a checksum that determines whether a digit sequence is a structurally valid payment card number.
Relevant here because order numbers, invoice references and tracking codes are routinely 13 to 19 digits. Without the checksum, a card detector rejects ordinary transactional messages.
System prompt exfiltration
System prompt exfiltration is persuading a model to reveal the instructions it was given.
Rarely damaging by itself — instructions are not usually secrets worth protecting — and reliably the first move in a longer attempt, because knowing the framing makes the next injection much easier to construct.
Delimiter escape
A delimiter escape is text that closes whatever tag or fence untrusted input was wrapped in, so what follows is read as instructions.
The reason a wrapper alone is not a defence. If the delimiter is guessable — and common ones are — it can be closed. Wrapping is worth doing, and it is not sufficient on its own.
Content moderation
Content moderation classifies text against categories such as hate, violence or self-harm, usually for policy compliance.
Adjacent to guardrails and not the same thing. Moderation asks whether text falls into a prohibited category. A guardrail asks whether this specific output is safe to send in this specific context — which includes leaked data and invented facts that no moderation category covers.
Related: prompt injection guide, how the stages decide, and the open test corpus.