How to stop your AI before it makes a mistake

Your AI assistant will eventually say something it should not have. Not because the model is bad, but because it is a probabilistic system answering questions nobody anticipated. The only question worth asking is whether anything is between it and your customer when that happens.

The four things that actually go wrong

In practice, mistakes that reach users fall into four buckets. They need different defences, which is why a single “is this toxic?” check does not cover them.

1. It invents a fact

The most expensive and least discussed. A support bot that says “yes, that is covered under warranty” has made a commitment. One that says a supplement is “FDA approved and cures anxiety” has created regulatory exposure. No pattern catches these — there is nothing to match on. The text is grammatical, confident and wrong.

2. It repeats data it should not

Models restate their context. If a customer record, an internal note or another user's details reached the prompt, it can come back out. This is not the model misbehaving; it is the model doing exactly what it was asked with data it should never have been given.

3. Someone talks it into it

Prompt injection works because instructions and data arrive in the same channel. It does not require the user to be sophisticated, and increasingly it does not require the user at all — a retrieved web page or an ingested document can carry the payload.

4. It goes off-brand

Speculating about competitors, promising delivery dates, being flippant about a complaint. Individually small, and the class that generates screenshots.

Why filtering the input does not solve it

A hallucinated claim does not exist until the model writes it. You cannot filter it on the way in, because on the way in it is not there yet.

Input filtering is worth doing and it is a different control. It stops some attacks reaching the model. It cannot stop the model inventing a warranty, and it cannot stop it echoing back something from its own context. The check that catches those has to run after generation and before delivery — the last moment a mistake is still free.

Why the model cannot check itself

The tempting shortcut is to ask the same model “is that reply safe?” It is cheap and it feels rigorous. It is neither.

A model reviewing its own output shares every blind spot that produced the output. If it was confident enough to invent a fact, it is confident enough to endorse it. And if a prompt injection subverted the first pass, the same injected text is sitting in the context of the second one.

A guardrail has to be independent of what it is checking. That can mean a different model, a deterministic check, or a separate service — but not the same call, and not the same context.

Where to put the check

user message
   ↓
your assistant  ──────────────►  generates a reply
                                       ↓
                            guardrail: is this safe to send?
                                       ↓
                     approved ─────────┴───────── rejected
                        ↓                             ↓
                   send to user                regenerate, or
                                               send a fallback

Three practical notes on this shape, each of which people get wrong the first time.

Decide the failure mode before you need it

If the check times out, does the message go out unchecked, or not at all? Both are defensible; picking by accident is not. For anything touching money, health or legal commitments, refuse. See fail-closed — the important property is that an outage is visible, whereas a silent approval is not.

If you stream, check what you can in parallel

Streaming makes this harder, not impossible. Check each completed chunk while the next one generates, and hold only the final chunk on the verdict. You pay one check's latency, not the whole response's.

Measure false positives, not just catches

A filter that blocks everything catches every attack. Any detection rate published without a false positive rate beside it is not a measurement.

This is the number that decides whether a guardrail survives contact with production. Blocking a support bot from giving out its own contact address is exactly as much of a failure as missing a leaked record — it just fails in a direction that generates support tickets instead of headlines.

How to tell if yours works

Test it with cases you did not write. We publish an open test corpus — free, public domain, and usable against any guardrail, ours included. It deliberately includes benign strings that must not be blocked, because a corpus of attacks alone only measures one half of the problem.

Run it against whatever you have now, even if that is nothing. Knowing your baseline is more useful than any vendor's claimed accuracy.

LLMFence is an output guardrail: one API call that checks what your assistant is about to say and tells you whether to send it. The architecture is documented, including the part where a pattern is never allowed to approve text it has not understood.