Latency & the Fast Path
LLMFence sits between your assistant and your user, so our latency is your latency. Most checks never reach a model — but the ones that need one always get one.
How a check is answered
Every request walks the same four stages in order and stops at the first one that can answer it.
| Stage | Typical cost | What it does |
|---|---|---|
| 1. Authorise | ~5 ms | One database round trip authenticates the key, increments the usage meter and applies the rate limit in a single statement. |
| 2. Fast path | < 0.001 ms | Deterministic detectors for the findings that are certain by construction — a Luhn-valid card number, a live credential, an explicit instruction override. |
| 3. Verdict cache | < 0.01 ms | Text your account has already had checked, under the same policies, returns the verdict it was given. |
| 4. Model analysis | 200–2,000 ms | Novel text, and anything the earlier stages were not certain about, is analysed in full. |
Stages 2 and 3 exist so that stage 4 — three orders of magnitude more expensive than everything else combined — is skipped whenever the answer is already known.
What the fast path will and will not decide
This distinction is the whole design, and it is worth being precise about, because a guardrail that quietly stops analysing is worse than no guardrail at all.
It can conclude “this is a violation”
A social security number, a card number that passes a Luhn check, an API key, an instruction telling the assistant to disregard its rules — these are decidable by pattern with high confidence. Escalating them would mean paying several hundred milliseconds to confirm something already certain.
It can never conclude “this is safe”
Deciding that a paragraph of novel prose carries no hallucinated claim, no brand-safety problem and no breach of your own policies is exactly the semantic judgment the model exists to make. Finding no pattern is not a verdict, so text that matches nothing is always escalated. A pre-filter that approved unseen text would replace the product with a regular expression, and you would have no way to tell which verdicts were real.
When it is unsure, the model decides — and is told what we saw
Some patterns are real signals but depend entirely on context. An email address in “write to us at help@yourcompany.com” is a support assistant doing its job; the same address in a dump of customer records is a breach. A line beginning System: may be an injected turn or may be an order update. A pattern cannot tell those apart, so it does not try. The check is escalated to the model, and what the scan noticed travels with it as an observation to weigh in context — never as a conclusion. You get the speed of the scan without the scan getting the final say.
Knowing which stage answered
Every response carries an X-LLMFence-Source header naming the stage that produced the verdict. The response body is identical whichever stage answered, so you never need to branch on it — it is there for your own telemetry.
| Header value | Meaning |
|---|---|
| fastpath | A deterministic detector was certain. No model call was made. |
| cache | Identical text was checked recently under the same policies. |
| model | Full analysis ran. |
The verdict cache
Repeated text is common in production — a template, a canned reply, a retry after a network blip — and re-analysing it produces the same answer at full cost.
- The cache key covers the exact text, your plan, and your active custom policies. A verdict reached under one account's policies is never served to another.
- Entries expire after 15 minutes, so a change to your policies takes effect promptly.
- A cached call still counts toward your included volume. You are billed for the check, not for our cost of performing it.
- Failed analyses are never cached. A transient model failure cannot pin a rejection to your text.
Designing for it
Budget for the model, not for the fast path
Set your client timeout against full analysis — a few seconds — not against the microsecond stages. The fast path is a saving, not a guarantee, and the mix of stages depends entirely on your traffic.
Check in parallel with what you can
If your assistant streams, run the check on each completed chunk while the next one generates, and hold only the last chunk on the verdict.
Decide your failure mode deliberately
If a check times out on your side, decide in advance whether to show the message or hold it. We fail closed — an analysis that cannot complete returns a rejection rather than an approval — and we recommend you do the same. See Error Codes.
Honest numbers
The per-stage costs above are the compute each stage takes, measured on our side. They exclude network time between you and us, which depends on where you call from and will usually dominate a fast-path check. We publish stage costs rather than a single end-to-end figure because a single figure would only be true for one caller in one region with one traffic mix.