What a confidence score actually means

Start here: it is not a probability

If a vendor's confidence score were a real probability, then across a thousand answers scored 0.80, exactly two hundred would be wrong. That is what calibration means, and essentially nobody in this category has demonstrated it.

Ours is not calibrated either, and we would rather say so on our own website than let you discover it. 0.82 does not mean "82% likely correct." What it means is: a model that read the customer's question, the source passages, and the drafted answer judged the answer to be well supported.

That still makes it useful — high-scoring answers are reliably safer than low-scoring ones, which is all a gate needs to do — but it is a relative signal, not a measurement. Anyone selling it as a measurement is selling you something they have not tested.

How the number is produced

The mechanism is deliberately boring, and the boring part is the point: the model that writes the answer never scores its own work.

1. Retrieve. The customer's message is used to pull the most relevant passages from your help centre and policies. Nothing outside your own content is in scope.
2. Draft. One model call writes an answer using only those passages, and returns the specific passage IDs it relied on. If it decides your content does not cover the question, it says so and the ticket escalates immediately with a knowledge-gap reason — no answer is written at all.
3. Verify the citations. Every ID the model returned is checked against the passages that were actually retrieved for this ticket. Anything that does not match is discarded. If nothing survives, the answer is never sent.
4. Grade. A second, separate model call receives the question, the same source passages, and the draft — and judges whether the draft is genuinely supported by those sources. This call runs with extended reasoning enabled, because it is the guardrail and it is worth the extra cost. Its judgement is the confidence score.
5. Gate. Below your threshold, or flagged unsafe by the grader for any reason, the ticket goes to a human with the reason attached. Above it, the answer sends with its citations visible.

Why a separate grading call matters

Asking a model "how confident are you in what you just wrote?" in the same breath as writing it produces a number that mostly tracks how fluent the answer sounded. Fluency and correctness are different things, and a confidently-written wrong answer is the exact failure this whole design exists to prevent.

Giving a fresh call the sources and the draft — with no memory of having written it — turns the question into something closer to "does this text follow from these documents?" That is a question a model is genuinely decent at, because it is a comparison rather than an act of introspection.

Two different reasons to escalate, and why we keep them apart

A number below the threshold and a grader flagging the draft as unsafe are recorded as separate reasons, because they ask you to do different things.

Collapsing these into one "low confidence" bucket is common, and it destroys the only thing that made the escalation useful. "The AI could not handle it" tells you nothing. "Nine customers asked about warranty and you have no warranty page" tells you what to do on Monday.

Choosing your threshold

The default is 0.75, set per merchant. A sensible way to pick yours:

• Start at the default and run in shadow mode. Every answer is drafted as a private note, so you can read the scores against real tickets without a customer ever seeing one.
• Raise it to ~0.9 for anything that costs money if wrong — refunds, warranty claims, delivery dates, anything that reads as a promise.
• Lower it only with evidence. If you are escalating tickets that were obviously fine, the score is telling you your source article is vague. Fixing the article is usually better than moving the gate.

And treat a high escalation rate as information rather than failure. An agent that answers everything is an agent that is inventing things.

Four questions to ask any vendor

• Where does the number come from? If the answer is "the model", ask whether it is the same call that wrote the answer. Self-scoring is weak and they should be able to say which they do.
• Do you verify citations, or just display them? A model can produce a plausible-looking source reference for a document that was never retrieved. Displaying it unchecked is worse than showing nothing, because it manufactures trust.
• Can I set the threshold, and can I set it per category? A fixed internal threshold means the vendor decided your risk tolerance for you.
• What happens on an internal error? The right answer is "the ticket goes to a human with context". If a tool can fail quietly, the confidence score is irrelevant — you will never see the failure it was supposed to catch.