Here is the retrieval benchmark. Including the part where it fails.
A retrieval system that answers everything, wrongly, is worse than one that knows what it doesn't cover. So this page publishes the false-positive rate and the failing queries alongside the headline score, not after it.
Queries
232
Knowledge base
45 chunks
Out-of-scope
26
Failing cases shown
all
How to re-run it
git clone https://github.com/djoudad292/ai-virtual-receptionist
cd ai-virtual-receptionist/backend
npm ci
npm run eval
# tighten the gate
EVAL_MIN_F1=0.8 npm run eval
# change the retrieval depth
EVAL_TOP_K=3 npm run evalNo API key, no network, no database. The run is hermetic and deterministic, so you get the same numbers I get. It exits non-zero if F1 drops below the floor, and it runs on every push in CI — so it is a regression guard, not a number in a file.
Threshold sweep — vector (production path)
| Thr | TP | FP | FN | TN | Prec | Rec | F1 | FPR |
|---|---|---|---|---|---|---|---|---|
| 0.20 ← | 198 | 20 | 8 | 6 | 90.8% | 96.1% | 93.4% | 76.9% |
| 0.30 | 196 | 19 | 10 | 7 | 91.2% | 95.1% | 93.1% | 73.1% |
| 0.35 | 188 | 17 | 18 | 9 | 91.7% | 91.3% | 91.5% | 65.4% |
| 0.40 | 174 | 17 | 32 | 9 | 91.1% | 84.5% | 87.7% | 65.4% |
| 0.50 | 147 | 6 | 59 | 20 | 96.1% | 71.4% | 81.9% | 23.1% |
| 0.60 | 78 | 1 | 128 | 25 | 98.7% | 37.9% | 54.7% | 3.8% |
Best F1 is 93.4% at threshold 0.2 — and it is the worst row for false positives. That is the whole point of publishing the sweep instead of the best row.
The finding: F1 is the wrong thing to maximise
| Retriever | Best | F1 | FPR |
|---|---|---|---|
| vector | 0.20 | 93.4% | 76.9% |
| keyword | 0.30 | 91.2% | 42.3% |
The keyword fallback costs 2.2 F1 points and buys half the false-positive rate. For a receptionist workload, where most inbound traffic is genuinely out-of-scope, refusing correctly matters more than recalling one more chunk. So the safer default is not the one that maximises F1 — and a benchmark that only reported the best-F1 row would have hidden that entirely.
| Keyword thr | Prec | Rec | F1 | FPR |
|---|---|---|---|---|
| 0.30 ← | 94.3% | 88.3% | 91.2% | 42.3% |
| 0.35 | 95.9% | 79.6% | 87.0% | 26.9% |
| 0.50 | 96.8% | 73.3% | 83.4% | 19.2% |
| 0.60 | 99.1% | 53.9% | 69.8% | 3.8% |
Where it fails
Sampled from the 8 false negatives and 20 false positives at the best threshold. An audit that only reports wins is marketing.
“amex accepted?”
retrieved: (nothing above threshold)
Terse, all-lowercase, zero stopwords. A real visitor types exactly this.
“What kinds of dental work do you do?”
retrieved: (nothing above threshold)
The broadest question in the set, and the one the corpus answers most directly. The embedder has no concept dimension for it.
“Do you treat pets?”
retrieved: faq-services, faq-whitening, faq-privacy
Tokens 'you', 'treat' overlap with service language. Answering this is a confident wrong answer.
“Do you take passport photos here?”
retrieved: faq-cosmetic, faq-whitening, faq-privacy
'photos' matches the cosmetic consultation's photography. Single-token collision.
“Can you sponsor a visa for your dental assistants?”
retrieved: faq-insurance
Out-of-domain request answered from insurance vocabulary.
“How much does it cost to ship a replacement night guard to Brazil?”
retrieved: faq-nightguard, faq-insurance, faq-mouthguard
Contains a genuine domain noun inside an out-of-domain request. The hardest class of false positive.
What this does not measure
This measures the retrieval policy, not the embedder
The harness uses a deterministic offline embedder (concept lens + hashed lexical residual) so runs are reproducible with no network and no API key. Production uses text-embedding-3-small and degrades to keyword matching. Absolute numbers here are a regression guard for the harness and the threshold — not a score for text-embedding-3-small.
Retrieval, not answer quality
A question counts as answered when a gold chunk surfaces. Nothing here scores whether the written answer is correct, whether the citations support it, or how well an agent chooses between tools. Those are separate measurements and I do not claim them here.
Synthetic queries, not visitor logs
These 232 questions are hand-written paraphrases in five registers — colloquial, formal, terse, multi-clause, typo-laden — authored against this corpus. They are not captured real traffic. They tell you how the retrieval policy behaves under phrasing pressure; they do not tell you what your customers actually ask.
One knowledge base
45 chunks from a single dental clinic. Every figure here is n=1 corpus. A CTO's fair follow-up is whether this generalises across domains and corpus sizes, and the honest answer is that this benchmark cannot answer it.
Every one of those limits is also something I measure when I audit someone else's system — which is what the free five-day audit produces, on your documents rather than mine.
Run it and check my work.
Clone the repo, run one command, and you get the same table. If my numbers here don't match yours, that's a finding worth telling me about.