Most AI pilots die between the demo and production. That’s the part I do.
Only 25% of enterprises have moved 40% or more of their AI pilots into production. Deloitte, 21 Jan 2026 The rest are stuck somewhere between a convincing demo and something you can be paged for. This audit measures where exactly they’re stuck.
What gets measured
Retrieval and authorization audit on your own documents. Precision/recall/F1 against a goldset built from your corpus, refusal rate on out-of-scope questions, citation correctness, p50/p95 latency and token cost, and a server-side authorization review.
Precision, recall and F1
Measured against a goldset built from your corpus, not a demo set. If retrieval misses, the answer is downstream of a known-bad input — this is the first thing to check.
Refusal rate on out-of-scope questions
The number that tells you whether the system knows what it doesn't know. A system that answers everything has no boundary, and you will find that out in production.
Citation correctness
Does the answer point at the passage that actually supports it, or merely at a passage from the same document? These are different, and only one of them is useful.
Latency and cost
p50 and p95, and token cost per conversation, so the economics conversation happens against real numbers instead of an estimate.
Server-side authorization review
Retrieval that ignores permissions is the fastest way an AI pilot exposes data it should not. This is checked before anything else.
Failures, published alongside the scores
The report lists what broke. An audit that only reports wins is marketing, and you would be able to tell.
What you get
- A written report — PDF and markdown — that you own outright
- The goldset built from your corpus, in your repository, so you can rerun it
- The measurements themselves, not just the conclusions
- The permission findings, listed explicitly
- A recommendation, including "you don't need this yet" if that's the truth
Failures are published alongside the scores. Credited in full against any build that follows.
Who it's for
Worth requesting if
- A mid-market team whose AI feature works in a demo and stalls on real traffic
- An agency with a client-facing AI system nobody has independently measured
- A team about to launch and wanting to know what breaks before customers find it
- Anyone who has been told a system is 'working' and has never seen the numbers
Not worth it if
- You want a demo, a strategy deck, or a sales call. This is none of those three.
- You have no documents and no corpus to score against. There is nothing to measure yet.
- You want a guaranteed accuracy figure in advance. I will not commit to a number I haven't measured on your data.
- You need the answer before five working days. Then this isn't the right entry point.
Eligibility is simple: you need a real corpus, a real system to point at it, and someone who owns the process for the five days. I review every audit request personally and decline most. That is the filter — not a form.
What it is not
- Not a demo. It runs against your data, not mine.
- Not a strategy deck. There is no roadmap in it.
- Not a sales call. If the report says you don't need this yet, that's the report, and you keep it.
- Not a commitment to work together. Scoping a build without looking at your data is guessing, and you would be paying for the guess.
The reason this is free: it replaces the sales pitch with the work. Most studios argue from slides and case studies, and you can argue back from a report. If the answer is that you should not build this yet, the report will say that and you will still have the map.
Send me something to measure.
A folder of documents, a link to a knowledge base, or a description of the system that isn’t performing. Five working days later you get the report. If you want to keep building afterwards, the audit comes off the price — full builds start at $3,500.
Reply within one business day — including “this isn’t for you”.