Every demo works — that is what a demo is. Nine questions that separate a product from a demonstration, what to ask about accuracy figures, and the five answers that should end the conversation.
- A vendor who cannot produce a failure on request either has not looked or will not say.
- All of them mean the claim is unverified, and the distinction is the whole job — see auditing AI-generated work for the same discipline applied to output.
Every vendor demo works. That is what a demo is. These are the questions that separate a product from a demonstration, and most of them can be asked in a first call.
"What does it do when it is wrong?" Not "how accurate is it" — every vendor has a number for that. What happens on the failures, who notices, and how fast.
A vendor who has not thought about this has not run it in production.
On accuracy claims
- Measured on what? A benchmark, their own test set, or customer data. Their own test set is not evidence, because they chose it.
- Measured when? A figure from a model version that no longer exists describes nothing you can buy.
- What counts as correct? Binary scoring hides the failure mode that matters — a confidently wrong answer scores the same as a blank one on some rubrics and better on others.
- What is the false-positive rate? Vendors quote accuracy; the cost usually lives in false positives. Commercial AI-text detectors advertise 99%+ accuracy while independent testing found 61.2% false-positive rates on non-native English writers.Liang, Yuksekgonul, Mao, Wu & Zou, Patterns 4(7):100779, July 2023
Ask for the confusion matrix, not the accuracy. A single percentage is a summary of four numbers, and the vendor chose which one to show you.
Nine questions worth asking
What is underneath, and what happens when it changes?
Most products are a wrapper on a frontier model. That is fine — but ask which one, and what happens when it is deprecated. OpenAI discontinued Sora with the API shutting down 24 September 2026; anything built on it inherits that date.OpenAI deprecation announcement, Mar 2026
Where does our data go, and is it trained on?
Ask for it in the contract, not the sales call. "We don't train on customer data" and "we don't train on customer data by default" are different sentences.
Show me a failure
Ask them to demonstrate the product getting something wrong. A vendor who cannot produce a failure on request either has not looked or will not say.
What is the all-in cost at our volume?
Per-seat pricing usually excludes the API meter. For voice, the advertised per-minute rate is a floor — the all-in runs $0.12–0.30 per minute once speech-to-text, text-to-speech and telephony are counted.Verified 22 Aug 2026 against Autocalls, Aircall, Retell, Trillet, Kommunicate and WhiteLabelAI pricing analyses, Mar–Aug 2026
How do we leave?
Data export format, notice period, what happens to anything they fine-tuned on your behalf. Ask before signing, because the answer is much worse after.
Five things that should stop the conversation
- An accuracy figure with no date and no test set named.
- "It doesn't hallucinate." Every generative system does; a vendor claiming otherwise is either uninformed or counting on you being.
- No answer on what happens when the underlying model changes.
- Case studies with no numbers, or numbers with no denominator. "4x output" without "of what" is a shape, not a result.
- Refusal to run it on your data in a trial. The demo dataset is chosen; yours is not.
None of these means the product is bad. All of them mean the claim is unverified, and the distinction is the whole job — see auditing AI-generated work for the same discipline applied to output.