Checking everything is not a strategy. Four risk tiers with a different check for each, the five signals worth looking for specifically, and the monthly questions to ask of your own use.
- These are exactly where failure rates are structurally highest, because rare facts cannot be reliably recalled — the mechanism is on why AI makes things up.
- Check the thing that would be expensive to get wrong, not a sample of the easy parts.
Checking everything is not a strategy — nobody sustains it. Checking by risk is, and it takes a fraction of the time.
Sort by what it costs to be wrong, not by what looks uncertain. The parts that look uncertain are already getting your attention. The dangerous parts are the ones that look finished.
Four tiers, checked differently
Irreversible or public — check everything
Anything sent, published, paid or filed. Read it fully, verify every factual claim, and have someone else read it if it carries your name. The cost of being wrong is not recoverable, so the check is not optional.
Specific and verifiable — check every instance
Names, dates, figures, citations, quotes, URLs, legal references. These are exactly where failure rates are structurally highest, because rare facts cannot be reliably recalled — the mechanism is on why AI makes things up.
A fabricated citation looks exactly like a real one, complete with volume and page numbers.
Executable — check by running, not reading
Code, queries, configurations. Published vulnerability rates in AI-generated code run 25% to 62% depending on model, language and method.Range across studies incl. Snyk 2026 (~48%) and Pearce et al. 2022 (~40%) — see the full correction
Reading code for correctness is slow and unreliable. Running it against a test is fast and definitive. If there is no test, that is the first thing to generate.
Structural or stylistic — spot-check
Formatting, tone, organisation, summaries of things you already know. Errors here are visible and cheap, so sampling is proportionate.
What to look for, specifically
- Confident specifics you did not supply. A figure, a name or a date that arrived from nowhere is the highest-risk sentence on the page.
- Anything that agrees with your premise. If it echoed your framing back, that is not confirmation — see how AI influences you back.
- Round numbers. Real measurements are rarely round; "roughly 40%" often means the source said something else.
- Attribution without a date. "Studies show" and "research suggests" are the shape of a citation, not one.
- Completeness claims. "All", "every", "none" — the hardest thing for a system to establish and the easiest for it to assert.
Check the thing that would be expensive to get wrong, not a sample of the easy parts.
Auditing the process, not just the output
Once a month, worth asking of your own use:
- What did I ship without checking? Not what did I check — what got through.
- What did I catch, and at which tier? If everything you catch is tier 4, you are not looking at tiers 2 and 3.
- Where did it actually save time, net of the checking? Some tasks cost more to verify than to do.
- What would I have noticed if it were quietly wrong for a month? If the answer is nothing, that is the thing to instrument.
The last one matters most, and it is the same failure mode as silent drift in automations: the output that keeps arriving on schedule, still confident, no longer correct.