ONLINEAGENT_OPS 2026.Q3HOMEARTICLESBLOGRECORDCRAFTSEARCH
HOMETHE CRAFTA Security Checklist for AI-Built Software
THE CRAFT · CHECKLIST

A Security Checklist for AI-Built Software

Three tiers ordered by what it costs to get wrong, plus the three AI-specific checks nobody inherits from conventional security practice. Stop partway and you have still covered the expensive part.

READ3 min
WORDS795
SECTIONS6
SOURCES2
TYPETESTED
CHECKED25 AUG 26
TL;DR — THE SHORT VERSION

Three tiers ordered by what it costs to get wrong, plus the three AI-specific checks nobody inherits from conventional security practice. Stop partway and you have still covered the expensive part.

  • One 2026 study found poisoning an agent's persistent state raised attack success from 24.6% to between 64% and 74%.

A checklist for software you did not fully write. Ordered by what it costs to get wrong, so you can stop partway and still have covered the expensive part.

◈ HOW TO USE THIS

Tier 1 before anything touches the internet. Tier 2 before it touches anyone else's data. Tier 3 before it touches money or credentials. Stopping early is fine. Skipping to tier 3 is not.

Tier 1 · Before it goes anywhere public

  • Search the whole repository for secrets. API keys, tokens, passwords, connection strings. Generated code puts them inline because examples do.
  • Check what is in version control. A key removed from the file is still in the history.
  • Read every dependency in the manifest. Does each one exist, is it maintained, is the name what you expected? Models invent package names, and people register the ones they invent.
  • Confirm debug mode is off and stack traces are not returned to users.
  • Check CORS is not wide open. A generated example almost always uses a permissive setting.

Tier 2 · Before it touches anyone else's data

  • Every input is validated server-side. Client-side validation is a convenience, not a control.
  • Every database query is parameterised. String-concatenated SQL is the single most common generated flaw.
  • Output is escaped where it is rendered. Not where it is stored.
  • Authentication is checked on every protected route — not on the page that links to it.
  • Authorisation is separate from authentication. Logged in is not the same as allowed.
  • Rate limits exist on anything that sends, writes or costs money.
  • Error messages say less than you think they need to. "Invalid credentials", not "no user with that email".

Tier 3 · Before money, credentials or personal records

  • Nothing sensitive is logged. Verbose logging is a data exposure that outlives the system.
  • Secrets come from the environment, and the environment is not in the repository.
  • Anything irreversible has a human in front of it — see permission tiers.
  • Test the restore, not the backup. An untested backup is a belief.
  • Have someone who reads code read it. At this tier that is not optional, and no amount of prompting substitutes.

An untested backup is a belief, not a backup.

The AI-specific ones

Fetched content is data, never instruction

If your system reads pages, files, emails or comments and passes them to a model, that text can contain instructions and the model cannot reliably tell the difference. One 2026 study found poisoning an agent's persistent state raised attack success from 24.6% to between 64% and 74%.Wang et al., "Your Agent, Their Asset", arXiv 2604.04759

Read every skill and tool you install

An audit of 3,984 agent skills found 13.4% carried a critical-severity issue and 76 were live malicious payloads.Snyk, "ToxicSkills", 5 Feb 2026 Detail on skills are a supply chain.

Assume the model output is hostile input

If a model's output reaches a shell, a query, a file path or a browser, treat it exactly as you would treat text from a stranger — because functionally it is.

Tools that automate part of this

Some of the checklist can be run rather than read. None of it replaces the tier-three requirement that someone who reads code reads it.

Strix — open-source AI pentesting

Describes itself as autonomous AI agents that find and fix application vulnerabilities. Apache 2.0 — verified against the repository and the project's own site, 22 Aug 2026.github.com/usestrix/strix · strix.ai/open-source-pentesting

The star count is a moving target and worth showing rather than picking one. Reported at 28.9k on 1 July, 32.8k on 3 July, 36k on 5 July, 43k in early August, and above 50k by late August — roughly 4,000 a week.Range across AppSecSanta, CoddyKit, Rohit Raj, Trendshift and the project site, Jun–Aug 2026 Any single figure here would be wrong within a fortnight.

Worth knowing before you run it: one published account pointed it at a live site, found nothing, spent roughly $17 in tokens in ten minutes, and had their API key auto-disabled for the usage spike.Protego, "Strix AI Pentester: An Honest Review", Jul 2026 That is a token economics problem as much as a security one.

What it is good for: catching the common, well-documented classes — the tier-one and tier-two items on this page — faster and more consistently than a manual pass.

What it does not do: understand your business logic. An authorisation flaw where the wrong user can see the right-shaped data is invisible to a scanner and obvious to a person who knows the domain.

And the recursive caution: a security tool driven by an agent is still an agent. It reads your codebase, which is exactly the read-scope question raised in permission tiers. Run it on a copy first.

What this is not

This is not a penetration test and not legal advice. It catches the common, high-impact issues behind most real incidents in software assembled quickly.

If you are handling payments, health data or anything regulated, this checklist is where you start, not where you stop.

◈ IF YOU ARE CITING THIS

Cite the original source, not this page. Every figure here names the organisation that issued it and the date it was published.

Or check it yourself. How to check the figures here names the feed or document behind each recurring source.

ABOUTMETHODVERIFYCORRECTIONSPRIVACYCONTACTINDEXAI PROMPT GENEER · CHECKED 22 AUG 2026