The Limits of AI-Written Code: How Far You Can Trust It

"It runs" is not "it's right"

You ask an AI for a function. It writes twenty confident lines. You paste them in, hit run, and it works. Job done?

No. Running proves the code is syntactically valid. It says nothing about whether the code is correct. Those are completely different claims, and the gap between them is where AI-written code quietly hurts you — days or weeks later, on an input you didn't try, in a way that's expensive precisely because everything looked fine.

This lesson is the mental model for using AI to write code heavily while staying the person responsible for it being correct — which you can be, but only if you know exactly how it fails.

Why nobody explained it this way

Because AI coding tools are sold as magic, and magic doesn't come with a failure-modes appendix. The demos always work. The marketing says "build an app without knowing how to code." What it doesn't say is that the model produces plausible code — code that looks like correct code — and that plausible and correct part company in specific, repeatable ways that a beginner has no way to spot until they've been shown them.

---

1. What the model is actually doing

An AI code generator is predicting the most likely next tokens given your prompt and the millions of code examples it learned from. It is extraordinarily good at producing code that resembles code that solves your problem.

But notice what it is not doing: it isn't reasoning about your specific system, running your code, checking it against a spec, or verifying correctness. It's pattern-completion, not proof. So its output is plausible by construction — and plausibility is exactly the property that makes wrong code hard to catch, because wrong code that looks wrong is easy; wrong code that looks right is the dangerous kind, and that's the model's specialty.

---

2. The failure modes, by name

You can't audit what you can't name. Here are the ones that actually bite:

The through-line: the model optimises for looking done, and "looking done" means the common case, written confidently. The uncommon case — which is where bugs and security holes live — is exactly what it under-serves.

---

3. Why "it runs" and even "it passes" fool you

Two false comforts, ranked by how much they should comfort you:

• "It runs." Comforts you the least it should. Running checks grammar, not meaning. A sentence can be grammatically perfect and factually false; so can code.
• "It passes my tests." Better — but tests only prove behaviour on the cases you thought of. The AI's blind spots and your blind spots overlap heavily (you both forgot the empty-list case), so your tests are least likely to cover exactly what the AI most likely got wrong.

There's a documented, uncomfortable twist here: research (Perry et al. at Stanford) found that people with an AI assistant sometimes wrote more insecure code while feeling more confident it was secure. The tool doesn't just occasionally err; it can quietly lower your guard at the same time.

---

4. The audit protocol

Treat every piece of AI-written code as a pull request from a fast, tireless junior who never says "I'm not sure." You are the senior reviewer, and your name goes on it. Before you trust it:

1. Read every line and explain it out loud. If you can't say what a line does and why, you can't ship it. This alone catches most confident-wrong-logic bugs.
2. Verify the APIs are real. Check that every function and parameter it called actually exists, in the version you're using. (AI-07's hallucination-hunting is the same reflex, applied to code.)
3. Test the edges and the failures on purpose. Empty input, zero, negative, huge, malformed, the network down. Don't test the happy path it already handles — attack the cases it skips.
4. Scan for security smells. Any secret in the file? Any string-built query? Any unvalidated user input reaching something dangerous? These recur; learn to spot them on sight (OWASP is the map).
5. Never paste real secrets or private data into the prompt to get help — that's a leak, not a convenience.

---

5. The trust ladder: match verification to stakes

You don't audit everything to the same depth. Trust more where a mistake is cheap; verify harder as the blast radius grows:

• Trust more: throwaway scripts, boilerplate, code you'll read and run once, low-stakes glue where a bug just means "rerun it."
• Verify hard: anything touching security, money, personal data, or that others depend on. The more it matters, the more of the protocol you run — and the less you let "it looks right" stand in for "I checked."

The skill isn't "trust AI" or "distrust AI." It's calibrating: how much would a subtle bug here cost, and have I verified in proportion to that?

---

What this means for you

AI can write most of your code, and you can go far faster with it — and you remain the one accountable for whether it's correct, because the model won't be. Held that way, it's a huge multiplier: a junior who types instantly and never tires, reviewed by a human who understands the stakes. Held the other way — "it ran, ship it" — it's a machine for producing confident, plausible, occasionally dangerous mistakes, faster than you could make them yourself.

---

Try it: audit AI-written code (30 min)

1. Ask an AI to write a small but real function — for example: "parse a CSV of names and amounts, and return the total per person," or "validate and store a user-submitted email." Keep it small enough to read fully.
2. Read every line and explain each one in a comment or a note. Mark any line you can't fully justify.
3. Hunt for at least two real problems. Aim across categories: one unhandled edge case (empty input, a malformed row, a duplicate) and one security or assumption issue (unvalidated input, a silent assumption about the data, a hardcoded value). Prove each by feeding it an input that breaks it.
4. Fix them yourself and re-test with the breaking inputs until they pass.
5. Write it up: the two problems you found, the input that exposed each, and the fix — three or four sentences.

✅ Finish check: you have one AI-written function, at least two concrete problems you found and reproduced, the fixes applied, and a short write-up. You audited code instead of trusting it.

---

Summary card

• "It runs" proves syntax, not correctness. "It passes my tests" only covers cases you thought of — which overlap the AI's blind spots.
• The model produces plausible code by pattern-completion; plausible-but-wrong is its most dangerous output because it looks right.
• Named failure modes: confident wrong logic, hallucinated APIs, insecure patterns, outdated idioms, silent assumptions, happy-path-only.
• AI assistants can raise confidence while lowering security (Stanford, 2023) — so guard against the comfort, not just the code.
• Audit protocol: read and explain every line · verify APIs are real · attack edges and failures · scan for security smells · never paste secrets.
• Trust ladder: cheap mistakes → trust more; security/money/data → verify hard. You're the accountable senior reviewer, always.

---

Sources

1. OWASP — Top 10 and secure coding guidance
2. Perry et al. (Stanford) — Do Users Write More Insecure Code with AI Assistants? (2023)
3. NIST — Secure Software Development Framework (SSDF)
4. Model provider documentation on limitations and hallucination (Anthropic, OpenAI)

---

Next lesson: TC-13 — Building a Project End to End: Idea to Live (L3) Related: AI-07 Hallucination Hunting (L2) · TC-04 Python From Zero (L1) · TC-05 Your First Real Script (L2)