Trust, but Verify: Quality Control

How automations actually fail

"How do I know my automations aren't quietly making mistakes?"

That question — the fact that you're asking it — is the mark of someone ready to run a real system. And the honest answer starts with understanding the failure mode, because it's not what most people picture. Automations rarely fail loudly. There's no error message, no crash, no smoke. They fail quietly, gradually, and confidently: the drafts get a little more generic, the summary misses one meeting's nuance, a template references something that changed last month, an extraction skips an item — and everything *looks* fine, because AI output always looks fine. Confidence is the costume; drift is the failure.

Why does drift happen? The world changes around your instructions: your prices update but a template doesn't; a new client type arrives that your criteria never met; your own standards rise while your samples stay old; the input format shifts subtly. None of these are AI 'breaking' — they're a fixed system meeting a moving world. Which is exactly why the answer isn't anxiety or blind trust. It's process: the same boring, reliable verification that professionals wrap around anything important. Trust, but verify — with a schedule.

Here's the good news: you've been doing informal QC since Module 2 (the test log), Module 3 (spot-check new plumbing like a new employee), and every review gate since. This lesson formalizes what you've practiced — tiers, checks, logs, audits — so 'is it still working?' becomes a question with a documented answer.

QC tiers: review depth scaled to stakes, formalized

You've applied 'review scales with stakes' intuitively since Module 3. Now it becomes a written property of every workflow in your System Doc: its QC tier. Three tiers cover everything:

The canary signs

Between scheduled checks, drift announces itself early — if you know the signs. Train your eye on five canaries, each cheap to catch and expensive to miss:

Outputs getting shorter or more generic — the most common drift: your instructions have gone stale relative to the inputs, or an asset (Voice Brief, library) needs refreshing. Format wobble — the summary that always had three sections shows up with two: something changed in the input or the instruction chain. Rising fix rate — your review edits are growing week over week: the improve-loop has stopped feeding fixes back into the instructions. The too-smooth week — zero catches across many outputs sometimes means excellence, and sometimes means your review attention drifted; run one deliberate close-read to find out which. The stale reference — an output mentions a price, name, or fact you KNOW changed: stop, because every workflow drawing on that asset is now suspect — update the source, then spot-check its dependents (your system map from Lesson 1 tells you exactly which ones).

Notice the pattern: every canary is caught during normal use — no extra sessions, no paranoia. QC isn't a new job; it's sharpened attention during the reviews you already do.

The error log and the fix-the-instruction habit

When review catches something — and it will; that's the gate doing its job — two moves turn the catch into system improvement instead of a private grumble.

Log it. One line in the System Doc's error log: date · workflow · what was wrong · root cause if obvious. Ten seconds. The log's value isn't the individual entries — it's the pattern across entries: three logged catches from the same workflow in a month is a workflow asking for instruction surgery; three catches all tracing to one stale asset is a source-of-truth problem; a log that's filling up faster each month is a system outgrowing its instructions. You can't see any of this without the log — memory flattens patterns into vague unease.

Fix the instruction, not just the output. The habit you learned with Maria's 'never invent deadlines' in Module 2, now stated as law: *every correction you make twice becomes a rule in the standing instructions.* Fixing the output solves today; fixing the instruction solves forever. During your Weekly System Check, scan the week's log entries and ask of each: what one line, added to which instruction block or asset, would have prevented this? Add it, date it, done. This is the entire secret of systems that get better with age — and the reason your six-month-old automation will outperform your six-day-old one.

The monthly audit — and knowing when to pause

The last formal piece: the monthly audit — thirty minutes, calendar-scheduled, deeper than the weekly check. The run-sheet: (1) Tier review — does every workflow still sit in the right tier? Clean Tier-1 track records can graduate to Tier 2; anything with recent catches moves up, not down. (2) Deep-read sample — pick two workflows and close-read their last few outputs like a first-time reviewer; fresh eyes catch what routine misses. (3) Log patterns — read the month's error log for the patterns above; schedule instruction surgery where it's asked for. (4) Asset freshness — skim the high-traffic assets (price sheet, libraries, Voice Brief samples): anything stale gets updated NOW, because of what stale sources do downstream. (5) The retirement question — is anything running that no longer earns its keep? Retiring a workflow that stopped mattering is system hygiene, not failure.

And the last skill of the lesson — the one that makes you a trustworthy operator: knowing when to pause. Pause an automation immediately when: its error rate jumps and the cause isn't obvious · its source assets are known-stale and you can't update them today · the world just shifted under it (price change, policy change, big news in your niche) · or a catch reveals a failure mode you don't yet understand. Pausing is not defeat — it's the review gate applied at system scale: the workflow goes back to fully-manual (you still know how; you built the recipe) while you diagnose calmly. A system you're willing to pause is a system you actually control. That control question — what should run at all? — is exactly where the next lesson goes: the honest art of NOT automating.