A few weeks ago the South African government quietly withdrew its first national AI policy. The AI used to help draft it had cited academic papers that were never written: real journals, named correctly, with studies inside them that didn't exist. It had already cleared Cabinet by the time anyone noticed.
It's easy to laugh at, and plenty of people did. But it's worth being clear about how it actually happened, because it's the way AI fails most often in real work. The policy didn't fail loudly. It read like a finished document: confident, properly formatted, plausible. So nobody thought to check the sources.
That's the failure that should worry anyone putting AI near real work. The model that breaks or says something obviously wrong is easy to catch. The dangerous one hands you a clean, professional answer that happens to be false, and reads well enough to sail through. A made-up figure in a customer report or a board pack sticks. It becomes a number someone repeats, with your name on it, long after you could quietly fix it.
We spent a few weeks on exactly this problem, so I'll tell you what we did, plainly, because the how is the useful part.
What we did
One of Africa's leading financial-software companies came to us with years of their content to move into a new brand voice: case studies, blog posts, and customer testimonials.
The testimonials are the part that made it hard. The figures and the quotes in them had been signed off by their clients' own executives, at large enterprises, for this company to publish. Change one number or soften one line, and you're not rewriting content any more. You're putting words in a customer's mouth they never approved. For a business whose credibility rests on those relationships, that goes well beyond a typo. You've published a fabricated testimonial under a real customer's name. So nothing could change. That was the brief.
The temptation, and it's a real one now that the tools are this good, is to hand the whole pile to a capable model and let it rewrite. It would read beautifully. It would also, somewhere in that pile, quietly round a figure or soften a quote, and you'd find out when a client did. The more fluent the model, the more convincing the one wrong line. It's the same trap the policy fell into, scaled down to a content library.
So we built it the other way around. We constrained what the AI was allowed to do rather than reaching for a cleverer model. Here's what that means, part by part. Each one closes a specific way the thing goes wrong.
Rules run before the model
Before any AI rewrote a word, a plain deterministic check went first, built from fixed rules rather than another model. It holds the exact figures the company stands behind, the quotes that can't move, and a banned-word list, and it catches a wrong number to the last cent before a model is ever asked for an opinion. The cheap, boring check runs first; the clever one only runs on what passes. What it solved: a wrong figure is caught before it costs anything.
The writing happens in stages
A single long rewrite drifts. The model loses the thread halfway, and the back half quietly contradicts the front. So the work moves in small steps, each short enough to stay honest. What it solved: the coherence cliff, where a long answer slowly stops agreeing with itself.
Judging is kept separate from writing
The thing that drafts shouldn't be the thing that grades it. Four critics read each draft on their own terms (fidelity, voice, essence, persona), and an arbitrator settles them on one fixed rule: fidelity outranks style. A sentence that reads beautifully but bends a fact gets cut, and a plainer, true one takes its place. What it solved: polish stops winning arguments against accuracy.
A person signs off last
At the end the run stops and puts everything in front of a human: the original with its protected quotes, the critics' notes, every revision, the final draft, to approve, edit, or send back. That gate never comes off. What it solved: nothing reaches the outside world on the model's say-so alone.
None of that is glamorous. It's the unglamorous part of AI, and it's the part that let us hand back work that came back with no review notes.
The rules did more than keep us honest. They let us move. Once the pipeline could be trusted not to touch a figure or a quote it shouldn't, we could run the whole back catalogue through it at volume: over 300 pieces rewritten in a matter of days. The same job by hand, with someone checking every number against the source, is months of work. The discipline is what let us run it fast, because you only get to go quick once you can trust what comes out.
Why a smarter model isn't the fix
The instinct after a story like the policy is to reach for a smarter model, one more accurate, less prone to making things up. We've found that doesn't fix it. These tools are already fluent enough to be believed, and a more capable model mostly makes a wrong answer more convincing. The model that drafted that policy could write perfectly well. It just couldn't tell which of its sentences it was allowed to be sure about.
That's the gap our work lives in. A model writes; it doesn't check facts. It can't tell a fact it knows from one it invented, because both come out in the same confident voice. The judgment has to come from somewhere else: the rule that checks before it speaks, the trained "I don't know," the person who signs off. Anyone can buy the model now; they're a commodity. The discipline around it is the part that's actually yours.
If you're using AI where it matters
A few honest questions, while the answer can still be no.
- What checks the facts before the model speaks? If the only thing between a number and a customer is the model's own confidence, that's a hope, not a check.
- Is "I don't know, get a human" something it's actually trained to do? If it always has an answer, it will eventually invent one.
- Who signs off last? If you can't name the person, the gate isn't there.
The withdrawn policy had a review step. People read it. It still got through, because the only check was someone trusting that it looked right. A skim isn't a gate.
We'd rather show you the boring gates than the clever demo: the check that runs first, the critics that rank truth over polish, the named person who approves it last. It's the least glamorous work in AI, and it's the part that lets you put your name on what comes out.
It's time to clean up.
The Fabrication Risk Scorecard gives you a read on where your own AI could do this: eight questions, about two minutes. Or book a call and we'll walk your highest-stakes use case together.