This is a log of one real feature shipping, not a tidy case study written after the fact. I re-enabled a multi-card synthesis feature (see the previous article in this series), tested it, and the first output was weak — generic, threw away the most specific details, opened with meta-commentary. Fixable with prompt tuning. I tuned it.

Then I ran it on an actual contradictory pair of cards — one saying "wait, be patient," the other saying "act now, don't hesitate" — and the model quietly invented a detail. It described one card as "warning against suspicion." Neither source card ever mentions suspicion. It's a real word, sounds plausible, fits the sentence — and it's just made up.

First instinct: tune the prompt harder

I added explicit instructions: name real tension directly instead of resolving it into a fake story, cite specific details instead of vague traits, ground every claim in the actual source text. Real improvement — the meta-commentary went away, the specificity got better. But on the next test, the same fabricated word showed up again, verbatim, in a fresh generation. Prompt tuning had a ceiling.

Second instinct: use a stronger model

The generation was running on a fast, cheap model tier. Switching to the stronger tier seemed like the obvious next lever — more capable models are generally more reliable at following strict grounding instructions.

It ran clean on the first look. No "suspicion." Tighter prose. I almost called it fixed.

The stronger model had fabricated something too — just harder to see

Independent review caught it: the "stronger" output described one card's energy as "a high-voltage current running beneath the surface." Neither source card mentions electricity, current, or anything like it. That's a bigger invention than "suspicion" was — a full sensory metaphor with zero textual basis — just delivered in more confident, polished prose. Confident prose is harder to catch as a hallucination, not easier. A better model raised the polish. It didn't raise the grounding discipline.

That's the actual lesson: model quality lowers the rate of fabrication, it doesn't zero it out. Betting the whole fix on "use a smarter model" means the failures just get better at hiding.

The real fix: a second call whose only job is to check the first one

Same principle as this app's existing architecture for chart readings (see the earlier hallucination-and-architecture piece in this series): generation and verification are different jobs, done by different calls, not trusted to one pass. I added a judge — a second, cheap model call that takes the generated synthesis plus the actual source card text and checks: is every specific detail here traceable to the source, or was something invented?

If the judge flags a problem, the generator gets one retry with the specific violation named. If it's still ungrounded after that, the summary gets dropped entirely — the reveal screen just shows the two cards' real text with no synthesis, which is exactly what it already does today when the network call fails. Nothing broken-looking ever ships; it just quietly degrades to the version that was already correct.

The judge itself wasn't reliable on the first try

This is the part that actually surprised me. The first version of the judge rejected a phrase that was a completely fair paraphrase of real source content — and, in the same test, missed the actual fabricated word sitting two sentences away. It was strict about the wrong thing and lenient about the right thing.

The fix was giving the judge a real calibration example, not just a rule: here is a paraphrase that should pass (different words, same idea, both present in the source), here is an invention that should fail (a new concept the source never touches at all, "suspicion" specifically, since that's the exact case that kept recurring). Concrete examples, not abstract instructions, is what made the judge's own judgment usable.

Where this landed

Two clean runs afterward: no fabricated details, and the judge correctly told apart a fair restatement from an invented one. That's not a permanent guarantee — model output is stochastic, this could still slip through occasionally — but it's a structural check that exists now, checking every generation, not a one-time prompt tweak hoping for the best.

Why this is the actual takeaway, not "pick a better model"

Every fix that looked like progress and wasn't shared the same shape: it made the failure quieter, not rarer. A cheaper model's fabrication was obvious. A stronger model's fabrication was smoother. The fix that actually held up wasn't about which model generates the text — it was about never trusting one generation call to grade its own homework.

Diagram list:

Hero: two model tiers, both flagging a fabricated word, one subtler than the other
A clean-looking output with a hidden, circled flaw
Generation → judge → retry-or-fallback flow diagram