Testing a feature that never gives the same answer twice.

The first test anyone writes against an AI feature is an equality assertion on the model's output. It passes locally, fails in CI that evening, gets labeled flaky, and quietly gets deleted — and the team walks away convinced AI features can't be tested. They can. You just aimed the test at the one component that will never promise the same answer twice.

Draw the line at the API call

An AI feature is two systems glued together: a nondeterministic model you don't control, and a pile of perfectly ordinary code you do — prompt assembly, request shaping, response parsing, validation, retries, fallbacks, error mapping. The surrounding code is where most of the feature actually lives, and it's where the failures that page you live too: the parser that chokes when the model wraps its JSON in code fences, the retry loop that hammers a rate-limited endpoint instead of backing off, the handler that swallows a billing error and shows "something went wrong."

None of that needs a live model to test. Stub the API call with a canned response and the entire scaffold becomes deterministic. Feed it a fenced reply and assert the parser strips the fences. Feed it a rate-limit error and assert the code moves to the next model in the fallback list instead of retrying the dead one. Feed it an over-length answer and assert the result gets trimmed to the field limit rather than shipped raw. These are unit tests in every ordinary sense — fast, cheap, run on every commit — with the one unpredictable component replaced by a fixture.

Collect the model's bad days

Where do the fixtures come from? Production. Every time the model does something your code didn't expect — a friendly preamble before the JSON ("Sure! Here's the layout you asked for:"), a paragraph of commentary after it, an empty response, an answer cut off mid-token by an output cap — that reply is a specimen. Save it verbatim as a test fixture and make the parser handle it.

This is the regression suite most AI features are missing. The model won't repeat itself token for token, but it absolutely repeats its failure modes — fenced JSON isn't a one-off, it's a habit. A parser hardened against twenty real malformed replies is worth more than any amount of prompt engineering that asks the model nicely not to produce them.

Assert properties, not strings

For output you can't predict, the assertion isn't "is it this exact string" — it's "does it satisfy the contract." Does it parse, with every required field present? Does it fit the 30-character store field it's destined for? Is it in the language you asked for? Is it free of code fences, markdown, and placeholder text like "[App Name]"? Did you get five headlines when you asked for five? Properties like these are deterministic to check even when the text isn't deterministic to generate.

The same checks shouldn't live only in the test suite. In ShotCanvas, the metadata generator treats every model reply as untrusted input: the prompt asks for a maximum of 30 characters, and the code enforces the cap anyway, truncating to the store's real limit after cleanup. A limit stated in a prompt is a request; a limit enforced in code is a guarantee. When a reply fails validation, the same scaffolding that you unit-tested retries or falls through to a fallback model. And if you can push part of the contract into the API itself — JSON mode, schema-constrained output — do that first; we've written about that ladder before. Validators catch what the schema can't.

Golden sets: score, don't gate

Mocked tests prove your scaffolding; they say nothing about whether the whole feature — prompt, model, parser, validator — still produces good output. For that, keep a golden set: twenty to fifty real inputs pulled from actual usage, run through the live feature on a schedule. Nightly, weekly, before a release — but not on every commit. Live calls are slow, cost real money, and a transient model hiccup shouldn't block an unrelated pull request.

The output of a golden run is a score, not a verdict: 46 of 50 outputs satisfied every property. Treat it like a performance benchmark — track it over time and react to movement, not to any single failure. This matters more than it used to, because the model under your feature can change without you deploying anything. Providers offer auto-updating aliases that roll forward to each new release; ShotCanvas calls Gemini through one, and the app was upgraded to a new model the day it shipped, with no code change on our side. That's exactly what we want — and a scheduled, scored run is how you find out what the upgrade did to your output before your users do.

When "good" isn't checkable, judge carefully

Some qualities refuse to be a property check. Is the headline compelling? Does the German translation read naturally? At indie scale the honest answer is often a human one — read a sample of outputs yourself once a week; thirty seconds each. The automated alternative is using a second LLM as a judge against a written rubric, which works with caveats: judges favor their own phrasing, reward length, and drift when their underlying model updates. Use a judge to rank — did prompt B beat prompt A on this rubric? — rather than to gate a release on an absolute score, and hand-check a sample of its verdicts before you trust it.

Why not just turn the randomness off?

The tempting shortcut is temperature zero: make the model deterministic and assert strings again. Two problems. Even at zero, hosted models don't promise identical output — batching and floating-point arithmetic on the provider's side can flip a token, and seed parameters, where offered, are described as best-effort. We covered the mechanics in the sampling post. The deeper problem: you ship at production settings. If the feature runs warm because users want variety, an eval run at temperature zero is testing a calmer feature your users never see. Test the configuration you actually ship, and let scores absorb the noise that remains.

Where we landed

Every AI feature in ShotCanvas — headline generation, metadata copy, translation — is built on this split: deterministic scaffolding under test, hard limits enforced in code, and model output treated as untrusted until validated. It's the reason a generated subtitle never blows a store's character limit, whatever the model felt like saying that day.