Moving AI work that can wait onto a batch API.

Anthropic, OpenAI and Google all sell the same models at half price, and the only thing they ask in return is patience. Submit your requests as a batch, accept that the answers may take up to a day, and the bill for that work drops by 50%. Most apps send every call down the live endpoint anyway, including plenty of work no person is waiting on. Here is how to tell which calls can wait, and what changes in your code when they do.

What you are actually trading

A normal API call holds a connection open until the model answers. A batch call doesn't. You upload a list of requests, get back a job id, and come back later for the results. All three providers price batch work at 50% of their standard rates, and all three quote 24 hours as the window. Anthropic says most of its batches finish in under an hour, and the others say theirs often finish early, but none of them promise it. If your design only works when the batch comes back in an hour, it doesn't work.

So the trade is half the cost for latency measured in hours. The hard part is being honest about which calls can live with that.

Three questions that sort every call

Walk your AI call sites and ask each one the same three things.

• Is a person watching a spinner? If someone tapped a button and is waiting on the answer, the call stays live. That covers most of what users see: a chat reply, a generated headline, an image description on upload.
• Would anyone notice if the result landed tomorrow morning? A weekly digest of reviews, tags for last month's records or a summary emailed overnight can all arrive late without anyone noticing.
• Can it fail and simply be tried again? Batch jobs fail request by request, as covered below. Work that can be retried tomorrow suits a batch. Work with a hard deadline doesn't.

Run that test on a typical app and the batch candidates are rarely the features. They are the jobs around the features:

• Evals. When you change a prompt or switch models, you rerun a few hundred test cases and compare the output. Nobody needs those numbers in thirty seconds, and it is often the largest AI spend a small team has that no user ever sees.
• Backfills. You ship a feature that summarises, tags or translates new records. Now the 40,000 records written before the feature existed need the same treatment, once.
• Catch-up localisation. You add Korean as a supported language. Every existing string, help article and store listing needs a first draft in Korean, and none of it has to be ready this minute.
• Embeddings for a search index. Rebuilding the index after changing the embedding model is pure bulk work. OpenAI's batch endpoint accepts embeddings requests directly.
• Scheduled reports. Anything that runs on a cron job was never real-time in the first place.

Do the arithmetic before you build anything

A batch pipeline is real engineering: a job table, a poller, a step that matches results to requests, and retry handling. That only pays for itself when the job is large or runs often. Take an illustrative price of $1 per million input tokens and $4 per million output tokens, and a backfill of 2,000 records at 1,500 tokens in and 300 tokens out each. That is 3 million input tokens ($3.00) and 600,000 output tokens ($2.40): $5.40 live, $2.70 as a batch. Saving $2.70 once doesn't justify a day of work. Just run it live.

Now make it the eval suite you run every time a prompt changes, at 20,000 cases a week. The same arithmetic gives $54 a week live and $27 as a batch, which is about $1,400 a year from a job nobody was looking at. Multiply out your own token counts before deciding. The answer is usually obvious once the numbers are written down.

Four things that will catch you out

Results come back in any order. Anthropic and OpenAI both say outright that output order may not match input order. Every request carries a custom_id you choose, and that id is the only reliable way to match an answer to its question. Derive it from the record (listing-8412-ko), not from the request's position in the file, so a resubmitted request still points at the right row.

A batch can finish without every request succeeding. Each request gets its own outcome. On Anthropic's API it can be succeeded, errored, canceled or expired. A batch that reports "ended" may contain a few hundred failures, so your code has to collect them and decide what happens next: resubmit them in tomorrow's batch, or send a handful through the live endpoint.

Expiry rules differ between providers. Anthropic expires any request not processed within 24 hours and doesn't bill for it. OpenAI moves an unfinished batch to expired at the end of its window, keeps the responses that did finish and charges for them. Gemini expires jobs still pending or running after 48 hours, and an expired job returns no results at all. Read the rules for the provider you use before you count on partial results.

Results don't wait for you forever. Anthropic keeps batch results for 29 days after the batch is created, and OpenAI deletes the output file 30 days after the batch completes. If your poller dies on a Friday and nobody notices for a month, the work is paid for and gone. Download results as soon as the job ends and store them yourself.

Build it as a pipeline, not a script

A script that submits, polls and waits works the first time and then fails the first time your laptop sleeps. Treat the batch as state in your database instead:

• Write one row per request (its custom_id, the source record and a status) before you submit anything.
• Store the provider's batch id on submission, and let a scheduled poller check every open batch rather than one process sitting on one job.
• When results arrive, write each answer against its row with an upsert, so processing the same results file twice changes nothing.
• Requests left errored or expired go back to pending, and the next run picks them up.

Built that way, a crash at any step loses nothing, and you can see in your own database exactly how much of a backfill has finished.

Stacking it with prompt caching

If every request in a batch shares a long system prompt or the same reference document, caching can cut the cost further. Anthropic says the two discounts stack, but warns that batch requests run concurrently and in any order, so cache hits are best-effort. It quotes hit rates anywhere from 30% to 98%, and suggests the one-hour cache lifetime rather than the five-minute default, because a batch can easily outlast five minutes. Gemini's batch mode lets each request reference cached content, billed at its normal caching rates. Put the shared material first and keep it byte-identical across requests, exactly as you would for live calls. Our prompt caching explainer covers how to arrange a prompt so the cache actually hits.

One more benefit, at least on OpenAI: batch usage has its own rate limit pool, separate from the per-model limits your live traffic draws on. A 20,000-case eval run no longer competes with your users for the same quota. Anthropic applies separate batch rate limits too, so check the published numbers for your tier before you plan a large run.

Everything a person does in ShotCanvas has someone waiting on it, so it all runs live. Keep what people watch fast, and move the rest to the cheaper queue.