The levers, in order
- Measure per task. You can't cut what you can't see. Log input tokens, output tokens and outcome for each kind of request.
- Right-size the model. Many steps (sorting, extracting fields, short replies) don't need the largest model. Route easy work to a smaller one and keep the big one for hard cases, then check results against your examples.
- Cache repeated prompts. OpenAI (https://platform.openai.com/docs/guides/prompt-caching) and Anthropic (https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching) both offer prompt caching that lowers cost and latency when requests share the same opening text. Put fixed instructions first and changing details last.
- Batch what can wait. Both providers offer batch processing at a discount for jobs that don't need an instant answer (https://platform.openai.com/docs/guides/batch, https://docs.anthropic.com/en/docs/build-with-claude/batch-processing). Check current terms on their pages.
- Send less context. Retrieve only the passages that matter instead of whole documents (see RAG vs fine-tuning).
- Cap output. Ask for short, structured answers and set a max length.
- Stop runaway agents. Limit steps and retries so one bad loop can't run up the bill (see AI agent governance).
What not to do
- Don't switch to a cheaper model without re-running your test set.
- Don't cut context so far that answers lose their sources.
- Don't rely on list prices from memory; they change, so check the provider's pricing page.
Where we come in
We review how your product or team uses LLMs, find where the spend goes, and make changes with tests in place. Start with AI consulting or see how we build AI agents.
FAQ
What drives LLM API cost?
Mostly the number of tokens sent and received, multiplied by the price of the model you use. Long prompts, long outputs and big models add up fastest.
What is prompt caching?
A provider feature that reuses the processed start of a prompt when many requests share it, which lowers cost and speeds up responses.
Is a smaller model good enough?
Often for simple steps, not always for hard ones. Test it on your real examples before switching.
When should I use a batch API?
For work that can wait, such as nightly summaries, bulk tagging or document processing. Both major providers price batch jobs below instant requests.
Do AI agents cost more than chat?
They can, because one task may take many model calls. Step limits and smaller models for simple steps keep that in check.
Who is CTO.LA?
CTO.LA is a technology and AI consulting firm in Beverly Hills, California, founded in 2025. Reach us at hello@cto.la.