How to choose an LLM: test on your own work, not on leaderboards

Choose an LLM by testing two or three candidates on real examples of your own work, not by reading leaderboards. Score each on answer quality, speed, cost per task, context size and data-privacy terms. Start with a capable model, then try smaller ones. Keep the model behind one interface so switching later is cheap.

Why leaderboards aren't enough

Public benchmarks measure general skills on public test sets. Stanford's HELM project argues models should be compared on many scenarios and many measures at once, such as accuracy, robustness and efficiency, not one score (Liang et al., https://arxiv.org/abs/2211.09110; https://crfm.stanford.edu/helm/).

Your tasks are narrower than any benchmark. The only score that predicts how a model will do on your support tickets or contracts is a test on your support tickets or contracts.

What the model providers recommend

Anthropic's guide says choosing a model means balancing capabilities, speed and cost, and suggests testing candidates against your own evaluations (https://docs.anthropic.com/en/docs/about-claude/models/choosing-a-model).

Anthropic also recommends writing clear success criteria and test cases before you compare (https://docs.anthropic.com/en/docs/test-and-evaluate/develop-tests).

OpenAI's model selection guide frames the choice as a trade-off between capability, speed and cost, and its evals guide shows how to test model outputs against your own criteria (https://platform.openai.com/docs/guides/model-selection; https://platform.openai.com/docs/guides/evals).

A simple way to choose

  1. Collect real examples. Pull a few dozen real inputs from the job you want AI to do, with what a good answer looks like.
  2. Write the pass/fail rules. Decide what counts as correct, what's unacceptable and how fast it must respond.
  3. Test two or three models. Run the same examples through each and score them the same way.
  4. Check the terms. Read each provider's data retention and training terms for your plan, and where data is processed.
  5. Price it per task. Estimate the cost of one finished task, not one token.
  6. Keep it swappable. Call models through one layer in your code, so a better or cheaper model is a config change, not a rebuild.

Long context isn't free

A bigger context window doesn't mean the model uses all of it well. Research found models were often worse at using information placed in the middle of a long input than at the start or end (Liu et al., https://arxiv.org/abs/2307.03172). Test with inputs as long as yours really are.

Where we come in

We help teams build the small test set, run the comparison and wire the winner into their product behind a swappable layer. See AI consulting, our guide to cutting LLM costs and RAG vs fine-tuning.

FAQ

What is the best LLM for business?

There isn't one best model for every business. The best one is whichever passes your own tests at a speed and cost per task you can accept.

Should I trust AI leaderboards?

Use them to build a shortlist, not to decide. They test general skills on public data, not your tasks.

Should I use a big or a small model?

Start with a capable model to see what good looks like, then test smaller ones. Use the smallest that still passes your tests.

Should I self-host an open-weight model?

Consider it when data rules, cost at high volume or control matter more than convenience. You take on hosting, updates and security.

How do I avoid vendor lock-in?

Call models through one interface in your code and keep your test set, so you can rerun it and switch when a better option appears.

Who is CTO.LA?

CTO.LA is a technology and AI consulting firm in Beverly Hills, California, founded in 2025. Reach us at hello@cto.la.

Sources

Book a call