How to Choose an LLM for Your AI Product (Practical Guide)

Knowing how to choose an LLM starts with a narrow question: what is the one job your product needs a model to do? Pick the job first, then choose the cheapest model that does it well on your own examples. Do not start from a leaderboard or from whatever model is in the news this week.

The lesson "Choosing Models: Big, Small, Open, Closed" in the course The AI Product Stack covers this decision in about nine minutes, alongside lessons on what a model request costs and how to give a model your data. You sign in to open it.

Age of AI Entrepreneur teaches how to build, launch and grow a business with AI in short hands-on lessons. If you are choosing a model for a first product, you can pair this guide with the lessons and activities that practise each step.

Start with the job: the first step in how to choose an LLM

"Which model is best?" has no useful answer. "Which model is good enough for this task?" does.

Write the job as one sentence with an input and an output. For example:

• Input: a customer support email. Output: one label from a list of six.
• Input: a customer support email and three help articles. Output: a draft reply in your brand voice.
• Input: a long contract. Output: a list of dates and amounts.

These three jobs need different things. Labelling is simple and happens often, so it rewards a fast, cheap model. Drafting a reply needs better writing and good source material. Extracting from a long contract needs a model that handles long input without losing details.

Also write down what a bad answer costs you. A wrong label that a person fixes in two seconds is cheap. A wrong figure sent to a customer is not. The higher the cost of a mistake, the more you should spend on quality and on checking.

The main choices: hosted model APIs, open weights and small models

Most first products choose among three kinds of option.

Hosted model APIs

You send text to a provider over the internet and get text back. You do not run anything yourself. This is the fastest way to a working product, and it is where most people should start. The tradeoffs are that your data goes to a third party, the provider sets the price, and the provider can change or retire the model.

Open weights models

The trained model is published so you can run it on your own machines or on a host you pick. You get more control over where data goes and over the setup. You also take on the work of running it: servers, scaling, updates and monitoring. For a first product with no users yet, that work is usually a distraction.

Small models

A smaller model is cheaper and faster per request, and it is often enough for narrow jobs like the labelling example above. It tends to struggle more with long, messy or open-ended tasks. Small models can be hosted or open, so this choice is separate from the first two.

A sensible default for a first build: start with a hosted API, try one larger and one smaller model on the same task, and only look at self-hosting if you have a clear reason such as a data rule or a cost problem at real volume.

Quality, speed and cost: the three tradeoffs that matter

Every model choice trades these three against each other. You rarely get all three at once.

Quality

Quality means how often the output is right for your job, not how impressive the model seems in a demo. Measure it on your own examples (more on that below).

Speed

Speed has two parts: how long before the first words appear, and how long until the full answer is done. A chat feature feels slow if the user stares at a blank screen. A background job that runs overnight does not care. Match the speed you need to how the user waits.

Cost

Model APIs charge by tokens, which are chunks of text. A request costs roughly the input tokens times the input price, plus the output tokens times the output price. Work this out before you build. Take a typical request, count the tokens in what you send (including any instructions and documents) and in what comes back, then multiply by the number of requests a customer makes in a month.

Here is a worked example with placeholder numbers. Say each request sends about 1,500 tokens and gets back about 300. A customer makes 200 requests a month. That is 300,000 input tokens and 60,000 output tokens per customer per month. Plug in your provider's current prices and you know your cost per customer. Compare that with what you plan to charge. If the model eats most of your revenue, you need a cheaper model, shorter prompts or a different price. The post on usage-based pricing versus subscription for AI products shows how the model bill shapes that choice.

When you need retrieval and when a better model is enough

A common mistake is to switch to a bigger model when the real problem is missing information. A model does not know your company's refund policy, your product catalogue or last week's changes. A bigger model will just guess more fluently.

Retrieval fixes this. Your app looks up the relevant text from your own data and puts it in the request, so the model answers from that text. The course lesson "Giving Models Your Data: Retrieval" covers how this works, and the site's blog has a longer read on retrieval augmented generation.

A quick test to tell the two cases apart:

• If the answer is wrong because the facts are not in the request, add retrieval.
• If the facts are in the request and the model still reasons or writes badly, try a stronger model or a better prompt.
• If the model is right but too slow or too costly, try a smaller model.

Look at ten bad outputs and sort each into one of these three. The pattern tells you what to change.

Test candidate models on your own examples before you commit

Public benchmarks measure general skill. Your product needs a specific skill. The only way to know is to test.

Build a small test set before you pick:

1. Collect 20 to 50 real examples of the input. Use real emails, real questions, real documents. If you have no users yet, write them yourself or ask people in your target group for samples.
2. For each one, write down what a good answer looks like. For labelling, that is the correct label. For drafting, it is a short list of things the answer must include or avoid.
3. Run every candidate model on the same inputs with the same prompt.
4. Score the outputs. Count right and wrong, and note the kind of mistakes. Record the time and the token counts too.
5. Pick the cheapest model that meets your quality bar.

The set is small on purpose. You can run it in an afternoon, and it will show you differences a demo hides. Keep the set. When a new model comes out, you can rerun it in minutes. The Evals course goes deeper, with a lesson on building an eval set and judging outputs.

Keep your stack swappable so you can change models later

Whatever you choose now, assume you will change it. Prices move, better options appear and providers retire models. The post on AI model deprecation and how to prepare explains what happens when a model you depend on goes away.

A few habits make switching a small job instead of a rewrite:

• Put the model name in one setting, such as a MODEL_NAME variable, not scattered through your code.
• Wrap the model call in a single function. The rest of your app calls that function and never talks to a provider directly.
• Keep your test set and a script that runs it. This is how you know a swap is safe.

There is also a business angle. If your whole product is a thin layer on one provider's model, your customers have little reason to stay when that provider adds your feature. The course lesson "Building on Someone Else's Model" in the Distribution and Moats course looks at that risk.

Frequently asked questions

How do I choose an LLM for a first AI product?

Define the single job, start with a hosted API, and test one larger and one smaller model on 20 to 50 real examples. Choose the cheapest one that meets your quality bar.

Should I use an open weights model or a hosted API?

Most first products should start with a hosted API because it is the quickest way to something working. Consider open weights later if you have a data rule or a cost problem at real volume that justifies running it yourself.

Is a bigger model always better?

No. A bigger model is often slower and costs more per request, and a smaller one may do a narrow task just as well. Only your own test examples can tell you.

Do I need retrieval or a better model?

If the answer is wrong because your facts are not in the request, add retrieval. If the facts are there and the output is still poor, try a stronger model or a clearer prompt.

How do I avoid being locked into one model?

Keep the model name in one setting, call the model through one function, and keep a test set you can rerun. Then switching is a config change plus a quick check.

Get started

To work through model APIs, token costs, retrieval and model choice step by step, open The AI Product Stack. You sign in with your email and an emailed code. Every lesson is free to anyone with an account, as the terms page explains.

Start Age of AI Entrepreneur: short hands-on lessons on how to build a business in the age of AI, with every side of the debate and the sources to read next.