What Is a Foundation Model? A Plain Guide for Founders

What is a foundation model? It is a large AI model trained once on broad data at scale, then adapted to many different tasks. The model behind a chat assistant is one. So is the model behind an image generator. If you are building a product on AI, you will almost certainly build on one, so it helps to know what you are standing on.

The How Modern AI Actually Works course on Age of AI Entrepreneur teaches this in three short lessons, and a reader signs in with an email and an emailed code to open it. This post follows the same path with a founder's questions in mind.

What is a foundation model, in plain words

The term comes from a research report first submitted in August 2021. Its authors describe models such as BERT, DALL-E and GPT-3 as trained on broad data at scale and adaptable to a wide range of downstream tasks, and they call them foundation models "to underscore their critically central yet incomplete character".

Two parts of that definition matter for you. First, broad data at scale: the model is not trained for one job, so it picks up general patterns of language, images or code. Second, adaptable: you do not rebuild it for each product. You adapt it, with instructions, examples, your own documents or extra training.

The word "incomplete" is the useful one. A foundation model is a base, not a finished product. Somebody still has to shape it for a job and check that it does the job.

Why one model can power many products

The GPT-3 paper shows the shift. Its authors describe the older recipe as pre-training on a large body of text and then fine-tuning on thousands of task-specific examples. GPT-3, with 175 billion parameters, was instead given tasks and a few examples purely as text, with no fine-tuning. So the expensive training happens once, and many products are built on top. The picture shows the idea.

This is why a team of three can ship an AI product in weeks. It is also why your choice of model is a business decision. The 2021 report notes that the defects of a foundation model are inherited by all the models adapted from it. If the base is wrong about something, your product may be wrong in the same way, and so may your competitors' products.

How a raw model becomes a helpful assistant

The first lesson of the course covers next-token prediction: a language model reads text and predicts the next small chunk of it. Our post on how large language models work goes through it step by step. A model trained only for that job is a text continuer, and it is not yet an assistant.

The second lesson, From Raw Model to Helpful Assistant, covers what happens next. The research paper behind InstructGPT puts the problem plainly: making language models bigger does not inherently make them better at following a user's intent, and large models can give outputs that are untruthful, toxic or simply not helpful.

The paper's fix has two steps. Labelers wrote examples of the behavior they wanted, and the team fine-tuned GPT-3 on them. Then labelers ranked model outputs, and the team trained further with reinforcement learning from human feedback. In the authors' human evaluations, outputs from the 1.3 billion parameter InstructGPT were preferred to outputs from the 175 billion parameter GPT-3, even though it has about a hundred times fewer parameters. Our post on how ChatGPT is trained covers the same stages.

The takeaway for a builder: when an API lists a "base" model and a chat or "instruct" model, they are different products. The assistant version has had extra training so it answers instead of merely continuing your text. For most products, that is the version you want.

What a foundation model still gets wrong

The InstructGPT authors write that even their improved models still make simple mistakes. The third lesson of the course, What AI Gets Wrong, is about that. A fluent answer is not a checked answer.

Two earlier posts cover the practical side. AI hallucination examples shows real cases and what they teach, and why ChatGPT gives different answers to the same prompt explains why you cannot expect the same output twice. Both point to the same design rule: decide what a wrong answer costs your customer, and build a check where the cost is high.

Foundation model vs your product: a worked example

Say you want to build a tool that answers questions about a company's returns policy. Here is one week, using only ideas from this post.

1. Day 1: pick the assistant version of a model. Use a chat or instruct model from a provider you can call by API. Do not start by training anything. The lesson Choosing Models: Big, Small, Open, Closed in The AI Product Stack course covers the trade-offs.
2. Day 2: write the instruction. For example: "You answer questions about our returns policy using only the policy text below. If the policy does not say, reply that you do not know." The instruction is how you adapt the base to your job.
3. Day 3: give it the policy. Paste the policy text into the prompt. If your documents grow, you would move to retrieval, which the same course covers.
4. Day 4: collect 30 real questions from customers or colleagues, and write the correct answer for each one by hand.
5. Day 5: run all 30 and count the misses. Look at each wrong answer and ask which rule it breaks: an invented detail, a missed exception, an answer to something outside the policy. This habit is the subject of the lesson Building an Eval Set and Judging Outputs in the Evals: Shipping AI That Works course.

Notice what you did not do: you did not pretrain a model. You adapted a base that someone else paid to build, and you spent your own time on the part only you can do, which is knowing what a right answer looks like for your customer.

When a foundation model is the wrong tool

If you can write the rule in one sentence, write the rule. A returns window of 30 days is a number in a table, not a job for a language model. A model also adds cost and a chance of error that a rule does not have. Use a foundation model where the input is messy language, images or audio, or where the output is text that a rule cannot produce.

Frequently asked questions

Is a foundation model the same as a large language model?

Not exactly. A large language model is one kind of foundation model, trained on text. The 2021 report also counts models such as DALL-E, which works with images.

Is GPT-3 a foundation model?

The 2021 report names GPT-3 as an example. Its own paper describes an autoregressive language model with 175 billion parameters that was given tasks and examples purely as text.

Do I need to train my own foundation model?

Almost never for a first product. Most early products call a model someone else trained and adapt it with instructions and examples. Training from scratch needs far more data, computing power and specialist skill.

What is the difference between a base model and a chat model?

A base model is trained to predict the next token of text. A chat or instruct model has had further training, such as the human-feedback method in the InstructGPT paper, so it follows instructions instead of just continuing your text.

Why does the choice of model matter for my business?

Your product inherits the model's strengths and its defects, and a provider can change or retire a model, as our post on AI model deprecation explains. Test your own examples on any model before you commit.

Get started

Open Age of AI Entrepreneur, sign in with your email and an emailed code, and start with the How Modern AI Actually Works course. Each lesson takes about nine minutes and lists its sources, and the terms page says every lesson is free to anyone with an account.