Electrifiedby vlvd.net

Base Model vs Instruct Model: What Changes and Why It Matters

Base model vs instruct model: a base model is a language model straight out of pretraining, and it does one thing, continue whatever text you give it. An instruct model is that same kind of model after extra training on example requests and answers, so it treats your text as a request and tries to do what you asked. The base model knows a lot but does not know it is meant to help. The instruct model has learned that job.

This is the subject of From Raw Model to Helpful Assistant, the second lesson of the How Modern AI Actually Works course on Age of AI Entrepreneur, a Basic course for founders with no background; sign in with your email and an emailed code to open it.

What a base model actually does

A base model is trained on a huge pile of text with one goal: given the words so far, predict the next token. A token is a word or a piece of a word. Repeat that prediction billions of times over web pages, books and code, and the model picks up grammar, facts, styles and patterns of reasoning along the way.

What it does not pick up is the idea of a user. When you type "Write a tagline for a bakery app", a base model asks itself, in effect, "what text usually comes after this?" On the web, a line like that often sits in a list of writing prompts, so the model may carry on the list instead of answering it. That is not a bug. It is exactly the task it was trained for.

You can still get useful work out of a base model, but you have to shape the page. The GPT-3 paper (Brown and colleagues, 2020) tested a 175 billion parameter model by giving it tasks and a few worked examples purely as text, with no fine-tuning. Show three questions with their answers, then a fourth question, and the most likely continuation is the fourth answer. This is called few-shot prompting, and with a base model it is the main way to steer.

If you want the short version of why this works, our post is ChatGPT just autocomplete? sets out the debate around next-token prediction.

What turns a base model into an instruct model

The best-known recipe is from the InstructGPT paper (Ouyang and colleagues, 2022). Its abstract opens with the point of this whole post: making language models bigger "does not inherently make them better at following a user's intent." The fix it describes has two steps on top of a pretrained GPT-3:

  1. Supervised fine-tuning. People write example answers to a set of prompts, showing the behaviour they want. The model is trained on those demonstrations.
  2. Learning from rankings. People rank several model answers to the same prompt from best to worst, and the model is tuned further with reinforcement learning from human feedback (RLHF) to produce the kind of answer people ranked higher.

Our post RLHF explained walks through the second step in detail.

Diagram of raw text going through pretraining to a base model, then fine-tuning to an instruct model, with the same prompt answered by each

Pretraining makes the base model; tuning on written and ranked answers makes the instruct model.

Base model vs instruct model: the differences that matter

  • How it reads your text. A base model sees a document to continue. An instruct model sees a request to answer.
  • How you steer it. A base model is steered by the shape of the page: examples, headings, a half-finished format. An instruct model is steered by plain instructions, often in a system prompt.
  • When it stops. A base model can run on past the answer, inventing the next question too. An instruct model has learned where an answer ends.
  • Tone. A base model copies whatever voice the page suggests, good or bad. An instruct model has a trained default voice, usually polite and helpful.
  • Size versus helpfulness. In the InstructGPT study, people preferred answers from a 1.3 billion parameter InstructGPT model over the 175 billion parameter GPT-3, despite it having 100 times fewer parameters. Tuning changed more for users than size did.

Tuning has costs too. A model trained to please raters can lean toward telling people what they want to hear, which our post on AI sycophancy covers. The InstructGPT authors also note their model "still makes simple mistakes". Researchers disagree on how much tuning adds new ability and how much it only draws out what pretraining already put there; both readings have serious support.

A worked example you can run in ten minutes

You do not need to train anything to feel the difference. Some model providers and open model hubs offer a base version and an instruct version of the same model. If you can reach both, try this:

  1. Send the bare request to both. Use: "Write a tagline for a bakery app." Note what each does. The instruct model will likely answer. The base model may answer, ramble, or start a list of more prompts.
  2. Rewrite it as a page for the base model. Give it three lines to finish, each on its own line: Tagline for a gym app: Train on your schedule, not ours., then Tagline for a library app: Every book in town, one search away., then Tagline for a bakery app: with nothing after it.
  3. Compare again. The base model now has one obvious next line to write. Count how often, out of five runs, each model gives a usable tagline and stops.
  4. Write down what you learned. For most products the answer is plain: build on the instruct model, and keep the few-shot trick for formats it gets wrong.

If you only have an instruct model, you can still run step 2 against it. You will usually see that examples sharpen its format even though it does not need them to understand the request. That habit pays off when you write a system prompt for your first AI tool.

Which one should a founder build on?

Almost every AI product a small team ships today talks to an instruct model, often called a chat model, through an API. It understands requests, it stops when the answer is done, and it has been tuned against obvious harms. For a first product, that is the sensible default.

Base models still have uses. Teams that fine-tune their own model sometimes start from a base model so their own examples set the behaviour, without a trained voice to fight. Base models can also be better at pure continuation work, such as finishing code or matching a very specific writing style. Both of these are jobs for later, once you know what your users need. Our guide to foundation models covers where these models come from in the first place.

How the course teaches this

How Modern AI Actually Works has three lessons: Next-Token Prediction, From Raw Model to Helpful Assistant, and What AI Gets Wrong. This topic is the heart of the second. Each lesson takes about 9 minutes and lists its sources, and you answer questions as you go: choose, put in order, match, sort and estimate a number. Every attempt is graded, and you can retake a lesson any time.

Frequently asked questions

Is ChatGPT a base model or an instruct model?

ChatGPT is a tuned assistant, not a raw base model. It is built on the same idea as InstructGPT: a pretrained model taught to follow requests with human-written examples and human rankings.

Is an instruct model smarter than a base model?

Not necessarily. Tuning mostly changes how the model uses what it learned in pretraining, so it follows requests better. In the InstructGPT study, people preferred a much smaller tuned model's answers, but researchers still debate how much tuning adds new ability.

What is the difference between an instruct model and a chat model?

Both are tuned to follow requests. A chat model is an instruct model trained further on back-and-forth conversations, so it keeps track of turns and roles such as system, user and assistant.

Can I fine-tune a base model myself?

Yes, if you can get the base version of an open model and have good examples of the behaviour you want. For a first product, prompting a hosted instruct model is far cheaper and faster.

Get started

From Raw Model to Helpful Assistant is the second of three lessons in the How Modern AI Actually Works course on Age of AI Entrepreneur. Questions from lessons you finish come back later, spaced out, up to 5 a day, so the ideas stay with you. Sign in with your email and an emailed code to open the course.

Start Age of AI Entrepreneur: short hands-on lessons on how to build a business in the age of AI, with every side of the debate and the sources to read next.

Building a business in the age of AI, from a first project to a company.

Visit aientrepreneur.vlvd.net
Age of AI Entrepreneur on desktop
Age of AI Entrepreneur on tablet
Age of AI Entrepreneur on phone
0 likes

Comments

No comments yet.

or make an account to comment.