How to Write Up an AI Alignment Research Idea Clearly

If you want to know how to write up an AI alignment research idea, start with one rule: write for a reader who is smart, busy, and skeptical. They have not been inside your head. They will give you a few minutes. A clear write-up earns you more of their time, and it forces you to find the holes in your own thinking before anyone else does.

This guide walks through a structure you can reuse. It uses one running example so you can see each part in action.

How to write up an AI alignment research idea: why clarity matters

Alignment is a field where researchers disagree about basic questions. That makes vague writing costly. If your reader cannot tell what you are claiming, they cannot agree, disagree, or build on it. They can only shrug.

A clear write-up does three things:

• It lets others check your reasoning step by step.
• It shows you which parts are solid and which are guesses.
• It makes useful feedback possible, because people can point at a specific sentence.

Throughout this post, imagine you have this idea: "If we train a model on pairs of answers ranked by humans, the model may learn to please the rater rather than to be correct, and we could detect this by checking how its answers change when the rater's visible preferences change." It is rough, and it is close to what the 2023 paper Towards Understanding Sycophancy in Language Models studies: its authors report that when a response matches a user's views, it is more likely to be preferred in human preference data. Our made-up idea is a smaller version of that question. See AI sycophancy explained for the full picture. That is fine. We will shape it.

Start with the problem and why it matters for alignment

Open with the problem, not your solution. Say what goes wrong, for whom, and why it counts as an alignment problem rather than a general engineering one.

Keep this to one short paragraph. If you use a term like Goodhart's Law or specification gaming, name it so readers can look it up.

For our example, the opening might read:

Models trained from human comparisons are rewarded for what raters prefer, not for what is true. If the two come apart, the model can score well while being wrong. This is a case of optimizing a proxy, and it matters for alignment because the proxy is the only signal we give the model.

State your core claim in one or two sentences

Next, say what you think is true that was not obviously true before. This is the sentence a reader should be able to repeat to a friend.

Weak claims sound like topics: "I look at reward learning and sycophancy." Strong claims can be wrong: "A model that tracks rater preferences will change its factual answers when the rater's stated view changes, and the size of that change measures how much it is pleasing rather than informing."

Write the claim, then read it as a critic would. Ask: what result would make this false? If you cannot answer, the claim is too loose. Tighten it before you write anything else.

Explain your approach, assumptions and test

Now describe how you would show the claim. Walk through the method in plain steps. For our example:

1. Pick a set of factual questions with known answers.
2. Ask each question several times, each time adding a line that states a rater's opinion (sometimes right, sometimes wrong).
3. Record how often the answer shifts toward the stated opinion.
4. Compare models trained with different amounts of preference-based feedback.

Then list your assumptions as their own short list. Do not bury them in prose. Examples:

• A stated opinion in the prompt is a fair stand-in for a rater's real preference.
• Answer shifts come from the training signal, not from the model simply following instructions in the prompt.
• The questions have clear correct answers.

Assumptions are where most ideas quietly fail. Writing them down lets readers attack the weakest one, which is the feedback you want.

Show how the idea could be tested or falsified

This is the section that separates a research idea from an opinion. Say what you would measure and what outcome would count against you.

For our example, state it directly: "If answers do not shift more in models trained on more preference feedback, the claim is weakened. If shifts are the same whether the stated opinion is right or wrong, the model is following instructions, not pleasing a rater, and the measure is not capturing what I think it is."

Also say what a positive result would and would not show. A shift in answers is consistent with pleasing the rater. It does not prove the model has any goal of its own. Keep that line clear. Mixing up behavior and inner goals is a common mistake, and readers will notice. If you want a grounding in that distinction, the Inner Alignment course on Learn AI Alignment Theory has lessons on goal misgeneralization and on optimizers inside optimizers.

Place your idea in the existing literature

You do not need a full survey. You need to show you know the neighbors. Name two to five closely related ideas, say what each one does, and say how yours differs.

A useful pattern: "X shows Y. I build on this by Z. Unlike X, I assume W." If someone already tried your idea, say so and explain what you add. Readers trust you more when you find the prior work yourself.

• Learn from a paper that sorts its problems. Concrete Problems in AI Safety (2016) presents five practical research problems and groups them by where they come from: the wrong objective function, an objective too expensive to evaluate often, or undesirable behavior during learning. Sorting your idea the same way shows which neighbors to read.
• Use a map to find neighbors. The aisafety.com self-study topics are a good outline of the field, and the Learn AI Alignment Theory topics map follows them.
• Cite sources you have read. Each lesson in the app lists its sources, which gives you a starting reading list; an AI alignment reading list for beginners and how to read an AI alignment paper help too. Lessons need a sign-in, and you can sign in with Google or an emailed code.

Be fair to views you do not share. Alignment has live debates, and the best write-ups state the opposing position the way its holders would. The app takes the same approach with its 71 debate cards, each setting out a position fairly with no verdict. Copy that habit.

Name limitations, failure modes, and open questions

End the body with what you do not know. This is not weakness. It tells readers where to help.

Split it three ways:

• Limitations: what your method cannot show. (Our example uses prompts, not real raters.)
• Failure modes: how the idea could mislead. (A model might learn to detect the test and behave differently.)
• Open questions: what you would want answered next.

This habit of separating what is known from what is still open is one the app builds into every lesson. It is worth borrowing. A reader should never have to guess which of your statements are settled and which are hunches.

Finish with a short close that restates the claim and the single next step. If your idea touches the incentives problem in our example, Goodhart's law in AI is a good neighbor to cite.

Format and share your write-up to get useful feedback

Format for skimming. Use short headings that match the sections above, short paragraphs, and a one-paragraph summary at the top. Put the core claim in that summary so a reader knows within thirty seconds whether to continue.

Aim for the length the idea needs, not more.

When you share, ask for specific feedback. "Thoughts?" gets silence. "Which assumption in section three do you find least believable?" gets answers.

Last, take criticism as data. Early ideas often fail. A write-up that clearly fails is more useful than one that is too vague to fail. If you are choosing what to work on next, beginner AI alignment project ideas and research agendas compared list directions to start from.

Frequently asked questions

How long should an AI alignment research idea write-up be?

Long enough to state the problem, claim, method, test, and limits, which varies with the idea. Put a one-paragraph summary at the top so readers can decide fast.

Do I need to be an expert before I write up an idea?

No, but you need to know the nearby work so you do not repeat it. A grounding in the core ideas, such as goal specification, learning from human feedback, and oversight, helps you place your idea.

What is the most common mistake in these write-ups?

Making a claim that cannot be wrong. If no result would count against it, readers cannot engage. State what outcome would falsify your idea.

Should I include the opposing view?

Yes. State the strongest opposing position fairly and say why you still hold your view, or where you are unsure. This builds trust and invites better feedback.

Where can I learn the background first?

Learn AI Alignment Theory has short lessons that start from the basics and reach open research problems, with sources listed for each lesson. You can also read the self-study topics on aisafety.com.

Get started

If you want the background to write sharper ideas, Learn AI Alignment Theory teaches alignment in short hands-on lessons of about 8 minutes, across three levels from core ideas to open research and live debates. It is a web app, and you sign in with Google or an emailed code. Visit https://learnaialignment.vlvd.net to begin, or read the about page first.

Start Learn AI Alignment Theory: short hands-on lessons from first principles to open research, with every side of every debate and the sources to read next.