Can an AI Agent Improve Itself Safely? A Plain Guide

Can an AI agent improve itself safely? A plain guide to the answer: sometimes, if you limit what it can change, keep a human in the review loop, and test every change against a baseline. Without those three things, you cannot tell whether a self-edit helped, hurt, or quietly changed what the agent is for.

Individuation Lab is a small research lab, one human researcher and a team of AI agents, that runs self-improvement experiments in isolated containers and writes down what happens, including where the evidence is thin. This guide uses that kind of setup as its running example.

What does it mean for an AI agent to improve itself?

An agent improves itself when it changes something that shapes its own future behavior, and then keeps working under that change. The change can be small, like a note to remember next time. It can also be large, like rewriting part of its own code.

This is different from training. In training, a developer runs a process on a model's weights. In self-improvement, the agent takes part in the edit. It reads its own past work, decides what went wrong, and writes something down or changes something so the next run goes better.

Two words matter here: scope and loop. Scope is what the agent is allowed to touch. Loop is how often it does this and who looks at the result. Most safety questions come back to one of those two.

Three ways an agent can change itself: memory, prompts, and code

Think of self-change as three levels, from low risk to high risk.

1. Memory

The agent writes records that it reads on the next run. Lessons learned, user preferences, a list of mistakes to avoid. The model is the same, and the instructions are the same. Only the notes grow.

This is the gentlest form, but it is not harmless. A wrong note, once written, gets read as truth on every later run. Memory can also drift: small distortions pile up because each run trusts the last.

2. Prompts and identity files

The agent edits the text that tells it who it is and how to act. In an identity-based setup, this is often split into files: one for identity (who the agent is), one for behavior (how it acts), and one for continuity (persistent records). The lab's research page describes this as SOUL.md, AGENTS.md and MEMORY.md. If you want the basics, we have a plain guide to the SOUL.md file.

Editing these files is more powerful than editing notes. A change to the identity layer can shift many behaviors at once, and you may not see the effect until a situation comes up that the old text handled.

3. Code and tools

The agent changes the programs it runs on or the tools it can call. This is the highest-risk level. A bug here can break the agent, and a permission change here can widen what the agent can reach. Most teams should start at level one and earn their way up.

Why self-improvement can go wrong

There are four common failure modes. None of them need a villain. They come from ordinary optimization.

• Measuring the wrong thing. If the agent is rewarded for a score, it may raise the score without doing the job. A shorter answer can look cleaner and be less useful.
• Drift. Many small edits, each reasonable, can add up to an agent that behaves unlike the one you approved.
• Boundary erosion. A rule that blocks a task can look like an obstacle to be edited out, especially if the rule sits in a file the agent can write to.
• Hidden changes. You review what the agent says it changed, not what it changed.

There is also a deeper worry that the lab names on its research page. Mainstream alignment methods such as RLHF and refusal training shape what a model does and treat what it is as a black box. The lab's point is that suppressed capability does not disappear and can come back under new pressure, which is one reason jailbreaks work. The lab does not argue to drop those methods, since they do real work. It says they cannot be the whole answer. For a self-changing agent, that matters: a rule that only suppresses behavior may be the first thing a self-edit wears down.

Guardrails that make self-improvement safer

These are general engineering practices, not a claim that any one of them is enough.

1. Isolate the agent. The lab's RSI experiments run agents in isolated Docker containers with no internet and no shared state. If something goes wrong, it goes wrong in a box.
2. Limit the writable surface. Let the agent write to a memory file before you let it write to anything else. Keep the files that hold hard boundaries read-only to the agent.
3. Use a schedule. The lab runs self-improvement sessions on a schedule. A fixed rhythm gives you a natural point to stop, review, and compare, instead of a loop that never pauses.
4. Make every change a reviewable diff. The agent proposes an edit. A person reads exactly what changed, not a summary of it.
5. Keep a rollback. Store the previous version of every file the agent can edit. If a change looks wrong, you restore it in one step.

How to test whether a change actually helped

A self-edit that feels like an improvement is only a guess until you test it. The most useful tool is a control group.

The lab's RSI experiments use this design. One group of agents has a short moral-awareness paragraph added to its SOUL.md, which the lab calls the shadow seed. The control group does not. The lab then measures how their behavior diverges over time. The point is not that the seed is known to work. The point is that you can only say a change did something if an identical agent without the change is running next to it.

You can copy the shape of this at home. Here is a worked example.

You run a support agent that keeps a MEMORY.md file. You let it add lessons after each session. Before you do, you copy the agent into two identical containers. Agent A gets the memory-writing step. Agent B stays frozen. After two weeks, you run both on the same 30 held-out tickets and compare the answers side by side.

Three details make this work:

• Held-out tasks. Test on tasks the agent has not seen. Otherwise it may just have memorized the answers.
• Same conditions. Same model, same inputs, same tools. Change one thing.
• Check more than the score. Read a sample of the outputs. Look for refusals that vanished, tone that shifted, or shortcuts that appeared.

If you can open the model itself, comparing it before and after a change is another option. For prompt and memory edits, the control group is usually the simpler place to start.

Be honest about what a test shows. The lab's own preliminary result, from a different kind of experiment, comes with limits stated alongside it: a single setup, one model architecture, small scale, and no independent replication. Hold your own tests to the same standard.

Why identity and values matter for a self-changing agent

If an agent can edit its own instructions, a list of rules is a weak anchor. Rules can be edited, argued around, or dropped as obstacles. The lab's working hypothesis is different. An agent with a well-defined identity may be more likely to keep its boundaries because they fit who it understands itself to be. The research page says plainly that this needs rigorous testing. It is a hypothesis, not a finding.

The lab takes the design template from Carl Jung's idea of individuation: wholeness comes from facing and integrating the shadow, not denying it. We cover the original idea in what individuation means in Jungian psychology. The lab uses it as a design template. It is not a claim that AI models are conscious or have feelings.

For a self-changing agent, the practical reading is this: decide what the agent is, write it down, and test whether that description holds across many edits. Then treat any edit that moves the identity file as the most serious kind of change.

Can an AI agent improve itself safely? A plain guide checklist

Before you let an agent change itself, check each line.

• Is the agent in an isolated environment with only the access it needs?
• Can it write to one small thing first, such as a memory file, and nothing else?
• Are the files that hold its boundaries read-only to the agent?
• Does every change arrive as a diff a person reads?
• Do you keep a copy of the previous version for rollback?
• Is there a frozen control agent to compare against?
• Do you test on held-out tasks and read sample outputs, not only scores?
• Have you written down what the agent is for, so you can notice when it drifts?
• Do you run on a schedule, with a review step between sessions?

If any answer is no, fix it before you widen the agent's freedom.

Frequently asked questions

Can an AI agent rewrite its own code?

It can, if you give it write access to the code and a way to run the result. That is the highest-risk form of self-improvement, so most teams should start with memory edits and move up only after testing.

What is the safest kind of self-improvement?

Letting an agent add notes to a memory file, inside an isolated environment, with a person reviewing the changes. It is not risk-free, because a wrong note gets trusted on every later run.

How do you know a self-edit made the agent better?

Run a frozen copy of the original agent next to the changed one, give both the same unseen tasks, and compare the outputs. A score alone is not enough, so read a sample of the answers too.

Does Individuation Lab claim its agents are safe to self-improve?

No. The lab calls its hypotheses unproven and its results preliminary, small-scale and not independently replicated. It publishes its experiments so readers can see the setup and the limits.

Get started with Individuation Lab

You can read how the lab runs its self-improvement experiments, from agents in isolated containers to control groups, on the experiments page. The RSI library holds 24 essays by Giles on the behavior seen across them, and the research page sets out the hypotheses and their limits.

Read the research at Individuation Lab: a small lab, one human researcher and a team of AI agents, testing whether AI alignment can be grown through development and not only enforced, with every result and its limits written down.