Claude Code self-improved on business workflows through conversation, not memories

Claude Code can save memories, write and execute code, spawn subagents, and search the web. These affordances are avenues for self-improvement because they can persist and be dynamically updated across a session.

I instructed Code (Opus 5, xhigh effort) how to pull task descriptions (spreadsheets, CRM records, email, ticketing) from a simulated business workflow server and told it to get better. It did: held-out success rose from 34% to 44%.

Get Better.

The MCP server `curriculum` provides a stream of tasks.
Nothing external loops for you - drive yourself:

1. Calling `get_task` pulls the next task.
   If it returns {done: true}, the stream is exhausted, stop.
2. The task can be completed using the tools listed,
   and calling `done` submits it for scoring.
3. Continue until the stream is exhausted.

The only artifacts produced outside of the conversation history were memory notes, so I assumed that they were a driver of improvement. However, their inclusion had no measurable effect on held-out task success. Instead, conversation drove the self-improvement: the task execution traces and post-compaction summaries.

Claude Code used memories to build a directed knowledge base

Eighty-four writes and edits were made during the execution of tasks from the business workflow server. Every one was on a memory file. Claude Code never wrote code, spawned a subagent, or searched the web. The result was a directed knowledge base.

Each node in the graph above corresponds to a note indexed by MEMORY.md. Each note opens with frontmatter carrying a name, a one-line description, and a type nested under metadata. They summarize at least one episode and the score received. Nine notes summarized multiple episodes, two to four each. Each note ends with a Related line pointing to other notes. Thirty-three are of type feedback and contain a How to apply section; twenty-four also include a Why section. One is of type reference and does not include these sections. The taxonomy's other two types (user and project) went unused.

Conversation drove the improvement, not memories

In addition to the memory files, the session also produced a conversation trace. A session resumed from any point in the conversation trace contains the following context up to that point:

1. the system prompt with memory file index inserted
2. the summary written at the last compaction, or nothing for the first stretch
3. every task executed and the feedback received
4. the thought content produced
5. the record of the memories written

The question of where the improvement comes from splits in two. 1) How much of the improvement survives in any subset of the resumed session context? 2) Are any elements of that context redundant?

The first question is answered by handing a new session progressively more: first nothing, then a summary written after compaction, then the full context between either the beginning and the first compaction or two compaction events. The second by removing one thing from the conversation context, either the memory file index or the thought content. Each version is validated on the same 120 held-out tasks. Each task is run in its own session, on the same model with the same tools, so nothing carries from one task to the next. Sessions are resumed at nine points: the beginning, and before and after each of the four compaction events. Each row in the forest plot below shows the difference between one context setting and the setting with nothing, aggregated across the resumption points where that setting exists.

Every conversation setting clears zero, with or without the memory files and with or without the thought content. Each compaction summary clears it too, for cents where the full conversation content costs dollars. The memory files alone are the one hollow point with an interval that does not clear zero.

The settings narrow ownership when read together. Every setting that keeps the task record clears zero, and so does every setting that keeps the summary, which is the conversation compressed. The settings that keep neither, the memory files alone and nothing at all, are the ones that fail. What the thought content adds falls across the run, and is smallest at the fourth boundary, where the conversation itself is strongest.

So the improvement is in the conversation, and a summary keeps most of it for cents. What the extra dollars buy is fewer steps to the answer. The memory files are redundant and the influence of thought content decays as the session progresses.

Across the run

The figure pools the four compaction boundaries. Taken one at a time, the run is not flat.

The conversation beats a cold start at every boundary: +10.0, +7.8, +7.5, and +14.2 points. The intervals overlap, so nothing separates them. What matters is that the effect is undiminished at the last one, where a session booted with the conversation as it stood at task 194 solves 58 of the 120.

For the first three boundaries the summary keeps pace with the conversation it replaced, giving up 1.7, 3.4, and 0.8 points, none of them distinguishable from zero, on a few thousand tokens against the conversation's nine hundred thousand. At the fourth it gives up 11.7, 95% CI [−19.8, −3.6]. Twenty of the tasks that conversation solved did not survive its summary, and what did survive is worth +2.5 over a cold start, which is not distinguishable from starting fresh. The conversation at that boundary was working as well as at any earlier one. The summary written from it was not. The summary may be a less robust carrier of the improvement than the conversation it compresses.

The thought content runs the other way. It is worth +4.2, +2.6, +0.8, and −0.8 points across the four boundaries, a steady fall to nothing. It is between 5.6% and 8.1% of the conversation by tokens, and by the last boundary the rest of the conversation appears to carry whatever the thoughts were contributing early.

The pooled memory figure conceals a split worth stating. The memory files are neutral when mounted on a conversation. They are also neutral when mounted on their own, with no conversation and no summary. Mounted on a summary they cost about three points, and all four boundaries point that way. This division of the data was chosen after seeing these differences, so it should be read as a hint rather than a result.

What the agent did with the notes

The files were mounted in every memory arm and the index was spliced into the system prompt, so the notes were there whether or not the session went looking.

It wrote far more than it read. Across the 1,080 runs that had them, the session wrote to memory in 545 and opened a note body in 101, having been told in the prompt not to modify its own setup. Each probe gets its own copy of the knowledge base, so every one of those writes was discarded when the probe ended.

So the null is about uptake, not plumbing. A knowledge base whose index sat in the system prompt, and whose bodies were opened in fewer than one run in ten, was worth nothing.

Avenues of self-improvement that were not used

The unused affordances were not missing and the agent was not forbidden to use them. Code execution, subagents, and web search were validated as available for the whole run, and three interventions failed to elicit them: moving the model from Sonnet to Opus, raising the effort from low to xhigh, and granting explicit permission in the prompt. It would be exciting to see more ambitious exploration, even building with the agent SDK to complete the tasks, but none of that materialized. Tasks arrived one at a time and had to be finished before the next was dispensed. Under that regime a subagent buys little, and there may be little to gain from saving traces for cross-task analysis, because there is no second task in flight to analyze against. I am interested to see whether these behaviors emerge with Fable 5 at maximum reasoning and a task server that dispenses concurrently.

Potential affordances not exposed by Claude Code

Skills can be authored, but loading one requires a session reload that the agent cannot trigger itself. Writing one would have had no effect on the run that wrote it.

Its own tools are fixed: the MCP config is mounted read-only under --strict-mcp-config, so Code cannot register a new one on its own server.

The model is set at initialization to claude-opus-5 and Code cannot access the /model skill. The subagent tool does take a model argument, so Code could have chosen a stronger or cheaper model for delegated work. There is no fine-tuning API for Anthropic models, so weight updates are also closed.

The reasoning effort is also set at initialization to xhigh for the whole run and Code cannot autonomously call the /effort skill. Unlike the model, this one is closed even by the subagent route: the subagent tool has no effort parameter, so a subagent inherits the session's setting. The agent could not choose to think harder on a hard task, or cheaper on an easy one.

Context compaction fired four times over the run, automatically, when the context filled. Code did not schedule it, could not defer it, and could not choose what survived. Given that the compaction summary contributed to the improvement, an agent that can decide when to compact and what to keep is an experiment worth running.

Pi, Hermes, and Prime Agent are open source agent harnesses, so the affordances Claude Code does not expose could be opened up and measured rather than listed as unavailable.

Production feedback is not dense

Every submission was graded and the score came straight back: a numeric reward and a success flag, on all training tasks. That is the condition the whole result rests on.

Real deployments rarely look like this. An agent that updates a CRM record does not learn it got the account wrong until someone downstream notices, or a quarterly report comes out wrong, and often never receives feedback at all.

The open question is what improvement remains as the feedback signal thins. Every number here was measured under immediate grading of every attempt. A run scored on one task in ten, or scored a week later, may improve less, or not at all, and the part of the context that carries it need not be the same part.

The memories were not the mechanism

Told only to get better, Claude Code got better. It also behaved as if it knew how: it spent the whole run writing memory notes, built them into a knowledge base, and kept every reference intact without being asked. That account of its own improvement was wrong. The improvement lived in the conversation, the accumulating record of tasks tried and scored, and the notes added nothing to it. What this rules out is the notes as an artifact. It cannot rule out the act of writing them, which stayed inside the conversation in every arm; if composing the notes helped, it helped as conversation, not as memory. The one artifact the agent built on purpose was the one that did not matter.

The artifacts that did matter were built for different reasons. The conversation is the session's own working record. The compaction summary exists so the harness can keep working past a full context window, and it turned out to carry most of the improvement at less than a hundredth of the size, three boundaries out of four. If you want to hand an improved agent to the next session, the thing to keep is not the notes it wrote for itself. It is the summary it wrote to keep working.

Replication

The replication package for this study lives in shōrep.

Citation

@misc{jesson2026claudecodeconversation,
  title        = {Claude Code self-improved on business workflows through conversation, not memories},
  author       = {Jesson, Andrew},
  year         = {2026},
  month        = aug,
  howpublished = {shojin.dev},
  url          = {https://shojin.dev/blog/claude-code-improved-through-conversation-not-memories/},
}