What is a Recursive Language Model

In a previous post I compared the modern strategies AI agents use to search your codebase. One of them, Recursive Language Models, has been getting a lot of attention recently.

I wanted to see how it behaves at real codebase sizes, from a small library through to a codebase larger than the paper’s headline benchmark. So I built a small harness, pointed it at three codebases (fastapi at 176k tokens, vite at 421k, and the full VS Code source at 14.7M), and ran it against the obvious baseline of pasting the whole codebase into a single prompt.

What RLM is

The name is a bit misleading. RLM is not a new model. It is a scaffold around any existing LLM, with no retraining.

The mechanism:

1. The input you would have pasted into the prompt (a codebase, a 10,000-line log, a long document) gets loaded into a sandboxed Python REPL as a variable.
2. The wrapped LLM sees a short prompt saying “you have a variable called codebase. Here is a question. Write Python code to answer it.”
3. The model writes a snippet. The sandbox runs it. The model sees the output (truncated to a few thousand characters).
4. The model writes another snippet. Repeat.

The sandbox exposes a few built-in tools:

• llm_query(prompt) for a single sub-LLM call with a slice of the data
• llm_query_batched(prompts) for concurrent fan-out
• Standard library: re, json, collections, the usual Python primitives
• SUBMIT(answer) to terminate with the final output

So the model can scan with regex, chunk the data, fan out sub-LLM calls in parallel, and aggregate results in code. The main model only sees its own conversation history, never the full input.

DSPy ships RLM as a module; alexzhang13/rlm is the reference library.

Why people are paying attention

The MIT paper landed on arXiv at the end of December 2025 and was last revised in May 2026. The headline result: on BrowseComp-Plus (a benchmark released August 2025 where each question requires finding facts scattered across 6 to 11 million tokens of input, roughly 30 average novels’ worth of text) base GPT-5 scores 0%. Wrap the same GPT-5 in the RLM harness and the score jumps to 91.33%. Same model, same data, the gain is entirely from the scaffolding.

The numbers carry across other long-context benchmarks too. On OOLONG, standard frontier models drop below 15% on 75k+ token prompts because of “lost in the middle” attention dilution; RLMs hold above 80%. The paper also reports ~3.6x token efficiency on information-dense tasks. VentureBeat covered it as “10 million tokens without context rot”. Prime Intellect called it “the paradigm of 2026”. Machine Learning Mastery wrote a primer.

Two things drive the buzz beyond the benchmark numbers.

First, context windows stopped being the bottleneck people thought they were. Even at 200,000 tokens, models lose track of where things are, answering questions about one chunk with content from another. RLM dodges this because the model never holds the full input in its own context.

Second, the pattern composes with existing tooling. DSPy ships it as one module among many. You plug it into the LLM provider and prompt management you already use. No retraining required. A self-reflective variant called SRLM landed in March 2026, adding uncertainty signals to the recursion to shorten time-to-answer.

The paper itself flags one caveat: on shorter inputs that fit cleanly into a context window, rigid recursion can underperform vanilla prompting. The question I wanted to answer is where that crossover actually sits on a real codebase, not on a long-context benchmark.

Comparing RLM with naive prompting

The setup:

• Codebases: tiangolo/fastapi (176k tokens), vitejs/vite (421k tokens), and microsoft/vscode (14.7M tokens, full src/)
• Questions: one short answer (“how does dependency injection work, one paragraph”) and one exhaustive table (“list every place an exception is raised, with file, line, trigger condition”)
• Approaches: RLM via dspy.RLM, naive (paste the whole codebase into one Claude prompt)
• Model: anthropic/claude-sonnet-4.6 via OpenRouter, max_tokens=4000 per call

The result, with a third row testing the paper’s scale by adding the full VS Code source tree:

Naive cost scales with corpus size. 176k tokens costs $0.53; 421k tokens costs $1.56. Output is rounding error.

RLM cost scales with task complexity. Short-answer cost went $0.13 → $0.16 → $0.10 across a 70x corpus increase. The main model never reads the corpus, only its own sub-LLM outputs.

The 10x gap on large × short (RLM $0.16 vs naive $1.56) is the cleanest result. Same model, same question, quality matched.

At 14.7M tokens, naive hits a gateway limit before any context window: OpenRouter returns a 400 with "The total text input size exceeds 8 MB". Not a Sonnet refusal - a body-size cap. I expected this to happen at vite scale too, but the 519k input went through cleanly. The break is somewhere between 519k and 14.7M.

The XL long-table $0.09 is not comparable to the vite $1.02. Vite used per-row semantic enrichment: ~15-21 sub-LLM calls to write the trigger-condition column. At 2,100+ candidate sites the model switched to regex - the trigger-condition column is the raw source line above each throw. For long-answer tasks with per-row enrichment, the right estimate is roughly $1 per million tokens of corpus.

The small × long tie (both ~$0.53) is explained by enrichment cost. RLM saved on input (main model never reads the corpus) but the 15 sub-LLM calls for trigger conditions each passed a corpus slice, adding up to nearly a full naive pass.

Naive’s failure on large × long was not input size - it was the output cap. The $1.62 call ran for 100 seconds and truncated mid-row at 4,000 tokens. RLM produced 128 complete rows for $1.02 by iterating.

The paper’s 91.33% is on BrowseComp-Plus, which is needle-in-haystack retrieval. RLM is well-suited to that task shape. The vscode $0.09 short-answer run is the same regime. A “list every X across 8M tokens” task forces per-item enrichment and hits the same quality/cost tradeoff as the vscode long-table.

What this means in practice

RLM wins decisively when the answer is short. It ties on cost when the answer is long and requires per-row semantic enrichment. It wins on completeness when the answer is long and the corpus is large enough that naive truncates. At very large corpus sizes (millions of tokens) naive cannot run at all, and RLM still costs cents. No cell where naive strictly wins on both cost and quality.

Caveats:

• Three codebases, one model, one provider, two question types. The shape should hold; the multipliers will move with model pricing and task chunkiness.
• The 8 MB upload limit is OpenRouter-specific. Direct Anthropic API has different limits and would push the naive-breaks-here boundary in different places. The cost arithmetic doesn’t change.

Naive’s cost is corpus-bound. RAG’s cost is chunk-count-bound. RLM’s cost is task-complexity-bound. Pick the one whose cost dimension you can keep small.