Token Efficiency: Where the Tokens Go
A running map of where an agent's tokens go: weights, retrieval, resident prompt, generation, and the loop. Revised when I measure a layer or a vendor revises a claim.
Overview
One page that splits the token bill into five layers and keeps a measured figure against each one: the weights and decode step, retrieval into context, the resident prompt that loads before the first message, generation, and the loop that decides how often any of it runs. Each layer has tools claiming savings, and the claims are rarely comparable because they quote different denominators. The map holds what I have measured myself, what I can quote from a primary source with its caveats, and the gaps where I have nothing yet.
The figures below are the current state of each layer as I have measured it.
Weights and decode
Decode speed is memory bandwidth divided by the weights touched per token, which puts the layer on the memory-bound side of the roofline before model quality enters the argument. On a 4-bit Qwen2.5-7B under MLX that division landed within 9% of measured. It holds from 3B up and breaks below that: at 0.5B it overshoots by nearly 3x. Sparse attention and cache design belong here too. DeepSeek V4 chunks the past, compresses each group and attends to the top-k entries, per its model card, while Gemma 4 uses sliding-window attention and Qwen 3.6 uses gated DeltaNet. I have not benchmarked the three against each other.
Retrieval into context
Two MCP servers build a tree-sitter index of a repository and serve symbols instead of files. On the same private TypeScript and Go monorepo, graft returned 6 tools and 886 tokens of schema at session start against jCodeMunch’s 91 tools and 27,526 tokens. Both cut the cost of reading code. jCodeMunch charges 31 times that before the agent asks a question.
The recursive-language-model approach moves the corpus out of the prompt entirely. 14.7M tokens of VS Code source, held as a variable in a sandbox, answered a question for $0.10, with only short slices reaching the model. The naive version could not submit the request at all: OpenRouter rejects a body over 8 MB and the concatenated source is around 58 MB.
The resident prompt
In the chat prompt circulating as a GPT-5.6 leak, whose provenance I cannot verify, the tool namespaces were 12,142 of 17,124 words. Measuring the document is the only claim I make about it. The same split shows up in the two resident prompts I can measure directly: 27,526 tokens of jCodeMunch tool schema, and 2,225 tokens of skill descriptions.
Installed skills are the same cost in miniature. My 29 skills add 2,225 tokens to every message through their descriptions alone, with a median of 68 tokens each, while 190,298 characters of skill bodies sit on disk until something fires.
Generation
ilo is the layer I build in. The same program is 0.33x the tokens and 0.22x the characters of its Python equivalent, with type errors reported as codes before execution rather than stack traces after it. The instruction set is small and the benchmark reflects that.
Output-compression skills work at this layer too. caveman’s own breakdown gives 65% on prose-heavy output and 8.5% on agentic coding, the second measured by JetBrains across 86 SkillsBench tasks.
The loop
In the ai-coworkers runtime the quiet gate compares sensor state, pending rituals and operator notes before calling a model at all, so an idle coworker converges on near-zero cost. Model routers make the same call automatically, picking a cheaper model per task, which is the decision I make by hand between Haiku and Opus.
What earns a place here
A figure goes on the map with its denominator and the method that produced it. Vendor headlines get replaced by the vendor’s own breakdown: caveman’s 65% is output tokens on prose, and its 8.5% is an agentic coding workload measured by a third party. When caveman retired its 75% figure I had the old number live on ilo-lang.ai for weeks.
Two holes are open. I have no month of real sessions with and without graft, and no attempts-per-success figure for ilo against Python inside a live agent loop.