I wrote a Claude Code command called /audit-project to research a repository from primary sources instead of from memory, then ran it twice to see whether it earned its place. The second run found a number on my own documentation site that the vendor had retired four weeks earlier.
Here we will explore what the two runs turned up:
- a savings claim that splits by a factor of eight depending on the workload
- the difference between output tokens saved and session tokens saved
- a language that changed what it was in November and said so in July
- the arithmetic check that catches a compression claim before you read the source
The command exists because I keep being confidently wrong
Model weights carry a snapshot of every notable project, and the snapshot is wrong in a specific way. Positioning is the last thing a project updates. A repository can carry a new compiler for months while the README still describes the old tool, and a vendor can retire a headline figure while third-party write-ups keep quoting it.
So /audit-project has one rule at the top: any claim you could make before running a tool must be verified from a fetched source or deleted rather than softened. It then works through the GitHub API, a blobless clone, the release notes, the project’s own pages, and independent coverage, tagging every line in the output with which of those it came from.
The release-notes step is the one I would not have thought to include before running it. Both useful findings came from there.
The figure I was publishing had been withdrawn
ilo-lang.ai carries a page on ways to be token efficient. Its entry for caveman, a Claude Code skill that instructs the model to drop articles and filler from its replies, read “roughly 75% fewer tokens on prose-heavy turns.”
caveman’s own release v1.9.1, titled “65%, honestly”, says it “retired the ~50-65% range and the ~75% claim that still lived on the docs site.” The project corrected itself on 2026-07-03 and I was still publishing the old number on 2026-08-11. That is PR #113.
The correction is not a rounding change. Their docs/HONEST-NUMBERS.md gives the breakdown:
| Measure | Figure |
|---|---|
| Output reduction on prose | 65% average, range 22 to 87%, n=10 prompts |
| Input reduction | 0% |
| Input the skill adds | ~1 to 1.5k tokens per turn |
| Agentic coding, measured by JetBrains across 86 SkillsBench tasks | 8.5% |
| Terse replies, or billing by request rather than token | net negative |
Every figure in that table comes from caveman’s own documentation. I have run the skill but I have not A/B tested it.
Output saved is not session saved
The mechanism explains the gap. The skill is a system-prompt instruction that makes the model write shorter replies, and their documentation is explicit that it “does not compress your input, context, files, or the model’s thinking tokens.” It also occupies about 5 KB of context on every turn.
On a chat workload where the model writes a thousand tokens of explanation, cutting that by two thirds is worth more than the rules cost. On an agentic coding run, input dominates: the prompts, the file contents, the tool results and the injected rules are most of the traffic, so a shorter answer moves a small share of the total. That is where the 65% and the 8.5% come from.
Release v1.10.0 followed on 2026-08-03 and changed /caveman-stats to report net savings rather than gross, so that a workload where the skill costs more than it saves now shows as negative. Their limitations page says that if the A/B looks like that, turn it off.
What the numbers are for
I will give caveman credit for publishing the 8.5% figure at all, and note what publishing it did not change. The 65% remains the headline in the README, on the site, in the plugin manifests and in the skill description. The 8.5% is in a documentation page. A launch post for a browser tool built on the same stack claims a 129.8x reduction, 15,704 tokens against 121, on a 200-row page.
That last one fails an arithmetic check before you get near the source. A 200-row table does not fit in 121 tokens, so whatever produced that figure is not returning the rows. Comparing a full accessibility tree against a filtered view measures how much the filter discards. Both may be worth having and they answer different questions.
The five-layer product page for that stack lists one layer as live and four as in development, and the page for the compression layer states that “local token reductions are estimates labeled inferred, never verified savings.”
The same audit on a language
The first run was BAML, which I knew as a DSL for typed LLM calls. Its repository description now reads “The programming language for agents.”
The dates come from the history rather than the README. The baml_language compiler directory first appears on 2025-11-01, in a commit titled “Add foundational BAML language compiler infrastructure (#2684)”. The rebrand of the older tool lands on 2026-07-30 as “chore: brand legacy releases as BAML v0 (#4297)”. The repository now runs two release tracks at once, 0.225.0 for the original and 0.15.1 nightlies for the language, which is the thing a summary gets wrong most often.
Their parser is the interesting part for token cost. Schema-aligned parsing repairs malformed model output, unquoted strings and missing brackets and misnamed keys, rather than rejecting it and asking again. Boundary’s own figures on the Berkeley Function Calling Leaderboard put it at 92% against 87.5% for gpt-3.5-turbo function calling, n=1000. Those are vendor-run, and the comparison scores their compressed schema against JSON schema, so the columns are not measuring quite the same thing.
It is also worth being precise about what that benchmark answers. It measures first-pass accuracy, which bears on how many retries you pay for, and it does not measure the token cost of the verbose surface you write to get it.
Where this leaves ilo
ilo publishes savings claims and has no independent measurement behind them. Writing a limitations page would take an afternoon and would prove nothing. The benchmark is the hard part, and specifically one that reports tokens per attempt and attempts per success on the same task set, since a denser syntax is worth nothing against a higher retry rate.
The two runs cost about an hour and changed one line on a page I own. The command is at ~/.claude/commands/audit-project.md.