An A/B Against the Manifesto

ilo’s manifesto picks the side of total token cost. Prefix-Polish notation (+a *b c) generates 22% fewer tokens than infix across the 25 expression patterns I benchmarked in research/explorations/prefix-vs-infix/. That number is real and I have not stopped citing it.

Across 115 persona runs on main, the same handful of surface frictions recur:

• ?? x default written prefix and rejected with ILO-P009, hit by 7+ personas.
• ?h cond a b versus cond { ... } versus ?r { ~v:; ^e: }. Picking the right one. Hit by most.
• Match arm bodies that need two statements. Forces a helper function and the non-last-function safe-ending constraint. Hit by 5+.
• @x xs { body } and wh cond looking unfamiliar, agents reach for for x in xs and produce ILO-P003.

The retry tokens dominate. A persona writes ??v 0, gets ILO-P009, loads the error doc, regenerates, sometimes regenerates again. The 22% prefix saving has to clear that retry cost or it is a loss.

I built a branch to test the inversion.

The four concessions

compat/agent-natural is an additive parser-only branch off next. Every existing program keeps parsing. The skill docs on the branch lead with the natural shape; the prefix shape is still accepted everywhere.

Spec lives at SPEC-AGENT-NATURAL.md in the branch. It is honest about the cost: up to +15% generation tokens per program is acceptable if total tokens per task fall.

The setup

I sampled 25 personas from the 117-entry list with a fixed seed so the run is reproducible. Both sides run on Haiku (the same persona-token-cost rule the steady-state dogfood uses). Same prompt template, same skill discovery protocol, same task set. The only variable is which binary is on the path and which skill docs that binary serves.

Doc parity matters. The agent-natural branch ships with a separate SPEC-AGENT-NATURAL.md, but the canonical lead files (SKILL.md, ilo-language.md, ai.txt) on the branch still described the prefix surface from main. Without updating those, a “let it discover” run on Haiku measures nothing, because Haiku never sees the new shapes. I updated the lead docs first and committed (ed1f37c8) before any persona pairs ran.

The pilot pair

First persona was argmax-argmin, run on each side against the matching binary. Both produced a working program.

Initial natural-side numbers looked terrible: +20% tokens, +142% tool uses, +126% wall clock. I read the transcript. The agent reached for the canonical inline-lambda form, flt (x:n>b; x > 0) xs, and got ILO-P023. It tried again, got ILO-P023. It backed off to module-level predicates and pushed through to a working program, paying the full retry cost on every HOF call.

I ran the same shape against the natural binary by hand:

$ /tmp/ilo-targets/ilo-agent-natural/release/ilo run /tmp/lam.@
{"code":"ILO-P023","message":"named-args call on `flt` but no declared parameter names are known"}

Against main:

$ /Users/dan/code/ilo-lang/ilo/target/release/ilo run /tmp/lam.@
{"code":"ILO-T008","message":"return type mismatch"}   -- normal type error, parses fine

The natural branch carries a WIP commit (c7b8f8a6 WIP parser: named-args desugar (incomplete)). The named-args detector at src/parser/mod.rs:3479 triggers on the token shape Ident ( Ident : regardless of whether the callee is a user fn or a builtin. When the user writes flt (x:n>b; body) xs, the inline lambda’s (x: opens that detector and the parser routes into a named-args path that errors because flt has no declared parameter names.

If I had run the other 24 pairs through this state, the comparison would have measured the regression. The surface concessions would have been invisible underneath it.

What’s next

I paused the run, opened PR #611 off compat/agent-natural that gates the named-args detector on self.fn_param_names.contains_key(&name) so builtins fall through. The fix is small. The tests cover flt, map, fld with inline lambdas across tree, VM, and JIT, plus a cross-shape examples/named-args-and-lambda.@ so tests/examples_engines.rs keeps it covered. Merged into the branch.

Results so far

After the fix landed I reran the natural side of argmax-argmin and resumed the pair queue. Updated as each pair lands.

Verdict

The full 25-pair Haiku table aggregates to this:

The natural surface did not pay for itself on Haiku. Token spend is 11 % higher, tool-use count is 10 % higher, outcomes are slightly worse. Attempt count is within noise.

Haiku is the cheapest, weakest persona model in the rotation. A retry-heavy budget is exactly where natural-surface familiarity should matter most, and even there the manifesto’s prefix-canonical surface held. I reran the first 10 personas on Sonnet and Opus to test whether stronger models behave differently.

Three model classes

Same 10 personas, no-Python rule on every prompt:

The natural penalty shrinks with model class, but never crosses zero. Mean attempts tell a different story: at Sonnet and Opus, natural needs ~28 % fewer attempts. The model retries less when the syntax is familiar.

But each natural attempt costs ~45 % more tokens. Verbose if/else/for/while plus the discovery cost of the agent-natural docs. The arithmetic:

0.72 attempts × 1.45 tokens/attempt = 1.044 total tokens

Almost exactly the +4 to +6 % penalty observed. Haiku doesn’t reduce attempts at all (the model is too weak to benefit from familiar surface; it gets stuck on builtin and type-system specifics either way), so the +10 % per-attempt overhead falls straight through to the +11 % total.

The hypothesis was directionally right at Sonnet+. Natural genuinely reduces retries. But the verbosity tax is too high. To make natural pay off you would need to either trim the per-attempt overhead below ~28 %, or pick up another 5 to 10 percentage points of retry reduction. Neither is impossible. Neither is in the ticket. The data tells you exactly what each tier costs to cross.

Language gaps the run surfaced

One result I did not expect: the run surfaced a separate layer of language gaps that the surface A/B was never trying to measure. Both branches hit the same sha256-hex / hex-rev absence (block-validator, Sonnet and Haiku partialed; Opus refused to shell out to Python and reported partial honestly; Sonnet shelled out via bash + openssl to paper over it). Both hit the same jpar polymorphic-_ unification friction. Both hit the same Cranelift AOT parity bug under closure-passing HOFs. Both hit str numeric-only, wra-only file write, rd silently auto-parsing .json, the match-arm double-execute when used in non-tail position.

Those are filed as their own batch of Linear tickets in the ilo-lang workspace. Two of them have already shipped: sha256-hex / sha256d merged via PR #623, hex-rev is in review on PR #653. The rest are in backlog.

The Python baseline

The A/B against the natural branch had been ducking a different comparison. With the natural-surface question settled, I ran the same 10 personas on Opus in pure Python from scratch.

Python wins by 46 % per persona, 74 % on attempts, and solves every persona including the one ilo partialed on (block-validator, where pure-ilo lacked the raw-bytes SHA256 builtin until later that day).

The catch: ilo’s number includes spec-loading cost. Every Opus ilo run starts by reading ilo skill list plus 1 to 3 skill modules (about 5 to 10K tokens of context). Python’s number is zero on that front because the model already knows Python from training. Strip a conservative 7K of doc-loading from the ilo total and the gap closes from 46 % down to ~64 % on generation alone. ilo is still more expensive per task.

That gap isn’t an indictment of the manifesto. The spec-loading tax is the term the manifesto explicitly identifies. Two paths close it:

1. Cache the spec across many tasks per session. Already partially shipped via the modular skill split (Phase 1). The next steps are a tighter token cap (ILO-382) and a regression dogfood loop (ILO-384) so the cap can move without quietly degrading accuracy.
2. Fine-tune a model so it knows ilo natively. Filed as ILO-414. Anthropic does not expose tuning for Opus or Sonnet today, so the cleaner version is a local LoRA tune of an open-weights coder model (Qwen 2.5 Coder 14B on Mac, MLX LoRA, machine-time only) and re-run the same 10 personas in both ilo and Python on that one tuned model. Same model, both sides, appropriate priors. That’s the experiment that actually settles the manifesto’s central claim: with the spec-loading tax gone, does ilo’s denser source pay off vs Python? Cold-load today says we don’t know. The local tune answers it.

The honest read

The manifesto’s bet on prefix as token-canonical holds at every class measured against the natural-surface branch. The natural concessions don’t pay for themselves at Haiku, don’t pay for themselves at Sonnet, don’t pay for themselves at Opus. compat/agent-natural is parked, not merged.

The hypothesis behind the natural branch was that familiar-surface generation tokens would be cheaper than retry tokens. The persona corpus says retry tokens went down at Sonnet+ as predicted (28 % fewer attempts), but per-attempt token cost went up faster (45 % more per attempt). The retry-cost framing is real and worth carrying forward. The specific concessions in v0 are not the way to capture it.

The Python baseline says something harder. Even with the natural branch parked and the prefix-canonical surface as the candidate, ilo on a cold-loaded Opus costs more per task than Python on the same Opus. The path to winning runs through ILO-382 (tighter modular skill cap), ILO-384 (regression dogfood loop), and ILO-414 (local LoRA fine-tune of a coder model on the ilo corpus, then re-run the persona corpus on that one tuned model in both languages), not through surface-syntax negotiations with the agent’s training prior.

The actionable wins from this run are the language gaps.

Engine and correctness bugs surfaced

Both branches independently hit the same set of language-level issues, confirming these are not surface-canonical noise:

• Missing crypto primitives. No sha256-hex for raw-bytes hashing, no hex-rev for byte-pair reversal. Bitcoin’s Merkle tree algorithm needs both. Every block-validator run on every model class either partialed (Opus) or shelled out (Sonnet bash + openssl, original Sonnet Python). Both shipped or in review during this run: sha256-hex via PR #623, hex-rev in review on PR #653.
• Cranelift AOT parity. Multiple symptoms across personas: AOT hangs on --jit for nested mset aggregation, grp returns nil from AOT but works on tree/VM, prnt output silently dropped from AOT when lambdas are in scope. Pattern: dispatch around closure-passing HOFs is broken in the native backend.
• VM SIGSEGV on a user-recoverable type error. Config-shaper persona surfaced a clean repro: the same source that JIT diagnosed with ILO-R004 segfaulted the bytecode VM with exit 139. Type errors should never reach the segfault path.
• ?? on R t t silently produces text with a ~ prefix. No ILO-T041 diagnostic fires; the resulting text contains a literal ~ character. Both VM and JIT exhibit the bug, so it’s in shared semantic lowering. Reproduced on both branches.
• ?r{...} in non-tail position double-executes the arm body. Minimal repro: a ?r{~_: prnt "once"; ^_: _} block in a >_ function executes the matched arm twice. Tail position works correctly. Silent double side-effects are dangerous for writes and HTTP POSTs.
• rd auto-parses .json files into a Record without warning. Three independent sightings across the session. The rd : t > R t t signature reads as raw-text read; the auto-parse breaks the contract. Calling jpth! on the auto-parsed result then errors at runtime, not at type-check time.
• str is numeric-only despite skill docs claiming polymorphic. Skill says “coerces any value to text”; runtime errors on Bool and on Text (the latter especially surprising). Two attempts burned in three personas.
• rgxsuball documented but absent from the binary. Every regex-using persona that read skills/ilo/ilo-builtins-text.md reached for it and hit ILO-T005. Doc and binary disagree.
• Euler’s constant e shadows natural variable names. Match arm ^e: patterns and fold accumulator e = patterns both silently break. The reserved-name list does mention e, but the failure mode is wrong arithmetic, not a type error, which makes it hard to spot.

Each of these has a Linear ticket or sits under an existing one (ILO-371 covers the Cranelift family). Two of the eighteen tickets filed during the session have already shipped; the rest are queued.

The branch’s enduring value turned out to be this list, not the surface choice.