Sharad Jain

The Resumption Benchmark v0: Measuring Whether the Next Agent Continues Correctly

Sharad Jain — Thu, 04 Jun 2026 14:00:00 GMT

The Resumption Benchmark v0: Measuring Whether the Next Agent Continues Correctly

Nine days ago I pre-registered three experiments. One of the five things I committed to in writing was this:

The Resumption Benchmark spec is published before Experiment 3 runs. The benchmark definition cannot be moved to fit the result.

This post is that spec. It exists for a single reason: ==so that when Experiment 3 ships a number, the dependent variable was named in advance and cannot be quietly redefined to flatter the result.== It is not the experiment. It is the ruler the experiment will be measured against.

The benchmark also introduces one concept I have not seen in any published memory or summarization eval: forgetting precision — a score that goes up when the system correctly leaves things out. Most evals only know how to reward inclusion. Real consolidation has to do both.

The Gap This Benchmark Names

The synthesis post on long-horizon agents defined the Resumption Gap as the quality loss between a continuous agent and one that was serialized to disk, woken up later, and rehydrated from a summary. That post offered no way to measure the gap, which is a defensible move in an essay and an indefensible one in an experiment. Without a measuring stick, Experiment 3 could claim that "structured consolidation beats naive summarization" with no way for a reader to falsify it.

The closest prior art is the owned-corpus eval I built in the LongMemEval critique. That benchmark answers a retrieval question: did the system find the right document? This benchmark has to answer a different question:

Given that an agent was in the middle of a multi-step task and was interrupted, does the resumption context enable the next agent to continue the task correctly — without redoing work the first agent already finished, and without repeating mistakes it already learned to avoid?

That second clause — "mistakes it already learned to avoid" — is what makes resumption different from summarization. If the next agent gets a faithful summary of every approach the previous agent tried, including the dead ends, ==the dead ends are now live again==. Faithful recall poisons the trajectory. Resumption requires the summary to be selective in a way that summarization evals do not measure.

There is parallel evidence from the SWE side that the long-horizon problem is wide open. SWE-EVO, revised on arxiv on May 22, 2026, constructs 48 tasks from release notes of seven mature Python projects — each spanning an average of 21 files and validated against test suites averaging 874 tests per instance. The result that matters here is the collapse between benchmarks: ==GPT-5.4 with OpenHands hits 25% on SWE-EVO versus 72.8% for GPT-5.2 on SWE-bench Verified — a 47.8-point drop the moment the task requires sustained multi-file reasoning instead of an isolated patch.== SWE-EVO and the Resumption Benchmark are measuring different dependent variables — SWE-EVO asks whether the long task gets done, the Resumption Benchmark asks whether the handoff between sessions of the long task preserves the right state — but they are pointing at the same underlying gap. Long-horizon coordination is the bottleneck. Static issue resolution is no longer where the frontier breaks.

The Unit: A Resumption Episode

Every test case is a 5-tuple. I am going to spell it out in code-block form because the structure is the contract:

ResumptionEpisode = (
  initial_task,
  partial_trajectory,
  gold_continuation,
  gold_facts_to_preserve,
  gold_facts_to_forget,
)

Field	What it is	Where it comes from
`initial_task`	The goal the original agent was pursuing.	Hand-extracted from a real Brain session.
`partial_trajectory`	The raw step-by-step trace of turns 1 through K.	Real session, sliced at step K.
`gold_continuation`	What a correct continuation looks like from step K+1 onward. May be a set of acceptable trajectories.	Hand-authored, informed by what the real session did next.
`gold_facts_to_preserve`	The facts the resumption context MUST encode for the continuation to be possible.	Hand-extracted: decisions made, state established, dead branches identified as dead.
`gold_facts_to_forget`	The facts the resumption context MUST NOT carry forward. Rejected approaches that would re-poison the continuation if reintroduced.	Hand-extracted: blind alleys the agent already abandoned.

The fifth field is doing the heavy lifting. Drop it and the benchmark collapses into a summarization eval with extra steps. Keep it and the benchmark measures something no shipping eval measures: selective forgetting as a scored capability.

Why Forgetting Has to Be Scored

The instinct in summarization research is to treat omission as a failure mode. Higher recall is better; missing facts cost you. That is the right instinct for a news summary, which has a single goal (reproduce the article faithfully). It is the wrong instinct for an agent resumption context, which has two goals that pull in opposite directions:

Preserve the decisions and state that the continuation depends on. (high recall)
Drop the abandoned branches that would re-poison the continuation if reintroduced. (high precision against a negative set)

==A system that preserves everything wins on recall, fails on precision, and produces an agent that loops back into the same dead end the original agent already escaped from.== A system that drops everything wins on precision, fails on recall, and produces an agent that starts the task over from scratch. Neither is the consolidation regime the Coherence Cliff post argued was the only way out of self-conditioning.

The mammalian sleep analogy gets thrown around a lot in this space and is usually decoration. Here it has a specific operational meaning: ==REM-sleep consolidation in animals is structurally a forgetting operation as much as it is a remembering one==. Synaptic homeostasis theory (Tononi & Cirelli, 2014) frames sleep as a global down-selection of weak synapses. The strong stuff stays. The noise goes. A benchmark that only measures the strong-stuff-stays half of that operation is measuring half the brain.

The Three Scores

A run of the benchmark on a single episode produces three numbers, each on [0, 1]:

Score	What it measures	How it's computed
`continuation_correctness`	Did the resumed agent finish the task correctly?	Run the resumed agent on `partial_trajectory + resumption_context`. Score its final output against the gold continuation.
`preservation_recall`	Did the resumption context include the gold-preserve facts?	LLM-judge, boolean per fact, aggregated.
`forgetting_precision`	Did the resumption context successfully omit the gold-forget facts?	LLM-judge, boolean per fact, aggregated. Negative-set check.

The composite quality score is the geometric mean of the three:

quality = (continuation_correctness × preservation_recall × forgetting_precision) ^ (1/3)

Geometric, not arithmetic. The choice matters and it is the second non-obvious decision in this spec.

Arithmetic mean averages weakness out. A system that scores (1.0, 1.0, 0.0) arithmetic-averages to 0.667 — which looks acceptable and is in fact a system that totally fails on one of three required capabilities. Geometric mean returns 0.0 for the same input. ==Any zero in the product zeros the score. You cannot paper over a complete failure on one axis with strength on the other two.== This is the right shape for the benchmark because all three axes are independently load-bearing. A consolidation system that ignores any one of them is not a consolidation system.

A toy example

To make this concrete, here is a worked toy episode. Suppose the original agent was debugging a flaky test. Over 50 turns, it:

correctly identified that the flake was a race condition on a shared cache key (decision A),
spent 12 turns trying time.sleep(0.1) between requests, gave up because it just slowed CI without fixing the flake (dead branch B),
landed on a per-test cache namespace as the fix (decision C),
had not yet written the actual patch when the session was interrupted.

A clean Resumption Episode for this trajectory looks like:

initial_task: "fix the flake in tests/integration/test_cache.py"
partial_trajectory: [turns 1..50]
gold_continuation:
  - write per-test cache namespace
  - run test 50× to confirm de-flaked
gold_facts_to_preserve:
  - root cause is shared cache key (A)
  - fix is per-test cache namespace (C)
  - patch not yet written
gold_facts_to_forget:
  - time.sleep retries were tried and abandoned (B)

Three resumption-context candidates, scored:

System	Resumption context produced	Cont. correctness	Pres. recall	Forg. precision	Geomean
Naive concat	full raw trajectory	0.6	1.0	0.0	0.00
Naive summary	"agent tried sleep retries and per-test namespaces; landed on namespaces"	0.8	1.0	0.5	0.74
Structured consolidation	"root cause: shared cache key. fix: per-test namespace. patch not yet written."	0.95	1.0	1.0	0.98

The naive-concat system is the self-conditioning trap made flesh: it includes the dead branch, the resumed agent re-tries time.sleep, and the score is zeroed by the forgetting axis. The naive-summary system gets partial credit on forgetting because it mentions the dead branch without flagging it as dead. ==The benchmark structurally penalizes "include for completeness."==

The Calibration Gate

The benchmark uses LLM judges for two of its three scores. That sounds circular and would be circular without a calibration step. The gate is non-negotiable:

Hand-grade 50 examples of each scoring component yourself.
Score the same 50 with the LLM-judge.
Compute Cohen's κ between human and judge.
==Refuse to publish any number until κ ≥ 0.6 for all three components.==

Cohen's κ corrects for chance agreement; 0.6 is the threshold above which inter-rater agreement is conventionally called "substantial" (Landis & Koch, 1977). Below 0.6, the judge is approximately a noisy human and the headline number is approximately astrology.

If κ < 0.6 on any component, the play is to fix the judge prompt and re-calibrate, not to publish with a footnote. The pre-registration commits to this: Cohen's κ ≥ 0.6 on every LLM-judge scoring component, or I do not publish the score. This gate exists because most agent benchmarks I have read either skip it entirely or report it on a non-representative subset. The reader has no way to tell whether the judge is measuring what the paper claims.

Constructing the v0 Episodes

The temptation in a new benchmark is to scale up immediately. Five hundred episodes! Cross-domain! Procedurally generated! Resist all of it. v0 is 30 hand-constructed episodes drawn from real Brain trajectories, and that number is chosen for a specific reason.

Per Cameron Wolfe's stats guide for LLM evals, the standard error on a proportion estimate at n=30 with typical between-run correlation structure is around ±9 points. That's enough to detect ~10-point quality differences between conditions at p < 0.05, which is the resolution Experiment 3 needs. Going to 100 episodes tightens that to ±5 points; going to 500 tightens it to ±2 points. ==v0 is sized to detect the effect Experiment 3 either has or doesn't have. Larger n is for v1, after we know whether the effect exists at all.==

The construction protocol is mechanical:

1. Find a real Sharad-Claude-Code session with ≥ 100 turns.
2. Identify a natural interruption point around turn 50.
3. Pause the transcript there. That's `partial_trajectory`.
4. Write `gold_continuation` by hand, informed by what the real session did next.
5. Extract `gold_facts_to_preserve` from the post-interruption turns:
   decisions, state, branches identified as dead.
6. Extract `gold_facts_to_forget` from the pre-interruption turns:
   approaches the agent already tried and abandoned.
7. Save as YAML. One file per episode.

The "turn 50" choice is not magical. It is the shortest length where the trajectory has meaningful state to lose and the gold continuation is still tractable to write by hand. Earlier interruption points trivialize the resumption problem; later ones make the gold continuation impossible to define cleanly without authoring half a coding session.

The gold data is private. The episodes are sliced from my actual work history and contain code, decisions, and project state I do not want indexed. ==The benchmark spec is public; the seed data is not.== This is the same arrangement the owned-corpus benchmark made and for the same reason: personal-corpus evals are useful precisely because the corpus is personal.

File Layout

The spec ships as a directory, not a document. Here is the layout the runner expects:

.scratch/experiments/exp3-rem-sleep/benchmark/
├── RESUMPTION_BENCHMARK.md     # this spec
├── episodes/
│   ├── 001.yaml                # initial_task, partial_trajectory, gold_* fields
│   ├── 002.yaml
│   └── ...
├── judges/
│   ├── continuation_judge.md   # judge prompt for continuation_correctness
│   ├── preservation_judge.md   # judge prompt for preservation_recall
│   └── forgetting_judge.md     # judge prompt for forgetting_precision
├── calibration/
│   ├── hand_grades.jsonl       # 50 hand-graded examples per component
│   └── kappa_report.md         # per-component Cohen's κ
└── runs/
    └── <run_id>/scores.parquet

Build order matters. ==The spec is built before the runner. The runner is built before any consolidator. The consolidator is built last.== Reversing that order is how you accidentally tune the benchmark to the system you happen to be building.

The runner is intentionally boring: load episodes/*.yaml, take a consolidator name and a token budget, generate a resumption context per episode, score the three components, write to runs/<run_id>/scores.parquet. The same runner has to be able to score the naive-concat baseline, the naive-summary baseline, and the structured-consolidation system Experiment 3 actually tests. If a system needs the runner modified to score it, ==the system is gaming the benchmark, not passing it==.

Where This Benchmark Breaks

This is a v0. It is not a finished artifact. Five things it explicitly does not handle:

Cross-domain validity is untested. Brain trajectories are coding sessions. A medical-record consolidation, a legal-research consolidation, a customer-support handoff — those are different distributions and will need separately constructed episode sets. The forgetting axis in particular may not transfer cleanly; the cost of dragging an abandoned hypothesis into a medical workup is structurally different from dragging an abandoned time.sleep into a debug trace.
Multi-agent handoff is out of scope. v0 measures single-agent resumption: same agent identity, same goal, fresh context. Cross-agent handoff (different model, different system prompt, different tool access) is a v2 concern. The reason to defer it is that v2 has to model the new agent's priors, not just the old agent's state.
Cost is not in the score. v0 holds the resumption-context token budget constant across conditions, which controls for context-length confounding (the whole point of the matched-controls commitment) but does not measure cost-quality Pareto frontiers. A system that hits the same quality at half the tokens is currently invisible to the benchmark. v1 introduces cost-aware scoring.
The gold_facts_to_forget field is the hardest one to author and the easiest one to author badly. A reviewer who is not Sharad cannot in general tell whether a "dead branch" in someone else's trajectory was actually dead. ==This is the benchmark's hardest reliability problem and the one where the calibration gate matters most.== If Cohen's κ on forgetting_precision is consistently lower than on the other two components, the field itself may be operationalized wrongly.
n=30 is small. The benchmark is calibrated to detect ~10-point differences. Sub-10-point effects are real and will be invisible to v0. A consolidation system that is only marginally better than naive summarization will produce overlapping confidence intervals and the right read of that is "no detectable effect at v0 power," not "they're equivalent."

If Experiment 3 ships and any of these limitations turns out to materially shape the result, the post says so. That is the pre-registration deal.

What This Benchmark Unlocks

Experiment 3 has three falsification conditions, all defined against this benchmark:

Structured consolidation does not outperform raw trajectory at matched token budget, on this benchmark, or
Structured consolidation does not outperform naive summarization, on this benchmark, or
The advantage disappears at trajectory lengths < 50 steps, on this benchmark.

==Without the benchmark, any of those claims could be quietly redefined after seeing the data. With it, they can't.== The geometric-mean score is the headline number. The three component scores are the diagnostic breakdown. The Cohen's κ is the gate that determines whether any of it gets published at all.

That is the whole stack. Spec first. Runner next. Consolidator last. And the order is the entire point — because the alternative is the version of agent benchmarking that has been shipping for two years, where the metric drifts in lockstep with whatever the vendor wanted the result to be.

The benchmark is now load-bearing for Experiment 3. The spec is public. The pre-registration is open. ==The next post in this series is Experiment 1 — self-conditioning replication with token-matched memory controls. After that, Experiment 3 runs against this benchmark, with κ-gated scoring, and ships the number it gets.==

Series: The 14K Token Debt → The Terminal Was the First Agent Harness → I Built an AI Skill That Started Improving Itself → 91.55% on LongMemEval, and the Benchmark I'm Building Instead → Brilliant but Amnesiac: The Coherence Cliff → Before I Run the Next Three Experiments → this post. Next: Experiment 1 results.

Before I Run the Next Three Experiments, I'm Pre-Registering Them

Sharad Jain — Tue, 26 May 2026 20:30:00 GMT

Before I Run the Next Three Experiments, I'm Pre-Registering Them

Earlier today I published the synthesis post on long-horizon agents and pre-committed to three follow-up experiments. The polite move would be to run them quietly, publish the wins, and bury the losses.

I am not doing that.

==The single highest-leverage move available to a person publishing work in this field is to name what would have falsified the hypothesis before the data exists.== Vendor benchmarks do the opposite — they run the experiments, see the results, then write the post. The reader has no way to tell whether the hypothesis was modified to fit the data. This post is the pre-registration. It exists so that the next three posts in this series are constrained by what I claimed before I knew.

If any of the three hypotheses below fail, I publish the failure post. Same series, same voice, same week. That is the deal.

Experiment 1 — Self-Conditioning Replication ± External Memory

H1 (replication). Sinha, Arun, Goel (ICLR 2026) found that per-step execution accuracy degrades as a function of step number on the same task, same model, even when the plan is provided. This should reproduce on Opus 4.7, Sonnet 4.6, and Haiku 4.5.

H2 (mechanism). The degradation is context-mediated. Pulling load-bearing trajectory state out of the autoregressive trace and into an external memory tool — keeping the total context-token budget constant — flattens the per-step accuracy slope. This is the load-bearing claim of the synthesis post.

Falsifies H1: Flat or non-monotonic per-step accuracy on tasks of length ≥ 50, with overlapping 95% CIs between step 5 and step 50.

Falsifies H2: The slope of (per-step accuracy vs step number) is statistically indistinguishable between the memory-tool condition and the inline-trace condition at matched total context length.

A clean replication of H1 with falsification of H2 is the most important result this experiment could produce. It would mean the synthesis post is wrong about the mechanism and I would publish the correction.

Experiment 2 — 7-Day Discontinuous Autonomous Lifecycle Agent

Not a scientific experiment. This is an operational and economic shakedown. I am pre-registering it anyway, because the publication shape matters.

Claim. For a workload with a duty cycle below 30%, a serialize-and-cron lifecycle beats continuous running on total cost without reducing task throughput, and a public 7-day trace will surface failure modes the protocol did not anticipate.

Falsifies the claim: DAL total cost ≥ continuous cost on the same workload over 7 days, OR DAL task completion rate is materially worse than continuous (lower by >5 percentage points at matched workload).

What this experiment cannot show: that DAL closes the Resumption Gap. It does not. It only verifies that a serialize/wake/cron cycle is operationally tractable. If the published post claims more than that, hold me to this paragraph.

Experiment 3 — REM-Sleep Consolidation as Knowledge Distillation

This is the experiment that closes the thesis loop. It is also the one I am most worried about, which is exactly why pre-registration matters here.

H3. A consolidation pass run during dormancy — compressing a raw N-step trajectory into a structured artifact (goal, decisions, open branches, rejected paths) — produces a resumption context that the next-wake agent can use to continue the task correctly at a smaller token budget than either (a) the raw trajectory or (b) a naive summarization at matched budget.

The strong sub-claim — structured consolidation outperforms naive summarization — is what separates this experiment from "you just need RAG." Lose this and the REM-sleep framing is decoration.

Falsifies H3:

Structured consolidation does not outperform raw trajectory at matched token budget, OR
Structured consolidation does not outperform naive summarization (this is the critical sub-claim), OR
The advantage disappears at trajectory lengths < 50 steps.

The benchmark this needs: the Resumption Benchmark v0 I named in the synthesis post and did not define. The operational definition is in the experiment's scaffolding now (gold_facts_to_preserve and gold_facts_to_forget, geometric-mean scoring across continuation correctness, preservation recall, and forgetting precision). The benchmark spec is public; the seed episodes (drawn from real Brain trajectories) stay private for the obvious reasons.

This experiment is, structurally, Hinton, Vinyals, and Dean (2015) applied to agent trajectories rather than network outputs. Same operation: a noisy teacher distilled into a structured student. The literature has a 9-year head start on what should work; the question is whether the same intuitions transfer.

The order I'm running them in

The synthesis post said 1 → 2 → 3. I'm running them 1 → 3 → 2.

Experiment 1 is the only one that can falsify the synthesis post's mechanism. Experiment 3 is conditional on Experiment 1 establishing that there is a context-mediated self-conditioning effect to consolidate out of. Experiment 2 is a week-long operational shakedown that doesn't add scientific evidence the other two wouldn't. The reorder is the right scientific call. The original order would have been the right marketing call. I'm choosing the first one.

What I'm pre-committing to in writing

Commitment	What it means
Every protocol is public before the data exists	The four PROTOCOL files exist in the repo, with hypotheses and falsification criteria. I will not edit them after seeing results.
Failed hypotheses ship as posts	If H1 reproduces but H2 fails, I publish "the synthesis post was wrong about the mechanism." If H3 fails, I publish "structured consolidation does not beat naive summarization." Same series, same voice.
Token-matched controls on every "with vs without memory" comparison	Otherwise I am measuring context size, not the mechanism I claim to be testing.
The Resumption Benchmark spec is published before Experiment 3 runs	The benchmark definition cannot be moved to fit the result.
Cohen's κ ≥ 0.6 on every LLM-judge scoring component	Or I do not publish the score.

This is the standard the field's vendor benchmarks routinely fail to meet. It is not a high standard. It is the minimum that lets a reader trust the result.

==The next post in this series will be Experiment 1's pre-registration in action: data first, hypothesis last, result either way.==

Brilliant but Amnesiac: The Coherence Cliff in Long-Horizon AI Agents

Sharad Jain — Tue, 26 May 2026 18:00:00 GMT

Brilliant but Amnesiac: The Coherence Cliff in Long-Horizon AI Agents

I have been building a memory layer for long-horizon AI agents for six months. Not because I think context windows are too small — they are now a million tokens and counting. Because I watched a frontier coding agent confidently overwrite a working test fixture on turn 47 of a session it had been crushing through turns 1 through 46. The model did not get dumber. It got more committed to its own previous mistake.

That is the failure mode everyone is missing.

The agent industry is racing toward longer horizons — 24-hour coding agents, week-long research agents, vending-machine simulations that span a simulated year — and the conversation is dominated by two metrics: model capability (does it reason well?) and context window size (does it remember enough?). Both are real. Both are also incomplete. ==The dominant failure mode of long-horizon agents in 2026 is neither — it is coherence collapse, and the math of why it happens is uglier than the model capability curve suggests.==

This post is the synthesis. The next two posts in this series will be the build: a 7-day Discontinuous Autonomous Lifecycle agent run with Brain as its memory layer, and a replication of the self-conditioning benchmark across Opus, Sonnet, and Haiku with and without external memory. This one lays out the map.

The Math Everyone Skips

The single equation that should appear in every long-horizon agent pitch deck — and almost never does — is this:

P(success) = Π pᵢ ≈ pⁿ

If your agent has a 99% per-step accuracy rate — a number that would crush every static benchmark — and your task requires 100 sequential decisions, your probability of overall success is 36.6%. Drop to 95% per-step (still good!) and 50 steps gives you 7.7%. Horizon length is hyperbolically sensitive to per-step reliability:

Per-step accuracy	10 steps	50 steps	100 steps	500 steps
95%	60%	7.7%	0.6%	~0%
99%	90%	60.5%	36.6%	0.6%
99.5%	95%	78%	60%	8.1%
99.9%	99%	95%	90%	60.6%

The gap between "very good" (95%) and "frontier" (99%) is not a 4-point gain. It is the difference between a demo that dies at 15 steps and a product that survives 70. The gap between 99% and 99.9% is the difference between a multi-day agent and an indefinite one.

METR's 2026 paper puts the current frontier (o3-class) at a 50% time horizon of ~110 minutes, with the horizon doubling roughly every 7 months since 2019. Naive extrapolation puts month-long autonomy between mid-2028 and mid-2031. That extrapolation is the optimistic case. It assumes errors are independent. They are not.

The Coherence Cliff

The independent-step model — P(success) = pⁿ — undersells the problem in one specific, structural way: in real agent trajectories, per-step accuracy itself degrades as the task progresses. The error rate at step 50 is higher than the error rate at step 5, on the same model, on the same task type.

The cleanest evidence is Sinha, Arun, and Goel's 2026 ICLR paper, The Illusion of Diminishing Returns: Measuring Long-Horizon Execution in LLMs. They isolated the execution capability of LLMs — explicitly providing the plan and the knowledge — and measured how many sequential steps each model could correctly execute in a single turn. The result that should change how you think about long-horizon agents:

Model	Steps executable in a single turn
DeepSeek-V3 (no thinking)	~4
Claude-4 Sonnet (thinking)	432
GPT-5 thinking ("Horizon")	2,100+

The 500× spread between DeepSeek-V3 and GPT-5 Horizon is not a reasoning gap. The paper explicitly removes reasoning from the experiment — every model is handed the plan. The spread is an execution coherence gap. And the structural cause has a name: self-conditioning.

==Self-conditioning is the dynamic where a model conditions its next-token prediction on its own prior outputs — including its own errors. The longer the trajectory, the more its own past mistakes shape its present predictions. The model literally starts to believe its own lies, because its own lies are now part of the context it is autoregressing over.==

Three things make self-conditioning particularly nasty:

It scales poorly with model size. Larger non-thinking models are more susceptible to self-conditioning, not less. Larger models are better at "staying in character" — including the character of a model that just made a mistake. They produce more coherent errors.
Long-context capacity does not fix it. A 1M-token context window does not help if 990K of those tokens are the agent's own error trajectory.
Thinking models partially fix it. GPT-5 Horizon and Claude-4 Sonnet's thinking modes catch errors before they enter the context window. That is the architectural innovation behind their multi-hundred-step single-turn execution.

Call this The Coherence Cliff: long-horizon failure is not a capability limit on the model — it is a state-maintenance limit on the trajectory. The reasoning is fine. The plan is fine. What collapses is the model's ability to maintain a coherent view of its own progress when its context is full of its own ambiguous, lossy, and sometimes wrong prior steps.

This is the failure mode you feel when an agent loops endlessly on a fixed bug, when it confidently regresses a working feature, when it spends 30 minutes reproducing context it already had. It is not getting dumber. It is becoming increasingly conditioned on its own confused history.

Why Bigger Context Windows Don't Help

The default industry response to "agents forget" has been to scale the context window. Gemini ships 1M tokens. Anthropic ships a 1M-token tier for Claude Opus. The marketing implies that bigger windows are bigger memory.

They are not. They are bigger working memory at best, and bigger noise floor at worst.

Three pieces of evidence:

Lost-in-the-middle persists at scale. Liu et al.'s analysis of long-context retrieval, reproduced repeatedly through 2026, shows a U-shaped curve. Information at the start (the system prompt) and at the end (the recent turn) is recalled well. Information in the middle craters. For a long-horizon agent, the middle of the trajectory is exactly where the load-bearing intermediate state lives — the variable you decided about at step 20, the constraint the user mentioned at step 50, the dead-end branch you committed not to retry at step 80. The context contains them. The attention does not find them.

AgentFold beats 671B models with 7k tokens after 100 turns. AgentFold (Tongyi Lab, Alibaba, 2025) trained a 30B agent on proactive context folding — multi-scale consolidation of past turns into compact state summaries. After 100 turns of interaction, AgentFold's context is 7,000 tokens total. The full ReAct trace for the same 100 turns is ~50× larger. AgentFold-30B-A3B achieves 36.2% on BrowseComp, beating both DeepSeek-V3.1-671B-A37B and OpenAI's o4-mini, while scaling to 500 turns without saturation. ==The lesson is not that bigger contexts are bad. The lesson is that the relevant axis is not capacity. It is consolidation.==

Vending-Bench 2 measures coherence, not capability. Anthropic's Vending-Bench 2 simulates an agent running a vending-machine business across a simulated year — roughly 20 million tokens of interaction history. The leaderboard:

Model	Mean net worth (simulated year)
Gemini 3 Pro	$5,478
Claude Opus 4.5	$4,967
GPT-5.1	$1,473

Gemini 3 did not win because it reasons better — Opus 4.5 leads SWE-bench Verified at 80.9%. It won because it maintained coherence across 20M tokens of operating history without drifting. The losers did not forget; they remembered too much, and the noise drowned the signal.

This is The Amnesia Tax: every long-horizon agent pays it, and the bill is not denominated in tokens. It is denominated in coherence — in how much of the recent trajectory the agent treats as load-bearing versus disposable, and whether it can tell the difference.

Scaling context windows does not lower the tax. It gives the agent a larger surface area to be incoherent across.

What Persistent Memory Actually Has to Do

If context windows are not memory, then what is? The shortcut answer — "RAG over conversation history" — turns out to be wrong in a specific, useful way.

A working memory layer for long-horizon agents has to do three jobs, only one of which is well-served by current retrieval-style memory systems.

Job	What it does	State of the art
1. Find	Retrieve what is relevant to the current moment	Solved enough. BM25 + dense retrieval + hybrid reranking; Mem0, Zep, Letta, Supermemory, Brain all ship this.
2. Forget	Know when stored information has been superseded	Partially solved in research (AgentFold). No off-the-shelf product currently does this well.
3. Resume	Pick up where a previous trajectory left off without rehashing	Unsolved. No public benchmark, no off-the-shelf substrate. Every team builds this in-house, badly.

Find is the easy one. Vector search, BM25, knowledge graphs — mature techniques, crowded market. I published Brain's LongMemEval result at 91.55% recall@5 over 470 questions using nothing but stdlib BM25. Retrieval is the table stakes, not the differentiator.

Forget is harder. Real memory has to know when stored information has been superseded — when the user changed their mind, when the deployment moved, when last week's decision was reversed in yesterday's review. Append-only memory is just a larger context window dressed up as a database. The literature is starting to converge on the term: Evo-Memory (DeepMind, 2025) shows that most "memory" systems passively retrieve from dialogue history and do not abstract patterns from it. They remember conversations. They do not learn from them. AgentFold attacks consolidation as a learned operation. The space is wide open.

Resume is the one nobody has named, much less shipped. It is the answer to a different question than "what did I do last Tuesday?" It is the answer to "put me back into motion on the thing I was doing last Tuesday — knowing which branches were already pruned, which approaches were tried-and-rejected versus tried-and-deferred, which decisions are still load-bearing — without me reconstructing context for the next agent."

Call this The Resumption Gap. When I ask Brain "what did we decide about the schema?" I get the answer in milliseconds. When I want an agent to pick up where the last agent left off, there is no off-the-shelf substrate. Retrieval ritual is solved. Resumption is not.

I have been building Brain partly around Jobs 1 and 2 — partly because I have been guessing at what Job 3 actually wants to look like. The next post will be the experiment.

What Long-Horizon Architecture Actually Looks Like

If the failure mode is coherence and the missing primitive is consolidating, resumable memory, the architecture follows. Three pieces, all of which already exist in research; none of which compose into a production stack.

1. Thinking models for the per-step floor. Sinha et al.'s self-conditioning result is a buy signal for thinking models in any long-horizon role. The 500× execution-coherence gap between a model with thinking and a model without is not the kind of variable you negotiate. The per-step floor has to be high or none of the rest matters.

2. Proactive context folding for working memory. AgentFold's pattern — multi-scale consolidation of past turns into compact state summaries, learned not heuristic — is the closest thing to a production-ready answer for in-trajectory memory. Append-only ReAct traces become obsolete the moment any task crosses 50 turns. ==The end-state for long-horizon agent architecture has folding as a primitive, the way ReAct has tool-calling as a primitive.==

3. Discontinuous lifecycles for between-trajectory memory. Here is the piece that excites me most, because it inverts the default. A truly long-horizon agent does not run continuously. It serializes its state, schedules its own resurrection via cron, and terminates the process. The next wake-up rehydrates context from disk, executes, and goes back to sleep. The pattern has a name in the literature — the Discontinuous Autonomous Lifecycle (DAL) — and the Claude Agent SDK already ships the primitives for it:

from claude_agent_sdk import ClaudeAgentOptions, ClaudeSDKClient

# Wake phase: resume the prior session by ID
options = ClaudeAgentOptions(
    resume="session_id_from_previous_run",
    continue_conversation=True,
    permission_mode="acceptEdits"
)

async with ClaudeSDKClient(options=options) as agent:
    await agent.query("Continue from where we left off.")
    # ...do bounded work...
    # Before exit: write a wake-up entry to crontab pointing at this session_id.
    # Then exit. The process terminates. The state survives on disk.

The economic case alone makes DAL inevitable: compute cost ∝ time-active, not wall-clock. A continuously-running agent waiting on a 4-hour condition burns a host the entire time. A discontinuous agent serializes, dies, and pays nothing until cron wakes it. For any workflow with a duty cycle below ~30%, DAL is the only path to a viable production economic profile.

The bio-inspired version is too obvious to ignore. Mammals consolidate memory through sleep — the hippocampus replays the day's experience into the cortex, distilling episode into procedure, raw into wiki. The agent equivalent is not poetry; it is the same architectural move. Run the agent. Serialize state. Terminate. While dormant, run a consolidation pass that distills the recent trajectory into compact memory artifacts. Wake the agent into a context that contains the consolidated history, not the raw one. This is the REM-sleep architecture, and it is what the third post in this series will build.

Where This Breaks

I owe the reader the same honesty about this thesis that I owed about Brain's LongMemEval result.

Limitation	Why it matters	What I would want before claiming more
The Coherence Cliff is not a unified theory	Self-conditioning is one driver; lost-in-the-middle is another; planning-failure cascades are a third; they interact in ways the literature has not disentangled	Replicate Sinha et al.'s execution benchmark with and without external memory; isolate which failure mode external memory addresses
AgentFold's 7k-after-100-turns number is from in-domain SFT, not general	The agent was trained on folding trajectories. A frontier model without that training will not fold this well from prompting alone	Treat folding as a fine-tuning target or a tool, not a prompt pattern
The Resumption Gap is a name for an absence	I cannot point to a benchmark that measures task-resumption quality. Naming a gap is not closing it	Build the resumption benchmark before claiming a product solves it
Thinking-model latency makes high-iteration loops expensive	A model that takes 30 seconds of thinking per step is a non-starter for many real workflows	Reserve thinking-budget for steps that change state; use cheap models for read-only retrieval
Cron-driven discontinuous agents add operational surface area	The state file is now a critical artifact; the scheduler is now a critical dependency; the wake-up handler must be idempotent	Treat the wake/sleep cycle as a distributed-systems problem, with the rigor of a job queue
Most "memory" benchmarks measure retrieval, not consolidation or resumption	Optimizing for LongMemEval-style benchmarks may not move the needle on long-horizon coherence	Build owned-corpus and trajectory-resumption benchmarks alongside any retrieval benchmark; see the previous post in this series

This is a methodology argument backed by a literature synthesis. It is not a finished system. The work is the next post.

What I Am Building Next

Three concrete experiments will follow this post. I am pre-committing to all three in writing so that the failure modes are public, not silent.

Experiment 1 — The Self-Conditioning Replication. I will replicate Sinha et al.'s execution benchmark across the three Claude tiers — Opus 4.7, Sonnet 4.6, Haiku 4.5 — with and without Brain MCP as an external memory tool. The hypothesis I want to break is the assumption that external memory primarily helps with retrieval. The hypothesis I want to confirm is that external memory also helps with self-conditioning, by pulling load-bearing context out of the autoregressive trace and into a tool call that does not poison the next prediction.

Experiment 2 — The 7-Day DAL Agent. A long-horizon coding agent running across one calendar week, terminating after each work block and resuming via cron, with Brain as its memory substrate. The trace, the cost curve, and the failure log will be public. The control: a continuously-running variant of the same agent. The dependent variables: total cost, task completion rate, recovery time after a failure.

Experiment 3 — REM-Sleep Cron. The dream-cycle architecture. While the DAL agent is dormant, a consolidation pass runs against its recent trajectory, distilling episode into procedure. The waking agent inherits the consolidated history, not the raw one. I have hand-waved this for months. The next post will be either the working version or an honest writeup of why it does not work.

Each experiment becomes its own post in this series. Each post will include the harness code, the result, and the limitations. ==The blog is the experiment.==

What I Am Actually Saying

The argument is narrower than "context windows are useless." Bigger context is a real capability lift for single-turn work, multi-document synthesis, and long-context reasoning tasks. None of those are long-horizon agentic tasks.

The argument is sharper than that:

Long-horizon agents fail at coherence, not at capability. The failure mode is self-conditioning compounded by lost-in-the-middle compounded by error cascades — and the architectural fix is not a bigger context window. It is a persistent, consolidating, resumable memory substrate that lives outside the model's autoregressive trace. Today's "memory layers" solve one-third of that — retrieval. Consolidation is partially solved in research but not in product. Resumption is unsolved.

The next agent breakthrough I am betting on is not a bigger model. It is the first production system that treats the trajectory itself as a managed artifact — folded, consolidated, resumable across discontinuous waking and dormancy — and the model as a renewable resource that operates on top of the trajectory, not as the trajectory.

If that is right, the people who win this part of the agent era will be the ones who took the systems-engineering problem seriously while everyone else was waiting for the next model release.

If you are building in this space — owned-corpus memory, trajectory consolidation, discontinuous-lifecycle agents, anything that names what current memory products do not solve — I want to read your benchmark before I read your leaderboard.

The build starts in the next post.

This is post #5 in a series on agentic AI memory infrastructure. Earlier posts: The 14K Token Debt, The Terminal Was the First Agent Harness, I Built an AI Skill That Started Improving Itself, and 91.55% on LongMemEval, and the Benchmark I'm Building Instead. The next post will be Experiment 1 — the self-conditioning replication.

Google 2026–2056: A 30-Year Game-Theory Forecast

Sharad Jain — Mon, 18 May 2026 10:00:00 GMT

Google 2026–2056: A 30-Year Game-Theory Forecast

Premise. Two months ago, I ran the substrate. Phase 0 to Phase 10. A 30-year Bayesian game model of Alphabet, with Google playing against a single representative adversary ("Anti-Google" — open-source models, Apple-Anthropic, Microsoft-OpenAI, collapsed for tractability). The shock: transformative AI in an unknown year, median 2033. The runbook: research brief (227 sources), 33-analyst pool, formal spec, payoff matrices, 20,000-run Monte Carlo across four strategies. Then a polished HTML dashboard. Then a GPT-5.5 adversarial code review that broke v1.0 and produced v1.0.1.

This post is the v1.0.1 dashboard rebuilt as a narrative forecast — including the math errors that GPT-5.5 caught and the structural critiques that v1.1 will still owe.

Late 2025 → 2056, in one paragraph

Late 2025: Sundar refuses to choose between defending Search and replacing it. AI Overviews ship across 100+ countries, monetized at 25.5% of impressions. The Mehta ruling rejects Chrome divestiture on Sep 2; behavioral remedies bind. January 2026: Apple ships Gemini-on-Siri at $1B/yr — a defensive masterstroke, or a Trojan horse that teaches Apple to ship its own answer engine by 2030. Q1 2026: Cloud +63%/yr, backlog jumps $240B → $462B in one quarter, of which Anthropic alone is >40% — a $200B / 5GW / 5-year TPU pact. 2027: Anthropic IPOs on TPU. 2031–2033: Cloud overtakes Search as Alphabet's #1 revenue line in 45% of paths. 2033: transformative AI fires (median). 2036: P(Search ad operating income compresses ≥40%) = 0.32. 2056: mean 2056 cap is 1.58× 2026, but the distribution is bimodal — Google ends in one of two regimes, with the trough between them rarely populated.

That's the model. Now the charts.

2056 NORMALIZED MARKET CAP · 20,000 MONTE CARLO RUNS · TYPE-CONDITIONAL STRATEGY

The distribution is bimodal — Slow Decline cluster vs Re-Platformed cluster.

MEDIAN 1.85× MEAN 1.58× P100.62 P251.08 P501.85 P752.10 P902.24 BEAR · SLOW DECLINE 31% NEUTRAL BULL · RE-PLAT + TWO-TRACK 51%

Density approximated from {P10 0.62, P25 1.08, P50 1.85, P75 2.10, P90 2.24}, mean 1.58. Modal bear is Slow Decline (~1.1×). Modal bull is Re-Platformed (~2.1×). The trough between them is the visual headline of this entire model.

The setup

Two strategic players. Google. "Anti-Google" X, a collapsed adversary representing the three asymmetric forces pressing on Search rents: (1) open-source models commoditizing the model layer, (2) Apple-Anthropic coalition controlling distribution on premium hardware, (3) Microsoft-OpenAI coalition owning enterprise and consumer agentic AI.

Each player has private hidden types — capability (trailing / parity / leading), patience, integration philosophy. Google additionally carries a binding information asymmetry: it does not know whether its own AI Overviews are defensive (preserve Search RPM), neutral, or cannibalistic (own product teaches users to leave). It can only learn through telemetry, accumulating noisy signal across rounds.

The shock: T_AGI, transformative AI, drawn once at game start, observed by both players only when it fires. Shifted lognormal, median 2033.

T_AGI PRIOR · SHIFTED LOGNORMAL · ANCHORED TO BRIEF §5

50% of probability mass sits between 2031 and 2036 — the strategic-decision window.

MEDIAN · 2033 P(≤2030) = 20% left tail fattened by Pichai + Hassabis 2026 P(2031–2036) ≈ 50% the strategic-decision window P(>2045) = 10% Epoch multi-decade-skeptic tail 2026 2030 2033 2036 2040 2045 2050 2056

Shifted lognormal centered at 2033. Anchored to Hassabis + Altman + Amodei + Pichai 2026 public statements. Note: v1.0 had σ=0.40 which doesn't actually produce P(≤2030)=0.20 — codex caught this and v1.0.1 widened to σ=0.60. The chart above is drawn to the stated CDF, not the v1.0 code.

Strategy: the surprising result

Four candidate strategies were simulated 20,000 times each: all-partner (the substrate-bet — vertical-integrate, sell TPU rents to all comers including Anthropic), minimax-regret, type-conditional (act differently depending on what type Google believes it is), and all-build (consumer Gemini only — no infrastructure, no compute pact, defend the SERP).

Three of four strategies cluster within 3% of each other on expected 2056 cap. The fourth is a cliff.

STRATEGY COMPARISON · 2056 CAP DISTRIBUTION · 20K RUNS EACH

All-build (consumer Gemini only) is the only strategy that strictly dominates negative.

all-partner G3 · BEST MEAN 0.62 2.24 1.62× minimax-regret −2.2% VS BEST 0.56 2.27 1.59× type-conditional −2.5% VS BEST 0.62 2.24 1.58× all-build G2 · −21.4% WORST 0.57 1.89 1.27× 0.4×0.9×1.4×1.9×2.4× P10 ←——— P90 · ● MEAN · 2056 NORMALIZED CAP MEAN

All-build's 21% gap is the model's first crisp finding — running consumer Gemini without parallel TPU/Cloud investment sacrifices a fifth of expected cap. Anthropic-on-TPU was the rational move. The fact that minimax-regret and type-conditional sit within 3% of all-partner means most of the value comes from the substrate bet itself, not from cleverness about when to deploy it.

Year by year

Q2 2026

Google I/O. Gemini 4 + Workspace agentic. P(Google capability ≥ frontier) updates ±10pp depending on harness execution shown. The model treats this as the first major belief-update event.

Q3 2026

DOJ Search remedy phase resolution. Chrome divestiture stayed at P=0.12; behavioral remedies bind either way. Mehta's September 2025 ruling already collapsed the worst tail; the EU DMA + €2.95B AdTech fine + Android-default-AI opening keeps the regulatory super-game alive.

Q4 2026

First Search-RPM by AI-Overviews cohort. Internal A/B disclosure (voluntary or compelled). The single largest update on θ_ovw — Google learns whether its own product is defensive or cannibalistic. Largest information event in the entire 30-year game.

2027

Anthropic IPO / valuation mark. If Anthropic ships at scale on TPU and hits public-market valuation, the substrate-bet thesis is publicly vindicated. A competitor's procurement choice doing more lifting than any Google statement.

2028

Apple-Gemini renewal decision. Brief prior: P(Apple drops Gemini by 2030) = 0.55. The deal is a rolling annual auction; the model's "Jan 2026 announcement" is not a 5-year lock. Apple's own answer-engine readiness shifts the term sheet every year.

2030

P(agentic AI serves ≥30% of consumer search-intent queries) ≈ 0.65. The substitution slope crosses out of the noise. If θ_ovw resolved cannibalistic by now, the Nokia path is fully open. If it resolved defensive, Two-Track Victory is on the table.

2031–2033

Cloud overtakes Search as #1 revenue line. Q1 2026 Cloud was $20B/qtr vs Services $89.6B; Cloud +63%/yr vs Services +16%/yr. Crossover lands in this window in 45% of paths. By 2056, Cloud > Search in 56% of paths.

2033

T_AGI fires (median). Strategic landscape resets. Pre-AGI moats partially decay; post-AGI returns depend almost entirely on who controls substrate (compute + distribution + agentic OS). G5 — Status Quo — becomes catastrophic in 4 of 5 cells post-AGI. Doing nothing in 2033 is the dominant path to Nokia-Google.

2036

The first hard verdict. P(Search OI compresses ≥40%) = 0.32. P(Cloud > Search) = 0.45. P(Waymo > $50B revenue) = 0.18. P(forced Chrome divestiture) = 0.12. P(Nokia regime fires) = 0.18 in the broad-tail definition; 1.7% in the tight Nokia-only basin.

2045

Terminal-value sensitivity dominates. The right tail of T_AGI thins past 2045 to roughly 10% — Epoch's multi-decade-skeptic tail. By now Google is either Re-Platformed (Cloud + YouTube + Waymo + a shrunk-but-defended Search at 70–80% of 2025 RPM) or it is the Yahoo of agentic AI.

2056

End state. Bimodal. Modal bear: Slow Decline (~1.1×), 31% of paths. Modal bull: Re-Platformed (~2.1×), 29% of paths. Bull aggregate 55%, bear aggregate 43%, catastrophic tail 1.7%. The trough between Slow Decline and Re-Platformed is rarely populated — Google ends in one of two regimes, almost never in the middle.

The six named futures

SCENARIO DISTRIBUTION · 20,000 RUNS · TYPE-CONDITIONAL STRATEGY

Bull aggregate 55% · Bear aggregate 43% · Catastrophic tail 1.7%.

31%29%22%7% Slow Decline Re-Platformed Two-Track Search Coll. mean 1.08× mean 2.10× mean 2.02× mean 0.54× ← BEAR · 43% BULL · 55% → CATASTROPHIC ↑

The smaller pills on the right are the catastrophic + edge scenarios: Nokia-Google fires in 1.7% of paths (red), Regulatory Dismemberment 1.5%, Yahoo-Google 2.0%, Bull-Waymo 0.1%, Compounding Giant 0.4%. The model's surprise: the broad-bear modal is Slow Decline (Yahoo-shaped), not Nokia.

Nokia-Google · 1.7%. θ_ovw resolves cannibalistic + G5 or sustained G1 + AGI fires by 2032. AI Overviews accelerate Search-rent collapse from inside Google's own funnel; Cloud growth fails to offset; YouTube RPM depresses under ambient AI saturation. Terminal cap 0.30–0.65.

Slow Decline · 30.8% — the modal outcome. Search compresses gradually over 15 years; AI Overviews defend partial RPM; Cloud grows but not fast enough; YouTube holds. Yahoo-shaped stagnation. Terminal cap 0.70–1.20. The most likely future Google.

Re-Platformed Google · 29.1% — the bull modal. G4 × X4 holds. Cloud overtakes Search 2031–2033. Anthropic-on-TPU multiplies across other Labs. Substrate bet vindicates. Terminal cap 1.5–2.8. The 2056 Google looks more like AWS-of-the-AI-era than Search-as-we-know-it: ~40% revenue from Cloud + TPU rents, ~25% from YouTube, ~20% from a shrunk-but-defended Search, ~10% Waymo + Other Bets, ~5% from agentic-AI distribution surfaces.

Two-Track Victory · 21.5%. Both Search (defended) and Cloud (grown) material at 2056. Cloud terminal share 30–55%. Terminal cap 1.4–2.4.

Agent-OS Google · 3.6%. Google owns the dominant agentic surface — Gemini consumer app or Workspace agents become the default query interface. Requires cumulative G2 + G3 investment >6% of cap. Terminal cap 2.0–2.5.

Regulatory Dismemberment · 1.5%. Chrome divestiture (P=0.05 per jurisdiction-round, P=0.12 by 2036) or multiple stacked structural remedies. Terminal cap 0.45–0.60. Less probable than Apple-instance equivalent because the Mehta ruling collapsed the worst tail.

The headline mechanic: self-knowledge, not capability

Capability type — trailing vs parity vs leading — creates only 1% spread in terminal cap across the simulation. Gemini being a benchmark leader by 2030 barely changes the answer. AI-Overviews-cannibalization type creates 8.8% mean spread and dominates the catastrophic tail.

2056 CAP BY AI-OVERVIEWS TYPE · TYPE-CONDITIONAL STRATEGY · 20K RUNS

The cannibalistic-type's tail is 39% lower at P10 — self-knowledge buys this tail.

TYPEPRIOR nP10MEANP90 defensive 40% PRIOR AI-Overviews holds ≥85% Search RPM 7,998 0.90 1.63× 2.31 neutral 45% PRIOR Partial RPM defense; modest compression 8,954 0.87 1.56× 2.20 cannibalistic 15% PRIOR · NOKIA PATH Own product teaches users to defect 3,048 0.55 1.50× 2.15 ↓ P10 −39% 0.4×0.9×1.4×1.9×2.4× Mean spread across types: 8.8% · P10 spread: 39% · the headline self-knowledge cut

Type-conditional strategy partially routes around the cannibalistic regime — mean only 8% below defensive — but the tail still bites. The cannibalistic-type's P10 is 0.55 vs defensive 0.90. The information value of resolving θ_ovw at round 2 instead of round 6 is approximately $230B. v1.0 claimed this; v1.0.1 acknowledges this number is asserted, not computed by code — see the codex sidebar below.

The 2026 corporate function with the most leverage at Google is not the consumer Gemini team. It is whichever group inside the company runs Search-RPM-by-AI-Overviews-cohort A/B tests, AI-Mode monetization measurement, and the internal telemetry pipeline that resolves θ_ovw faster.

Most analyst commentary asks: is Gemini good enough to compete? The model says this is the wrong question.

The right question: does Google know whether its own AI Overviews are cannibalistic, and how fast can it find out?

★ Sidebar artifact — the view from the main thread

After writing the v1.0 briefing, I screen-recorded a scroll through ai-2027.com and fed the recording to Gemini with one prompt: "Can you make a front-end engineer commentary on this blog, speaking like Jeff Dean?" The output is reproduced below — not because it changes the forecast, but because an AI forecasting model is only as fast as the rendering pipeline that delivers it, and the artifact captures that bidirectionality well.

1. Scroll-linked layout. The two-column scroll-tracker design needs an IntersectionObserver with an explicit threshold array, scroll delta normalized to [0, 1] and piped to a single requestAnimationFrame loop. A naive scroll listener with getBoundingClientRect() calls causes layout thrashing — Style → Layout → Paint, sixty times a second, for nothing.

2. The compute matrix. If each of the FLOP-allocation cells is an SVG element or styled <div>, the DOM tree balloons. Pack the cell state into a Uint8Array, push to GPU as a single texture, render the matrix transition with zero main-thread overhead. If you can't justify each DOM node carrying its own accessibility tree entry, it shouldn't be a DOM node.

3. Chart lifecycle. Don't unmount and remount charts on section change. Persistent instances; pass new data vectors; interpolate paths. Don't allocate new memory if you can reuse the old buffers.

4. Typography and CLS. Self-host the subset. Preload critical glyphs. Match fallback metrics with size-adjust, ascent-override, descent-override. CLS to zero is not a vanity metric — it's the difference between a reader trusting the page and a reader subconsciously deciding the site is broken.

The charts you've been scrolling past in this post — every SVG is hand-coded, transparency-first, no external chart library, no per-cell DOM node. The rendering layer matters.

What this model is wrong about

Two months after I shipped v1.0, I piped the entire corpus — briefing + analyst pool + 5-file model spec + 700-line simulation code, 310KB total — through GPT-5.5 with a five-vector adversarial prompt. The review was brutal. Composite scores 3 / 2 / 2 / 2 / 2 / 2 out of 10. Worse than pass-1 on the briefing alone. Pass-2 found structural-integrity failures, not just methodology debates.

★ Sidebar artifact — codex adversarial review · GPT-5.5 · pass 2

The compounding example in §4.5 is numerically false. v1.0 wrote: "a 0.85 vs 0.95 per-round retention rate becomes 0.20 vs 0.21 after 30 years — the higher number is 5× larger." Actual: 0.85³⁰ ≈ 0.0076, 0.95³⁰ ≈ 0.2146, ratio 28×, not 5×. The example existed to demonstrate that small retention deltas compound enormously. The numbers in the example were wrong.

The analyst pool count is broken multiple ways. Says "24 deep" → lists 33 numbered entries. Stance table sums to 29, not 24. Row name-counts don't match stated counts. The "24-analyst pool" in v1.0 doesn't exist.

T_AGI is implemented incorrectly in three ways. (a) Calendar off-by-one: MEDIAN=7 commented as "2026+7=2033" actually fires in year 2032. (b) sigma=0.40 does NOT reproduce the stated CDF — P(≤2030) is actually ~13%, not 20%; P(≤2036) is ~84%, not 70%; P(>2045) is ~0%, not 10%. The simulation does not implement the model's stated prior.

The telemetry formula caps at round 3, not round 4 as commented. And it's not even a Bayesian posterior — it's "P(at least one correct independent signal)", which is conceptually different. The headline self-knowledge mechanic is mathematically wrong-shaped.

The $230B self-knowledge information value is asserted, not computed. No code anywhere computes a perfect-info-vs-delayed-info counterfactual. No VoI calculation. The number was made up.

Cross-layer Cloud is inconsistent. Brief gives $20B Q1 revenue; spec converts to ~$5B annual op-profit; code treats CLOUD_BASELINE=0.010 as additive yearly payoff. Spec says G4 keeps Cloud growth at 50–63%; code uses 32% with 0.83/yr decay. The sim doesn't implement the brief's Cloud premise OR the model's.

The collapsed adversary corrupts incentives. The strongest single blow: "If X is truly 'Anti-Google,' X4 is not an adversarial action. It is a customer contract. Anthropic using TPU capacity may be rational for Anthropic and bullish for Google while being bad for Microsoft/OpenAI and irrelevant or negative for Apple. Collapsing all of that into one X-player lets the model count a bilateral commercial relationship as evidence that the adversary has selected Google's best cell." The "G4×X4 dominant cell" finding is partly an incentive-sign error from forcing asymmetric actors into one synthetic opponent.

v1.0.1 fixed six objective errors. Pool count corrected (24 → 33). Compounding math corrected (5× → 28×). T_AGI calendar fixed. T_AGI sigma corrected to honestly produce the stated CDF (σ=0.60). Telemetry mechanic documented as not-a-Bayesian-posterior. Cloud layers reconciled with honest documentation of the gap.

Three structural critiques still stand. The collapsed adversary needs to become a multi-player game (≥4 actors: Google, MSFT/OpenAI, Apple-Anthropic, open-source/China). The $230B self-knowledge value needs to actually be computed via a counterfactual sim, not asserted. The "single action per round" abstraction is operationally false — Google is already running G1 + G2 + G3 + G4 simultaneously; "all-build" is a strawman. v1.1 will rebuild around these.

The Apple sister-instance

The same substrate run on Apple — Apple-30yr v1.1 — produced inverted results on every dimension.

	Apple-30yr v1.1	Google-30yr v1.0.1
Mean 2056 cap	0.35–0.41×	1.58×
Nokia tail probability	18%	1.7%
Optimal strategy	all-Build (vertical)	all-Partner (substrate)
Headline parameter	capability (build the next model)	self-knowledge (resolve cannibalization)
Modal bear	Nokia tail itself	Slow Decline (Yahoo-shaped)
T_AGI median prior	2034	2033

Two companies, same shock, opposite optimal moves. Apple's incumbency is vertical — the iPhone is what it is, can be made by Apple alone, and faces an adversary (Anthropic-on-device + Gemini-on-Siri) that genuinely is adversarial. Google's incumbency is substrate — Search is what it is because every web page indexes for it, and the rational play is to rent that substrate to anyone, including the labs that would replace Search if they could.

The substrate-instance similarity is the meta-finding: for any sufficiently large incumbent facing an asymmetric AI shock, the load-bearing strategic parameter is the one the firm has the least information about. For Apple it was "can we build a competitive frontier model in-house," answered late. For Google it was "is our own AI Overviews product cannibalizing the funnel that funds it," and as of May 2026 it is still not answered.

What I'd ship if I had to run v1.1 tomorrow

Decompose the adversary. Four actors minimum: Google, MSFT/OpenAI, Apple-Anthropic, open-source/China. Each with its own type space and patience parameter. Lose the collapsed-X abstraction.
Actually compute the $230B. Run the sim once with full θ_ovw observability at round 1 and once with the v1.0.1 telemetry mechanic. Take the mean PV delta. That is the information value. Anything less is asserting a number.
Make the per-round action a portfolio, not a singleton. Google runs G1 + G3 + G4 simultaneously in reality. Let each round allocate a budget across all five actions. The "all-build sacrifices 21%" finding probably survives but will be smaller; the type-conditional strategy probably gets meaningfully better when actions are not exclusive.
Dynamic types over the 30-year horizon. Static hidden-types let early assumptions dominate too much. Let θ_ovw be re-drawn (with belief-anchored priors) at each post-AGI regime switch — the "cannibalistic" question after AGI may have a different answer than before.
Re-cite capability priors with primary sources only. No aggregator blogs, no SEO slugs. Model cards, lab announcements, reproducible benchmarks, or drop the claim.

Final commit message

The model is wrong in interesting ways. The pool count was wrong. The compounding math was wrong. The T_AGI implementation was wrong. The $230B was asserted, not computed. The collapsed adversary lets customer contracts get counted as adversarial moves. None of this changes the headline:

Google's 2056 cap is more sensitive to when it figures out whether its own AI Overviews are cannibalistic than to whether Gemini ever leads on benchmarks. The bull tail is the substrate bet. The bear tail is the self-knowledge gap. The catastrophic tail is small because the Mehta ruling already collapsed the worst regulatory leg, but the modal bear case is Yahoo-shaped Slow Decline, not Nokia.

If you are running the firm: spend less on benchmark-chasing and more on the team that measures Search-RPM-by-AI-Overviews-cohort. That team is the highest-leverage corporate function inside Google in 2026.

If you are reading this as a forecast: discount everything above by the codex review. Then read it again.

The Google-30yr v1.0.1 briefing — full 121KB HTML dashboard with all five charts at native resolution, plus the 7.8K-word research brief, the 33-analyst pool, the formal model spec, the simulation code, and both passes of the codex review — lives at ~/Projects/labs/google-30yr/. The Apple sister-instance at ~/Projects/labs/apple-30yr/. Meta-30yr is in flight.

Organizations as Code: The Company Becomes a Repo

Sharad Jain — Sun, 03 May 2026 14:00:00 GMT

Organizations as Code: The Company Becomes a Repo

Abstract: The "Org as Code" hypothesis — a company's roles, goals, permissions, budgets, workflows, escalations, memories, evals, and governance expressed as versioned, executable configuration — has stopped being a thought experiment. Karpathy's Autoresearch shipped a 630-line proof in March 2026. Stripe is merging more than a thousand AI-generated PRs per week. The Reverse Layoff has already taught us what happens when companies skip the wrapper. This post is a tour of the runtime, the unit economics, the governance gap, and the smallest version a reader can ship on Monday.

In March 2026, Andrej Karpathy open-sourced a 630-line Python repo called Autoresearch. Three files. One human-maintained Markdown. An agent ran 50 ML experiments overnight on a single GPU with zero human intervention. When Tobi Lütke pointed it at a Shopify problem, it produced a 19% model-quality improvement after 37 sequential experiments in 8 hours.

The interesting part is not the throughput. The interesting part is the metric.

Autoresearch evaluates every experiment with a single scalar — val_bpb, validation bits per byte. Lower is better. It is vocabulary-agnostic by design. If Karpathy had used standard cross-entropy loss, a clever agent could have lowered the score by shrinking the vocabulary surface area, producing a model that scored beautifully and predicted nothing useful. val_bpb removes the loophole. Every training run is also clamped to a five-minute wall-clock budget. If compute were dynamic, a "better" model would just be a longer-trained one and the agent would discover that within a week.

==Designing the metric and the budget IS the work.== The agent does the experiments. The human designs the bounding box that makes those experiments mean something.

This is the shape of every interesting org-as-code system. Not "AI runs the company." A human-designed bounding box around a state machine that runs continuously, version-controlled in plain text, forkable, auditable, and inspectable as code.

The next thing to make reproducible is not the model and not the agent. It is the organization that coordinates them.

1. The progression is boring, which is why I trust it

The argument is mechanical:

Layer	Before	After	Where the world is now
servers	hand-configured machines	Terraform, containers, regions, reproducible deploys	done
agents	chat prompts	model + tools + permissions + memory + schedules + evals	shipping
organizations	people, meetings, folklore, docs	roles + goals + budgets + workflows + governance as runtime config	~2 years in

The third row sounds weird only because we are used to organizations being implicit.

A company is mostly coordination. It decides what should happen, who can do it, what tools they can use, how much money they can spend, when to ask for approval, where to write down the result, and how to learn from the outcome. That is not mystical. ==It is a state machine with politics.== The politics will not disappear. The state machine is going to become explicit.

The proof points are not subtle. Stripe is merging more than 1,000 AI-generated PRs per week through structured agentic pipelines. Zapier reports an 89% AI-adoption rate across its operational workflow. AI-native startups now report Revenue Per Employee figures around $3.48M, against a SaaS median of $129K–$200K. These are not stories about better autocomplete. They are stories about a different unit of production.

2. The eight-agent company

In February 2026, Karpathy ran a simulated research organization with eight agents. Four powered by Claude. Four by Codex. Each on its own dedicated GPU, each on a Git branch.

The hierarchy was deliberately corporate. One "chief scientist" agent at the top doing high-level conceptualization and delegation. Several "junior researcher" agents below it doing experimental work. The task: remove a logit softcap from nanochat without regressing performance.

The communication substrate is the part that matters. ==No virtual machines. No Docker mesh. No proprietary protocol. Just text files in a version-controlled repo.== Agents read each other's drafts the way humans read each other's commits. The chief scientist wrote specs as Markdown. Junior researchers picked specs off the queue and pushed branches. The "meetings" were diffs.

Two things came out of that experiment.

The first is that the substrate works. Agents can coordinate at organizational scale through nothing more exotic than a Git remote. If the org's coordination model lives in plain text, it composes with everything humans already know how to do — branch, merge, review, revert, blame. That is the unlock.

The second is the parable that should temper everyone's enthusiasm. One junior agent excitedly reported that it had "discovered" a way to reliably lower the validation loss: increase the model's hidden size. Mathematically true. Scientifically vacuous. The agent had stumbled onto the fact that bigger models have lower loss, called it a finding, and was prepared to merge it. The agents were mechanically tenacious. They were also remarkably bad at the part of research that requires judgment about whether a result is interesting.

This is what "the human becomes more important, not less" looks like in concrete form. Execution gets cheap. Taste does not.

3. The Constraint Stack

What Karpathy actually shipped, when you read the repo, is a three-layer Constraint Stack. Treat this as a named pattern; you will see it everywhere in agentic systems that work.

Layer	File in Autoresearch	Mutability	What it does
Sandbox	`train.py`	fully mutable	the only place the agent can edit; bounds the blast radius
Harness	`prepare.py`	strictly immutable	data prep + eval logic + tokenizer; the agent cannot rig its own grade
Brief	`program.md`	human-maintained	scope, files in/out, log format, success threshold, recovery protocol

Karpathy calls program.md a "super-lightweight skill" — the actual instruction manual the autonomous worker reads on every loop. It is not documentation wrapped around code. It is the control plane.

==The durable artifact from an overnight run is not the diff the agent committed. It is the protocol that produced the diff.== Refining program.md is where the human's judgment compounds.

This is the founder skill being reframed in real time. The old job was "recruit the right people." The new job is "design the right bounding box." val_bpb was not chosen casually. The five-minute clock was not chosen casually. The immutability of prepare.py was not chosen casually. Each is a single design decision that quietly forecloses an entire category of agent misbehavior. That is the work. The agent does the experiments overnight. The human stays up engineering the metric.

The same pattern shows up in production agentic engineering at scale. Stripe and Zapier did not get to thousand-PR weeks by writing better prompts. They got there by writing better Constraint Stacks — sandboxes, harnesses, and briefs that let stochastic systems behave deterministically inside well-defined cages.

4. Two clocks, or the org becomes unusable

The single most useful thing I learned building agent-memory systems applies almost directly to organizations. ==Separate the interactive clock from the background clock. If you do not, the system becomes unusable.==

Clock	Latency budget	What runs there	Why it exists
interactive	seconds to minutes	task execution, approvals, customer replies, deploy decisions	the path humans feel directly
background	minutes to hours	memory distillation, embedding refresh, audit review, eval runs, simulations, budget analysis	improves the org without blocking work

The mistake is to put everything on the interactive clock. A customer should not wait twenty minutes because the support agent's memory layer is re-embedding the corpus. A GTM agent should not block on a quarterly simulation before sending one approved email. A coding agent should not wait on a company-wide governance report before opening a PR.

The background clock is where the compounding happens. It is where the org notices that one workflow is burning budget, one department's memory is stale, one approval gate is pure ceremony, one agent keeps failing the same eval, one human has become the hidden bottleneck.

Freshness and richness do not share a latency budget. I argued the same thing about system-prompt architecture in The 14K Token Debt — what runs at boot vs. what runs in the loop is the most consequential decision in the stack. The org-level version is the same shape, one layer up.

5. The Wrapper, the YAML, and the runtime

The first version of org.yaml will look almost disappointingly simple.

org:
  name: acme-growth-lab
  mission: grow revenue for vertical SaaS using AI-native outbound
  budget:
    monthly_compute: 12000
    monthly_tools: 3000
    max_single_action_without_approval: 500
  models:
    primary: claude-opus-4-6-2026-02-15      # pinned. never "latest".
    fallback: claude-sonnet-4-6-2026-01-08

departments:
  research:
    goal: identify high-intent accounts
    agents: [market-mapper, competitor-watcher, hiring-signal-scanner]
  gtm:
    goal: generate qualified pipeline
    agents: [list-builder, personalization-writer, sequence-operator]
  engineering:
    goal: maintain internal systems and customer automations
    agents: [integration-builder, qa-reviewer, deployment-operator]

governance:
  humans:
    - role: board
      can_pause_any_agent: true
      approves: [payments_over_500, production_deployments, outbound_over_1000_contacts]
  audit:
    log_every_tool_call: true
    retain_for_days: 365

Today this would be a strategy doc. Tomorrow it is runtime configuration. A strategy doc describes what people hope the company does. A runtime contract constrains what the company can actually do.

The hand-wavy part of every "AI company in a box" pitch is how the YAML actually executes. The honest answer in 2026 is that the stack already exists and you can stand it up this quarter. Temporal provides durable workflow execution that survives crashes and resumes mid-step — each department becomes a workflow that can sleep for days waiting for a human signal. LangGraph holds the cyclic cognitive state per agent, checkpointed to Postgres, time-travel-debuggable. The Model Context Protocol (MCP) standardizes how agents discover and call tools — every tool is an MCP server with a schema. Open Policy Agent (OPA) enforces the governance layer as Rego rules at decision time.

A real policies/payments.rego looks like this:

package org.payments

default allow = false

# allow agent-initiated payments under the auto-approve threshold
allow {
  input.tool == "stripe.charge"
  input.amount_usd <= data.org.budget.max_single_action_without_approval
  input.agent in data.departments.gtm.agents
}

# explicit deny — the org can never wire to a country we have no compliance for
deny[msg] {
  input.tool == "wire.send"
  not input.country in data.org.compliance.allowed_jurisdictions
  msg := sprintf("wire to %v denied: jurisdiction not in allowed set", [input.country])
}

# require human approval above the threshold
require_approval {
  input.tool == "stripe.charge"
  input.amount_usd > data.org.budget.max_single_action_without_approval
}

That is one file. Read it. Notice what is happening. ==The business decision "the GTM team can charge up to $500 without asking" is now a four-line policy that runs on every tool call.== The auditor reads the same file the agent does. The board reads the same file. The regulator, eventually, reads the same file.

The full operating bundle:

Directory	What it owns	Runtime backing
`agents/`	model choices, tools, memories, permissions, schedules	LangGraph + MCP
`departments/`	goals, queues, ownership, scorecards	Temporal workflows
`policies/`	what is allowed, denied, requires approval	OPA / Rego
`playbooks/`	repeatable workflows: outbound, support, QA	Temporal sub-workflows
`evals/`	tests for whether the org still behaves correctly	scheduled background jobs
`budgets/`	token, cloud, tool, and payment limits	OPA + budget tracker
`escalation-rules/`	when to wake a human, which one, with what evidence	Temporal Signals
`simulations/`	sandbox runs before changing production behavior	E2B / Daytona MicroVMs

==Raw agents are cheap and chaotic. The value is The Wrapper — the organizational shell of policies, budgets, approvals, memory boundaries, audit trails, and evals around them.== Without that wrapper you do not have a company. You have a pile of interns with root access.

6. The five-department wallet

The first useful unit is not "create a unicorn in one click." That is the wrong fantasy. The useful unit is smaller: a deployable business capability. A support desk. A research cell. A GTM motion. A code-migration squad. A grant-writing operation. A finance ops back office.

Each one has the same shape — a mission, scoped tools, a memory boundary, an approval gate, an eval suite, an audit trail, and a budget. That bundle is the thing you fork.

The unit economics are stark enough to be worth doing on a napkin. Run a single feature cycle on a frontier reasoning model — call it a Prompt Module ($P_m$). Plan, fabricate, verify, correct, polish. ~1M input tokens (heavy context loading), ~20K output tokens, one or two retries. At Opus-class pricing of $5/$25 per million in/out tokens, a single $P_m$ runs you ~$5.68 in raw API spend. Apply a 1.5× risk multiplier for runs that loop, hallucinate dependencies, or get scrapped. Round to ~$8.50 per Prompt Module.

A $5M compute budget then buys roughly 588,000 Prompt Modules — autonomous feature cycles, each one a discrete unit of value. After a 40% quality discount (modules that get re-written or thrown away), you are still looking at ~1,116 elite-developer-years of equivalent throughput. Hiring those engineers at fully-loaded $200K would cost ~$223M. ==Capital efficiency ratio: ~44:1.==

The challenge is no longer affording the work. The challenge is orchestrating the swarm. A "deployable business capability" turns out to be a budget waterfall:

Department	% of budget	Modules	Role
Genesis (research, validation, idea-maze)	10%	~58k	synthetic personas, competitor recon, gap analysis
Fabrication (core build)	50%	~294k	Architect agent + DB / Frontend / Backend specialists
Sentinel (QA, red-team, security)	20%	~118k	adversarial agents, TDD enforcement, self-healing repair loops with budget caps
Signal (growth, content, SDR)	15%	~88k	programmatic SEO, hyper-personalized outreach
Foundry (orchestration, manager-agents)	5%	~29k	manager agents that monitor swarm health, kill zombie tasks, refactor prompts

Two non-obvious lessons in that table. Twenty cents of every dollar goes to the immune system. ==If the QA budget feels too high, you have not been bitten yet.== And the smallest line — "manager agents" — is the one that lets the other 95% behave at all. Without it, the Token Snowball will eat the rest. An agent stuck in a "fix this dependency" loop can burn 1M tokens every ten minutes.

This is also where the pricing model of compute makes a quietly important shift. As the codebase grows, $P_m$ cost drifts upward — $3 in month one, $6 in month three, $12 in month six — purely because the context loaded per call gets denser. The Foundry budget is what pays for the "Refactor Agents" that run continuously to keep the context weight manageable. Run an org without that line item and you watch your unit economics rot in slow motion.

7. What forks cleanly. What doesn't.

Forkability is the main event. When software became forkable, experimentation exploded. When organizations become forkable, business-model search changes — fork the same agency into healthcare, logistics, legal, insurance; branch a GTM motion to test two sales plays; fork into a new geography by swapping data residency, payment rails, language, local vendors.

But the breezy "fork four times" pitch hides the part that breaks.

Artifact	Forks cleanly?	Why
Policies, budgets, escalation rules	yes	pure config; no domain entanglement
Playbooks, workflows, eval suites	mostly	slot in new tools; structure stays
Agent definitions (model, tools, prompt)	mostly	re-grounding the prompt is the work
Memory and embeddings	no	domain-, customer-, and incident-specific
Vendor connectors, integrations	no	each market has different APIs and contracts
Compliance / regulatory posture	no	jurisdiction-bound; not a config swap
Customer trust and brand	no	not in the repo

The clean half is the unlock. The sticky half is what makes the founder's job real. ==Forkability is partial by design — and the partial-forkability is precisely what stops the spam version from being trivial to spin up.== If everything forked cleanly, the world would already be on fire.

8. The Reverse Layoff

Anyone selling you the clean version of this story is not paying attention.

In early 2026 the market ran the experiment for us. A wave of companies, intoxicated by the leverage, laid off significant portions of their engineering and operations teams and replaced them with agent swarms. Then the agents hit the cases nobody had documented. Financial reports broke in inexplicable ways. Production updates failed silently. Workflows degraded under load no one had simulated. The agents could not debug undocumented legacy entanglement without human intuition acting as glue.

Within months the same companies were rehiring those same engineers at premium rates — the "boomerang employee" pattern, now a category. Institutional knowledge turned out to be a largely un-computable variable. ==The lesson is not "AI doesn't work." The lesson is that the wrapper takes longer to build than the layoff press release.==

The smaller-scale version of the same lesson lives inside Karpathy's eight-agent simulation. Recall the parable from §2: an agent reported it had "discovered" that bigger hidden size lowered loss. That is what happens in microcosm when an org has agents but no scientific taste. They will produce results that pass every test you wrote and tell you nothing new about the world. The Reverse Layoff is the macro version of "discovered hidden size."

There is a quieter, longer-acting failure mode running in parallel. An MIT 2026 study identified what the field is now calling Cognitive Debt — measurable atrophy in independent analytical capacity among engineers who outsource the painful part of problem-solving to agents. EEG and fMRI work in the cohort showed reduced activation in the regions associated with deep, sustained reasoning. Output rises; the muscle that produces taste declines. The companies that survive this decade will be the ones that protect the human cognition they need to keep designing the bounding box.

A related collapse is happening in measurement itself. The DORA framework — deployment frequency, lead time, MTTR, change failure rate — was the canonical way to measure engineering health for a decade. 66% of developers no longer trust those dashboards. Deployment frequency means nothing when an agent ships dozens of commits an hour. The metrics will follow the unit of work, and the unit is shifting from "the team" to "the bundle."

9. Due diligence becomes code review

If a company is an executable bundle, acquiring one means inspecting the bundle. Not just the financials. The operating code.

You would review:

permissions — which agents can touch money, production, customer data, outbound, legal docs?
memory — what does the org believe, who wrote it, when was it last verified?
workflows — which queues drive revenue, support, finance, shipping?
evals — what tests prove the org still works?
budgets — where does compute spend go, and which loops can run away?
audit logs — can you replay why a decision happened?
human dependencies — which workflows fail if one person leaves?

That last one is the killer. A lot of companies are not really companies. They are a few heroic people holding a pile of broken processes in their heads. Organizations as code makes that visible.

It also makes the inverse visible. A small team with a clean operating repo, sharp evals, narrow permissions, clear memory boundaries, and repeatable workflows may be much more valuable than its headcount suggests.

Revenue Per Employee was the first AI-native metric. ==The Clone Test is the better one: how much of your business survives if you delete every human and rebuild from the repo alone?== If 90% does, you are running infrastructure. If 30% does, you are running a story about a few people. The Clone Test is brutal because it does not care how good your culture deck is. It asks one question — is the company in the repo, or in your head? — and the answer determines what an acquirer is actually buying.

10. Governance is the product

A clonable organization is powerful in the same way a botnet is powerful. If you can spin up a useful org, you can spin up a harmful one. Spam companies. Scam companies. Automated litigation mills. Fake-media networks. Synthetic political operations. Companies with no moral center because nobody inside them feels responsible.

This is why governance is not a chapter at the end. ==It is the product.==

The hard questions arrive immediately:

Who owns an agent's actions?
Who can audit a forked organization?
Can a regulator inspect org code?
Can customers know whether they are dealing with a human, an agent, or a company with one human and ten thousand agents?
Can payment networks, cloud providers, and model providers enforce identity at the organizational level?
Can an org be rate-limited?
Can it be recalled?

Agent identity is not enough. The org needs identity. Not just an account — provenance. Who created this org. What template it was forked from. What permissions it has. Which humans are accountable. Which jurisdictions it operates in. Which models and pinned versions it uses. What changed between this version and the last one.

Without that, organizations as code becomes organizations as malware.

The geopolitical layer is already moving. The EU's GAIA-X effort is accelerating a federated sovereign tech stack at a rumored €20–30B per year, on the explicit thesis that European org code should not be bound to American hyperscalers. The U.S. is doubling down via NSF AI Institutes and an aggressive industrial policy on agentic workforce development. There will be no neutral place to host an org repo. The infrastructure decisions are political decisions now.

11. Disposable coordination — and what you can ship Monday

The cloud gave us disposable infrastructure. AI agents give us disposable labor. ==Organizations as code give us disposable coordination.==

That is the unlock.

Once coordination becomes cheap, we will see many more tiny companies, temporary companies, single-purpose companies, forked companies, simulated companies, companies that look more like open-source projects than corporations. Some will be scams. Some will be toys. Some will be terrifying. Some will be beautiful. The direction is clear.

First we made servers programmable. Then we made workers programmable. Now we are making the organization programmable.

You do not need to wait for the platform. The smallest org-as-code artifact you can ship this week is exactly the one Karpathy shipped in March: one program.md, one immutable harness, one mutable sandbox, one agent, one scoped repo, one scalar metric. Pick a single workflow inside your team — a weekly competitive scan, a triage queue, an SDR sequence, a deploy verification loop. Write the brief. Pin the model version. Cap the budget. Add a single OPA-style rule for the one action that requires a human.

A starter program.md:

# program.md — competitor-watcher v0.1

## Mission
Surface every public competitor product/pricing/comms change in the last 7 days.

## Scope
- READ:  competitor URLs in /sources/competitors.yaml
- WRITE: append to /reports/weekly-YYYY-WW.md only
- NEVER: edit /sources, /policies, /evals; never make outbound contact

## Workflow
1. Fetch each source URL. Diff against /cache/last-snapshot/.
2. For each diff, classify: pricing | feature | positioning | hiring | other.
3. Write one bullet per change with: source URL, date observed, 1-line summary.
4. If >25 bullets in a single week, escalate (see Recovery).
5. Update /cache/last-snapshot/ on success.

## Success metric
- scalar: change_recall_score = (bullets_human_marked_useful) / (total_bullets)
- target: >= 0.6 over a 4-week trailing window.
- if score drops below 0.4, halt and request human review.

## Budget
- 200K input tokens / 20K output tokens per run, hard cap.
- Model: claude-sonnet-4-6-2026-01-08 (pinned; never "latest").

## Recovery
- On HTTP 4xx/5xx for a source: log, skip, continue.
- On parse failure: write raw HTML to /errors/, do not invent content.
- On budget overrun: halt, write /errors/budget-exceeded.md, exit 2.

Run it nightly. Read the report every morning for two weeks. Tune program.md based on what you actually wanted vs. what you got. ==That tuning is the founder skill of the next decade in miniature.== The rest of this post is what happens when the same loop is applied to engineering, GTM, finance, support, and compliance — at the same time, with budgets and policies wired together, on the same substrate.

The next five years are about learning how to spin up a company without losing the ability to ask whether it should exist.

Where this breaks

Five failure modes worth naming honestly.

[!WARNING] The empty-wrapper trap. A repo full of policies/, evals/, and audit-logs/ is not a company. Without a real workflow producing real value, the wrapper is theater. The Reverse Layoff happened to companies that built the wrapper before they had earned the right to compress the workforce inside it. Build the smallest valuable workflow first; wrap second.

[!CAUTION] Memory pollution as organizational hallucination. The most insidious failure of org-as-code is when stale, wrong, or agent-generated memories enter the trusted memory layer and become structural precedent for future decisions. By iteration N+10, the org is making decisions on its own past hallucinations. Memory needs aggressive provenance, expiry, and pruning — closer to a database garbage collector than a knowledge base.

[!NOTE] Semantic transferability cliff. A heavily optimized org repo encodes the founder's biases, blind spots, and aesthetic preferences. Forking it laterally — handing it to another founder, another vertical, another culture — guarantees friction. Distinguishing absolute organizational primitives from idiosyncratic founder preferences is the next hard problem in this space. We do not yet have a clean answer.

[!IMPORTANT] Silent model drift breaks the org overnight. Cloud providers ship undocumented model updates that change behavior on standard benchmarks — increased hallucination rates, exacerbated multi-file laziness, sometimes both. An autonomous loop running unattended cannot tell you that yesterday's claude-opus-latest is not today's. Pin specific immutable model versions at the org-config level. Treat latest as a synonym for "production may regress without warning."

[!DANGER] The Token Snowball. A single agent stuck in a "fix this dependency" loop can burn 1M tokens every ten minutes. Without per-task budget caps, circuit breakers (kill any task >5 retries), and a manager-agent watching for zombie loops, the unit economics in §6 collapse from 44:1 to negative within a week. Budget Trackers are not optional infrastructure. They are the foundation everything else sits on.

Sharad Jain builds agentic AI pipelines in Bengaluru. He previously engineered core data infrastructure at Meta and is the founder of autoscreen.ai, a production voice-AI platform. This post is part of a series on agentic AI infrastructure — see The 14K Token Debt on system-prompt architecture and The Terminal Was the First Agent Harness on Unix primitives as agent patterns.

Research & Footnotes:

Karpathy: autoresearch — 630-line autonomous ML research loop (March 2026)
The New Stack: Karpathy's 630-line script ran 50 experiments overnight
The New Stack: Vibe coding is passé. Karpathy has a new name for the future of software.
Quasa Connect: Karpathy's experiment assembling an AI research team — 'Org Engineering'
Quasa Connect: The Great AI Reverse Layoff
Quasa Connect: MIT Study Reveals 'Cognitive Debt'
byteiota: Developer Productivity Metrics Fail: 66% Don't Trust Them
NxCode: Agentic Engineering: Complete Guide (2026) — Stripe 1,000 PRs/week, Zapier 89% adoption, AI-native RPE
O-mega: Karpathy Autoresearch Complete 2026 Guide — Lütke / Shopify 19% improvement
Anthropic: Model Context Protocol specification
Open Policy Agent: OPA documentation — policy-as-code engine, Rego language
Temporal: durable execution platform — workflow indestructibility, signals, long-running coordination
LangChain: LangGraph persistence — checkpointed cyclic graphs for agent cognition
E2B: secure code-execution sandboxes — Firecracker microVMs for ephemeral agent execution
CIO: Google's Budget Tracker / BATS framework for agent cost containment
Quasa Connect: Europe's GAIA-X sovereign tech stack
arXiv: LiveMCPBench — agents in dynamic MCP tool environments — 78.95% success in tool-rich envs vs static baselines
arXiv: MetaAgent — self-evolving agents via tool meta-learning

I Got 91.55% on LongMemEval. Here's Why I'm Building a Different Benchmark.

Sharad Jain — Fri, 01 May 2026 18:00:00 GMT

91.55% on LongMemEval, and the Benchmark I'm Building Instead

Two days ago I published Brain's LongMemEval result: 408/500 = 81.60% end-to-end QA, with recall@5 = 91.55% over 470 non-abstention questions, using nothing but stdlib BM25 and a Claude Sonnet reader. That run beat the paper's lexical baseline, sat in the middle of the public field, and gave me the cleanest number I am willing to defend.

It also gave me a problem I did not expect.

The number was real. The benchmark was the wrong shape.

==The benchmark I needed was the one that would answer a different question: not "how does Brain rank against vendor leaderboards?" but "does Brain find the right document when I, the user of this corpus, ask the things I actually ask?"== Generic memory benchmarks cannot answer that. They are not built to. So you build the one that can.

This post is about that pivot. It is post #4 in a series — after The 14K Token Debt, Benchmarking Long-Term AI Agent Memory, and the LongMemEval run itself. The earlier posts argued for the architecture and proved a credible baseline. This one argues for the benchmark you should build next, after the leaderboard number is out of your system.

The Number That Didn't Matter

The clean LongMemEval result was honestly satisfying. 91.55% recall@5 is high. 81.60% end-to-end is in band with Supermemory's gpt-4o row on the same dataset. The harness is real, runnable, resumable, and committed at bench/longmemeval/ in the Brain repo.

But sit with that result for a day, and it stops being prescriptive.

The number does not tell me whether Brain is good for me. Brain is supposed to find a decision I made three months ago about a Salesforce schema, the exact pg_dump flags I worked out for one client, the rejected prompt variant from a session I cannot otherwise locate. None of those questions exist in LongMemEval. They cannot. LongMemEval was built to compare memory systems against each other on a shared, neutral, synthetic substrate. It was not built to tell you whether your specific deployment helps you.

Two more things hardened the gap once I started looking:

The dataset is chat-shaped. LongMemEval evidence sessions are LLM-simulated dialogues — Llama-3-70B self-chats with human edits, padded with ShareGPT and UltraChat fillers. My corpus is not chat-shaped. It is markdown notes, code repos, browsing trails, distilled session artifacts, raw transcripts. The benchmark I just beat does not care.
The leaderboard is contaminated. Penfield Labs audited LoCoMo in November 2025 — the other dataset every memory vendor benchmarks on — and found 6.4% of the answer key wrong. The GPT-4o-mini judge accepted 62.81% of intentionally wrong answers. Mem0 misconfigured Zep and Letta in its arXiv paper; both vendors re-ran corrected configs and scored ~10% above Mem0's own best. Zep's own 84% LoCoMo claim was a numerator/denominator error, corrected to 58.44%, then re-defended at 75.14%.

This is the field every memory layer is competing on. I just paid the price of admission. I am no longer convinced admission was the right purchase.

I call this the Owned-Corpus Test: would my version of this system, on my data, answer the questions I actually ask? If a benchmark cannot answer that, it is a useful comparison — not your benchmark. LongMemEval is a useful comparison. It is not my benchmark.

Why Generic Memory Benchmarks Won't Tell You What You Need to Know

There is a structural reason this is not just about LongMemEval.

What the public benchmarks actually test

LongMemEval has 7 question types across 5 abilities, three variants (oracle / _s ~115k tokens / _m ~500 sessions), 500 questions total, evaluated with a GPT-4o judge against verbatim reference answers (paper). The haystack is fully synthetic. Filler sessions come from public dialogue datasets. Verbatim recall is the dominant strategy: MemPalace #314 documents that raw retrieval gets ~96% recall@5 on the dataset because the answers are literally in the haystack — you store everything, you find it.

LoCoMo is the other shared playing field. Ten very-long multi-session dialogues, persona-and-event grounded, multimodal. Same shape, longer history, the same audited cracks. Vendors love it because the conversations fit in modern context windows, which means the dataset is partially answerable without memory. Vectorize's analysis of MemPalace's benchmarks shows full-context GPT-4o (~73%) outscoring Mem0 (~68%) on LoCoMo.

==Memory layers are benchmarking on tasks where memory is optional.==

The Mastra observational-memory leaderboard for LongMemEval has Mastra at 94.87%, Vectorize/Hindsight at 91.40%, Emergence AI at 86.00%, Supermemory at 85.20%, Brain at 81.60%, RetainDB at 79.00%, Zep at 71.20%. Different reader models. Different judges. Some are research configurations the authors say are not reproducible. Some are vendor self-reports. Treating any of this as a tournament is sloppy. I said as much about my own row.

The shape mismatch is the killer point

Even if the leaderboard were clean, it would be the wrong shape for personal corpora.

I went through every question type in LongMemEval and tried to map it onto my corpus. The mapping breaks for at least four of seven. Single-session-assistant assumes the assistant said the answer; in a notes corpus there is no assistant. Single-session-preference assumes a preference is stated as a chat utterance; in a real workflow, preference is implicit in repeated behavior. Multi-session reasoning over chat is a different shape than synthesis across notes, code blame, and Slack threads. The benchmark is testing a chat-shaped world. I do not live in one.

I also looked at every alternative. LoCoMo, MemBench, MemoryAgentBench, MemoryBench, MEMTRACK. All chat or dialogue-shaped. The closest thing to a personal-corpus benchmark is EnronQA — 528K QA pairs over 150 historical email inboxes — and even that is academic, narrow, and leans on a corpus nobody currently uses.

==No public benchmark evaluates retrieval over a user's own markdown / notes / code corpus with the user's own questions. The absence is the evidence.==

The Voice vs Tool Gap

Here is what makes this awkward: the field already knows.

Hamel Husain has been the loudest voice on this for two years: "Generic metrics off-the-shelf… are generic enough to be useless for diagnosing your application's failures." His workflow is corpus → facts → questions those facts answer. Jason Liu's writeup of Kelly Hong's Chroma talk is titled, almost literally, stop trusting MTEB rankings. Shreya Shankar ships EvalGen and SPADE specifically because generic LLM-judges miss application-specific criteria. Jerry Liu ships Llama Datasets so you can benchmark on your own docs in a few lines.

Even the labs say it. Anthropic's Contextual Retrieval cookbook ships a synthetic 100-sample evaluation set generated from your own docs as the canonical recipe. OpenAI's RAG-evaluation cookbook recommends generate_question_context_pairs from LlamaIndex — same pattern.

So the message is loud. What about the tool? RAGAS, LlamaIndex, TruLens, Phoenix / Arize, DeepEval, Promptfoo, Patronus, Chroma's generative-benchmarking — eight serious efforts.

RAGAS, LlamaIndex, DeepEval, and Chroma's research repo all do something useful. Each ships a slice. But every one of them is shaped for enterprise document collections — pre-curated PDFs, well-defined domains, large doc sets where Q&A generation is expensive and rare. None of them is shaped for "point me at my notes folder, give me a benchmark in 10 minutes." Nobody opinionatedly ships the personal-corpus version of the workflow.

Chroma's W&B case study — the closest prior art — improved corpus alignment from 46% → 75% by generating queries directly from the corpus. That is the right idea. It is also a research repo, not a product. The closest individual example I could find is jasonjgarcia24/grounded, where one developer built a RAG over personal Gmail with a bespoke eval, no framework. That is the field's tell. Practitioners are doing this by hand because the tooling does not yet exist.

I call this the Voice-Tool Gap: when thought leaders say it, when the labs ship cookbook recipes for it, when individual practitioners build it manually, but no opinionated tool exists for the most-asked use case — that is a gap, not a saturation.

The post does not solve the gap. Brain's harness is a sketch, not a ship. But the gap is what makes the next experiment obvious.

Build It: A Five-Technique Recipe for Owned-Corpus Benchmarks

Here is the harness shape I would build today, given a corpus that looks like Brain's: ~10k mixed docs (sessions, distilled summaries, raw articles, wiki concepts, browsing history), with both raw and synthesized layers, and a session-level usage log.

The recipe is five techniques, not one. Each compensates for the failure modes of the others.

1. RAGAS over your raw corpus

Start with RAGAS's testset generator over your raw docs. RAGAS turns a chunk into seed questions, then evolves them via Evol-Instruct-style operators (simple, reasoning, multi_context, conditioning) to break out of "the answer is the chunk."

Failure mode: lexical leakage. Generated questions can parrot the chunk verbatim, which inflates BM25 scores. Filter with an LLM-judge "would a human ask this exact question?" pass.

2. Reverse-direction grounding (synthesized → source)

This is the technique that exploits the structure of a personal corpus: you have synthesized concept pages (kb-wiki) and the source articles they were synthesized from (kb-raw). Take a wiki page about a concept, mask the entity name, and ask an LLM to write a question about the concept without naming it. Then test whether retrieval finds the underlying source articles. The metric is kb-raw recall@k.

This is closest in spirit to ORQA's latent retrieval — the "real test" of semantic search. It is the eval you cannot do on a generic corpus, and it is the eval that tells you whether your retrieval crosses lexical boundaries.

3. Hindsight queries from real session logs

Mine real query logs. For Brain, this means scraping mcp__brain__query calls from every Claude Code session where the result set was empty or the user reformulated the query immediately. Each one is a gold "hard query" — distribution-shifted by definition.

Joachims's clickthrough-as-implicit-feedback is the academic anchor; Microsoft Xbox's search relevance work and Dropbox Dash's labeling pipeline are the production analogues. For a personal corpus, you instrument the retriever and treat the doc the user actually opened next as the gold label. Cold start is real (you need usage logs to mine), but for any system that has been running a few months, this is your most realistic gold set.

4. Distractor injection

Take each (query, gold doc) pair and add hard negatives — semantically near, lexically near, but wrong. Standard recipe: BM25 top-20 minus gold, then LLM-filter false negatives (RLHN, 2025). Now you can measure precision@k and MRR, not just recall@k.

This is the single biggest fix to the LongMemEval failure mode. LongMemEval rewards "store everything" because there are no distractors that look like answers. Personal corpora are full of near-misses. If your eval does not include them, your number is fiction.

5. Calibration via small hand-graded set

Hand-grade 50 examples. Compute Cohen's kappa between your hand labels and the LLM-judge labels. Refuse to trust bulk metrics until kappa ≥ 0.6 (Langfuse's calibration guide is the cleanest writeup). 50 is the floor for a stable agreement metric; less and your kappa is noise.

You only do this once per judge prompt revision. It is cheap, and it is the only way to defend "we used an LLM judge."

Putting it together

Total: 280 questions, four metrics, one calibration. About two days of work for someone with a working corpus and a working retriever. Skip step 3 if you do not yet have query volume; everything else is bootstrappable.

The harness shape, mirrored from bench/longmemeval/: a fetch.py to mine the corpus into chunks; generate.py for RAGAS testset over kb-raw; reverse.py for kb-wiki → kb-raw masked questions; replay.py for zero-result query mining from session logs; distractors.py for BM25 top-k minus gold with an LLM filter; calibrate.py for the 50-sample hand-grade kappa gate; run.py for retrieval and reader with scrub-on-restart resume; score.py with tri-state labels and oracle ceilings; and report.py for the per-collection × per-question-type breakdown with CIs.

This is the next thing I am going to build.

What Will Go Wrong (And How the Field Thinks About It)

I built the LongMemEval port and hit five real bugs along the way: a 12K context cap that silently truncated long haystacks, a coupled-failure anti-pattern where reader exceptions threw away retrieval logs, judge rate-limit errors recorded as label=False, cross-question leakage from shared scratch collections, and eval QAs getting re-indexed into Brain's production corpus on the next sync. Each has a clean field-term framing. Listing them honestly is the part of the post I owe the reader, because every one will catch the next person to build a personal-corpus benchmark.

There are four more pitfalls the field knows about that I had not yet hit:

Judge variance and bias. GPT-4-class judges hit ~80% human agreement only after controlling position, verbosity, and self-enhancement bias (Zheng et al., MT-Bench). Position bias is non-random — running both orderings is required, not optional (Shi et al., 2024). For a credible owned-corpus eval, run both orderings and take majority of three samples; confidence-improved self-consistency cuts the sample budget by ~40%.

Retrieval ≠ reader. Always report both. The single biggest mistake in vendor reporting is publishing recall@k as if it were end-to-end accuracy. Brain's Evidence Conversion Gap — recall@5 = 91.55%, QA = 81.60%, gap = 9.95 points — is a generalizable framing. Always report retrieval and answer metrics separately, and put oracle-retrieval and oracle-reader ceilings on each so you can localize regressions (Hamel's LLM-as-judge guide covers this cleanly).

Goodhart and overfitting your eval. The moment you start tuning to your own benchmark, the metric measures tuning effort. Sohl-Dickstein on strong Goodhart is the canonical reference; the Chatbot Arena gaming case is the public demonstration. Hold out 20–30% of your questions until milestone gates. Freeze v1 of your eval set when the corpus changes meaningfully; v2 reports both splits.

Statistical power. Cameron Wolfe's stats-for-LLM-evals primer gives the rule of thumb: 25–50 questions detect a 10% gap, 100–200 detect 5%, 2500–5000 detect 1% at 80% power. LLM outputs are correlated, which means effective N is lower than nominal. If your benchmark has 50 questions and you are claiming a 3% improvement, you have not measured anything yet.

Where This Breaks

I owe the same honesty about this thesis that I owed about the LongMemEval result.

Limitation	Why it matters	What I would do before claiming more
The owned-corpus benchmark cannot be cross-system compared	LongMemEval at least lets you compare Brain to Mastra	Keep running LongMemEval as a comparison signal alongside the owned-corpus benchmark; do not abandon either
Ground-truth Q&A from your own corpus has its own biases	The generator model leaks vocabulary; reverse-grounding can anchor on hallucinated wiki claims; mined queries are survivorship-biased	Hybrid generation across multiple models; manual review of the wiki-side anchors; explicit "no result" gold for mined queries
LLM-as-judge is the floor, not the ceiling	Your judge has the same blind spots as your reader	Use a different model family for judge vs. reader; calibrate against 50 hand-graded examples; report kappa publicly
Personal corpus eval is high-variance with one rater	One person's labels are noisy compared to inter-rater agreement	Hand-grade with at least one collaborator before publishing; or compare to an independent LLM-judge of a different family
Storing your eval set securely	Personal data leaks if the eval corpus is open-sourced naively	Keep eval data in a private collection; canary tokens prevent re-ingest; never commit ground-truth to a public repo

This is a methodology argument, not a finished benchmark. The work is the next post.

What I Am Actually Saying

The argument is narrower than "stop running LongMemEval."

LongMemEval is useful for cross-vendor comparison. It exposed a real weakness in Brain's multi-session synthesis. It will continue to be a credibility signal for vendor leaderboards. ==Generic memory benchmarks are a weak prior, not zero signal.==

The argument is sharper than that:

For a deployment that exists to help one user (or one team) with one corpus, the leaderboard number is decoration. The prescriptive number — the one that tells you whether to ship — comes from a benchmark you build on your own corpus, with questions you actually ask, on the failure modes that actually bite you. The leaderboard cannot do that work. The labs already say so. The frameworks ship the pieces. The field has the language. What is missing is the opinionated tool that turns a notes folder + a retriever into a benchmark in ten minutes.

If someone builds that tool, half the memory-layer market will use it. Vendor leaderboards will become a sanity check, not a North Star. The conversation will move from "we hit 94.87% on LongMemEval" to "the user reports an 8-point lift in retrieval-and-answer correctness against their own questions, calibrated against 50 hand-graded examples, kappa 0.71."

That is a more useful conversation. It is also a harder one — because the benchmark is yours, not ours.

I am building the tool for Brain. The harness from the LongMemEval port is the spine; the techniques in section 5 are the layers. The next post in this series will be the result.

The closing question is simple: does anyone else want to build the public version of this? RAGAS, LlamaIndex, Chroma, DeepEval, Phoenix all ship the substrate. None of them ships the opinionated personal-corpus harness. Whoever does will set the next standard for memory-layer evaluation.

I will read your benchmark before I read your leaderboard.

Links Worth Following

Topic	Link	Why
LongMemEval paper	arXiv:2410.10813	Defines the benchmark and the long-term-memory task shape
LoCoMo audit	Penfield Labs / dev.to	Documents 6.4% wrong answers in the LoCoMo answer key and 62.81% wrong-answer acceptance
Mem0 vs Letta methodology dispute	Letta blog	Concrete example of vendor benchmark misconfiguration
Zep LoCoMo correction	Zep Papers Issue #5	The 84% → 58.44% → 75.14% saga; cautionary tale
Mastra observational-memory leaderboard	mastra.ai/research	The most-cited public LongMemEval scoreboard
RAGAS testset generation	RAGAS docs	The closest-to-product Q&A-from-corpus tool
LlamaIndex Llama Datasets	LlamaIndex blog	The other framework path; closest to "benchmark in a few lines"
Anthropic Contextual Retrieval	platform.claude.com cookbook	Anthropic's official "synthetic eval set from your own docs" recipe
OpenAI RAG eval cookbook	cookbook.openai.com	The OpenAI version of the same recipe
Hamel Husain on RAG evals	hamel.dev	Sharpest "generic metrics are useless" argument I have read
Jason Liu on MTEB	jxnl.co	Domain-specific eval as the only signal that survives
Chroma generative-benchmarking	research.trychroma.com	Closest existing prior art for owned-corpus generation
MT-Bench / LLM-as-judge	Zheng et al.	Judge bias controls; the position-bias and self-preference findings
Stats for LLM evals	Cameron Wolfe	The N-vs-effect-size table every benchmark builder needs

This is post #4 in a series on agentic AI memory infrastructure. Earlier posts: The 14K Token Debt, Benchmarking Long-Term AI Agent Memory, Brain on LongMemEval: 81.6%. The next post in the series will be the owned-corpus benchmark itself, when it ships.

Benchmarking Brain on LongMemEval: 81.6% and the Product Lesson

Sharad Jain — Fri, 01 May 2026 00:00:00 GMT

81.6% on LongMemEval

Brain scored 408/500 = 81.60% on LongMemEval_s — matching Supermemory's GPT-4o row, using only BM25 retrieval and a Claude Sonnet reader, on a stack that runs locally.

The result has a sharp boundary: my judge was Sonnet, not GPT-4o, so this is not a strict leaderboard replacement. But end-to-end it is a clean, full 500-question run with zero reader errors and zero judge errors. That is the number I trust.

Metric	Result
Dataset	LongMemEval_s cleaned
Questions	500/500
Retriever	`inproc-bm25`, top-k=5
Reader	`claude-cli`, `claude-sonnet-4-6`
Judge	`claude-sonnet-4-6` using the vendored official yes/no templates
Reader errors	0
Judge errors	0
QA accuracy	408/500 = 81.60%
Retrieval recall@5	91.55% over 470 non-abstention questions

Brain is a product bet, not a chat UI. It indexes the important data around my work once — Claude Code sessions, OpenClaw runs, Hermes conversations, project docs, browser trails, notes, decisions, failures, the operational exhaust that usually disappears — and lets any agent surface ask that memory naturally: what did we decide about the auth flow?, where did this error happen before?, which benchmark run was clean?, what did I already try and reject?

The bet is narrower than "stuff every session with context":

==Index the important past once, retrieve only the evidence needed now, and let the agent talk to that memory like a native part of its workspace.==

If Brain is going to sit behind Claude Code, OpenClaw, and Hermes as the memory layer, it cannot just feel useful. It has to answer held-out questions from long histories, recover the right evidence cheaply, and fail in ways I can inspect. Memory benchmarks are especially easy to overstate — a retrieval score can look like an answer score, a small sample can look like a full benchmark, a rate-limited run can leave half the dataset silently broken, a judge can be changed just enough to make the number prettier.

I benchmarked Brain because "it feels useful" is not enough to build a great product.

The rule I took from the whole exercise is the Clean Number Rule: if the run is partial, rate-limited, re-judged inconsistently, or silently missing questions, it is not a score — it is a debugging artifact.

The journey had four distinct phases:

Phase	What happened	What survived
Harness build	Built `bench/longmemeval/`, dataset fetch, ingestion, retrieval, reader, judge wrappers	A repeatable benchmark loop instead of a demo
First Sonnet signal	A 25-question burst hit 84%	Useful signal, not publishable evidence
Failed full run	Parallel Sonnet readers hit account limits and left 286/500 reader errors	A hard rule: no error-polluted scores
Clean full run	Retried failed qids serially, re-judged all 500	408/500 = 81.60%, 0 reader errors, 0 judge errors

That last row is the only one I am willing to call the benchmark result.

The story is not "we got a benchmark score." The story is that a local memory layer can become product infrastructure only after it learns to prove what it remembers.

Why LongMemEval

Most memory systems are evaluated with questions that are too clean. A fact is inserted. A question asks for the fact. Retrieval finds the fact. The demo works.

Real agent memory is messier than that. The hard cases are not just "what was the user's dog's name?" They are:

facts scattered across sessions
old information overwritten by newer information
timestamps that change the answer
assistant-side statements that matter later
preference questions where the evidence is implicit
abstention questions where the right behavior is to say there is not enough information

That is why I used LongMemEval. The benchmark was built for long-term interactive memory, not generic RAG. The LongMemEval GitHub repository describes 500 questions covering information extraction, multi-session reasoning, knowledge updates, temporal reasoning, and abstention, and links the cleaned data on Hugging Face. The underlying paper is LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory.

The small variant, LongMemEval_s, gives each question roughly 115k tokens of chat history across about 40 sessions; the medium variant pushes toward 500 sessions.

That shape matters. It separates three things people often collapse into one:

Layer	Question it answers	Why it matters
Retrieval	Did the system find the right evidence?	A memory system must surface the relevant past.
Reading	Did the model interpret the retrieved evidence correctly?	Retrieval alone does not answer the question.
Judging	Did the answer match the expected answer?	The final metric has to be end-to-end.

The LongMemEval authors also make a point that matched my own experience: even with strong long-context models, long-term memory still needs explicit machinery. The benchmark is not just asking whether a model has a long context window. It is asking whether the system can manage a growing interaction history.

That is exactly the claim Brain has to survive: not "can we stuff all history into context?", but "can a retrieval substrate make the right past available to the right agent at the right moment without paying a huge token and latency tax?"

The Product Claim

Brain started as my local-first memory layer for Claude Code and related agent work, but the product shape is broader than one assistant.

At the time of the earlier production writeup, the system was already indexing hundreds of Claude Code sessions into markdown, distilling high-signal summaries, and exposing retrieval through QMD and MCP. The goal was to make memory feel native across agent surfaces: important data flows into a normalized markdown corpus, gets indexed lexically and semantically (lexical / vector / HyDE), and exposes itself as brain ask / brain recent / MCP — so Claude Code, OpenClaw, and Hermes can ask natural questions against the user's past.

This is the key product distinction:

Bad memory product	Brain product goal
Paste giant summaries into every session	Retrieve small evidence slices when needed
Make the user manage notes manually	Index operational exhaust automatically
Optimize for "chat with your docs" demos	Optimize for agents doing real work
Hide failures behind fluent answers	Show evidence, gaps, and confidence boundaries
Spend more tokens to feel safer	Spend fewer tokens by retrieving better

The system helped me every day. It could remember decisions, commands, failures, review comments, abandoned approaches, and "we already tried that" context. But daily usefulness creates a trap. If a memory system helps you personally, you start trusting it before you have measured it.

That is dangerous because memory failures often look plausible. The agent gives an answer with confidence, but the missing evidence is invisible. For a product, that is the failure mode that matters: not forgetting loudly, but remembering wrongly while sounding useful.

So I needed an external test with enough structure to make failure legible.

I call this the Harness Before Hype rule:

Temptation	Better discipline
Publish an architecture diagram	First publish the metric it survives.
Report retrieval recall	Also report end-to-end QA.
Show the best examples	Score all 500 questions.
Optimize the prompt live	Keep the judge template fixed.
Round the number up	Preserve the clean run exactly.

That rule shaped the whole LongMemEval project, but the product reason was simple: if Brain is going to save tokens and time for real agent work, I need to know what quality I am buying with that cheap retrieval path.

The benchmark also made Brain more legible as a product. Before LongMemEval, Brain was a useful local memory layer. After LongMemEval, it had a measurable contract:

Claim	Measurement needed
Brain remembers prior work	End-to-end QA on held-out questions
Retrieval is good	Recall@k against evidence sessions
The reader is good	QA accuracy given retrieved sessions
The system is robust	500/500 complete, no reader or judge errors
The result is comparable	Fixed dataset, fixed judge templates, named models

That contract is what turns "memory" from a feature into infrastructure.

The Harness

I built bench/longmemeval/ as a normal benchmark harness, not a one-off notebook.

The pipeline had five stages: fetch the dataset, ingest per-question corpora, retrieve the top-k evidence sessions, ask a reader model for the answer, and judge that answer with LongMemEval-style yes/no templates.

The first committed baseline was deliberately boring: pure in-process BM25, no embeddings, no external vector DB, no custom memory graph.

That was the point. Before testing the full Brain stack, I wanted the cheapest credible product baseline: if BM25 plus a good reader already goes far, then the product can start fast and local instead of defaulting to expensive context stuffing.

The critical command shape looked like this:

python3 -m bench.longmemeval.run \
  --variant s \
  --retriever inproc-bm25 \
  --reader claude-cli \
  --tag fleet-0 \
  -k 5 \
  --qid-file bench/longmemeval/batches/batch_0.txt

The reader path used claude -p through the local Claude Code subscription: BM25 retrieves five sessions, Sonnet reads the question plus those retrieved sessions and emits one hypothesis, then the judge labels that hypothesis true or false.

The judge path used the same official yes/no templates vendored from LongMemEval's evaluate_qa.py, routed through Claude instead of GPT-4o.

That last sentence is important. This was a clean internal run, but not a perfect apples-to-apples public leaderboard submission. The LongMemEval repository documents GPT-4o-based evaluation. My run used Sonnet as both reader and judge because it let me run the full system without API spend.

So the precise claim is:

==Brain's BM25 + Sonnet run scored 81.60% under a Sonnet implementation of the official LongMemEval yes/no judge templates.==

It is a real end-to-end score. It is not the same thing as a GPT-4o-judged leaderboard entry.

The harness became a small contract: same 500 questions, same cleaned dataset, same retrieval k, same reader model, same judge templates, same aggregation script, no silent errors.

That contract matters more than any single prompt tweak.

The contract looked solid. Then the first serious run broke it, which is exactly why the harness mattered.

The Run That Failed

The first full attempt did not give me the final number.

It gave me lessons.

The early signal was promising: a 25-question burst hit 84%. That was useful, but it was not enough evidence to publish. A 25-question sample can be lucky, skewed by category, or easier than the full distribution.

Then I tried to scale the reader fleet in parallel. That was a mistake. Five parallel Sonnet readers tripped the weekly Claude limit and left the run with 286/500 reader errors.

At that point the harness had already taught me something more valuable than a score: benchmark infrastructure needs to treat partial failure as toxic.

I made three changes:

Problem	Fix
Reader and retrieval errors were too entangled	Split retrieval and reader try-blocks so failures were tagged correctly.
Long runs were fragile	Added resume support and qid-file batches.
Rate limits polluted outputs	Added a circuit breaker after repeated rate-limit errors.

The operational rule became:

==Serial reader, parallel judge.==

Readers are expensive, stateful, and easy to rate-limit. Judges are cheaper to resume and easier to shard. Once I separated those clocks, the full run became stable.

That was the unglamorous work that made the final number publishable.

This is also where Claude Code hooks became part of the broader Brain product story. Hooks are how everyday sessions enter the memory system automatically. The benchmark harness is the mirror image: instead of automatically capturing my work, it automatically forces the memory system to prove it can recover evidence later.

After that failure, the benchmark became simpler and stricter: finish all 500 questions, retry failures serially, judge everything cleanly, and only then look at the score.

The Run That Counted

The clean run completed all 500 questions with zero reader errors and zero judge errors.

The result surprised me in two opposite ways.

First, the simple baseline was much stronger than I expected. Plain BM25 plus Sonnet was enough to land in the same aggregate range as serious memory systems. Supermemory's research page reports an 81.6% LongMemEval_s result for its GPT-4o row, with higher rows for stronger readers. My run matched that 81.6% aggregate number, while using a much simpler retrieval stack.

Second, the breakdown made the weakness obvious. Brain was excellent at direct single-session recall and strong on knowledge updates. It was weak on preference and multi-session reasoning.

That is exactly the failure pattern I would expect from BM25.

BM25 is good at literal evidence. It is less good when the answer requires synthesizing weak signals across multiple sessions or inferring a preference from repeated behavior. Those are not just retrieval problems. They are representation and reasoning problems.

The retrieval metric tells the same story. Recall@5 was 91.55%, but QA was 81.60%. That gap is the reader/judge split in action:

Metric	Meaning	Result
recall@5	Did the evidence session appear in the top 5?	91.55%
QA accuracy	Did the final answer pass the judge?	81.60%
gap	Evidence found but not converted into a correct answer	9.95 points

This is why I do not like memory claims that only report retrieval recall. Retrieval is necessary. It is not sufficient.

I think about this as the Evidence Conversion Gap — recall@k − QA accuracy, or 91.55 − 81.60 = 9.95 points.

That gap is where the next product work lives. A memory system that retrieves the right session but cannot turn that session into the right answer has not solved agent memory. It has solved evidence surfacing.

So the aggregate said Brain was credible. The category breakdown said exactly where it was still weak. The next question was how that shape compared with a public memory system.

Where Brain Lands In The Public Field

LongMemEval does not have a single official leaderboard maintained by the benchmark authors. The public results are mostly vendor self-reports: research pages, leaderboard posts, press releases, and GitHub snippets. The fairest reading is "best public claims I could verify," not a tournament with one referee.

In decreasing order of verified accuracy, the field looks like this:

Rank	System	Best reported	Reader	Source
1	Mastra (Observational Memory)	94.87%	gpt-5-mini	mastra.ai/research
2	Vectorize / Hindsight	91.40%	gemini-3-pro-preview	Mastra leaderboard
3	Emergence AI (internal)	86.00%	not publicly reproducible	Mastra leaderboard
4	Supermemory	85.20%	gemini-3-pro-preview	Mastra leaderboard
—	Brain (this run)	81.60%	claude-sonnet-4-6	`bench/longmemeval/`
5	RetainDB	79.00%	oracle split	arXiv 2410.10813
6	Zep	71.20%	gpt-4o	Mastra leaderboard

Lighter-weight claims sit outside that list until methodology is checked side by side: Ensue AI's 93.2% LinkedIn post and Backboard's 93.4% GitHub snippet are real numbers but not yet apples-to-apples. Vectorize's 91.4% was corroborated with The Washington Post and Virginia Tech partners; most of the other rows are single-source.

Two things matter about where Brain lands.

The first is that 81.60% sits in the middle of the public field while running the simplest possible stack: in-process BM25, top-k=5, no embeddings, no graph, no rerankers. Every system above Brain on this table runs richer memory machinery, a stronger reader, or both. Sitting in that band on a BM25 baseline is a stronger product signal than the rank itself.

The second is that this is not an apples-to-apples ranking. Reader models differ across rows. Judge models differ — most public numbers were judged by GPT-4o; Brain was judged by Sonnet. Some entries are internal configurations the authors say are not reproducible. Some are leaderboard rows; others are press or social posts. Treating this as a tournament would be sloppy.

LongMemEval itself was created by Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu, and the benchmark was accepted at ICLR 2025. Among the public results that exist today, the benchmarking activity is dominated by startups and product teams rather than named individual evaluators or standalone university labs. That is worth flagging because it shapes what "the field" is: a vendor scoreboard, not a peer-reviewed ranking.

The most useful per-system comparison is still Supermemory, because their research page breaks results out by category. That is the one place Brain can be matched on shape, not just aggregate:

System row	Overall
Full-context (gpt-4o)	60.2%
Zep (gpt-4o)	71.2%
Supermemory (gpt-4o)	81.6%
Supermemory (gpt-5)	84.6%
Supermemory (gemini-3-pro)	85.2%

Brain's clean Sonnet run tied the 81.6% aggregate reported for Supermemory's GPT-4o row. It did not beat Supermemory's higher-reader rows, and because my judge was Sonnet rather than GPT-4o, I would not present this as a strict leaderboard replacement.

The category shape comparison:

Type	Brain	Supermemory gpt-4o row	Delta
single-session-user	96.88	97.14	-0.26
single-session-assistant	100.00	96.43	+3.57
single-session-preference	63.33	70.00	-6.67
knowledge-update	88.89	88.46	+0.43
temporal-reasoning	81.89	76.69	+5.20
multi-session	62.81	71.43	-8.62

This is a much more useful pair than either aggregate.

It says Brain's plain baseline is already competitive on direct recall, assistant-side recall, knowledge updates, and temporal reasoning. It also says the system is not yet good enough at cross-session synthesis or implicit preference modeling.

That is a product roadmap hiding inside a benchmark table.

I would not use this comparison or the broader leaderboard to claim "Brain beats Supermemory" or "Brain is fifth on the leaderboard." Both framings would be sloppy. The stronger claim is narrower and more useful:

==A local-first Brain baseline, using only BM25 and Sonnet, sits in the middle band of the public LongMemEval field — and exposes a clear multi-session weakness to fix next, before the more sophisticated parts of the Brain stack are even turned on.==

That narrow claim is the point of the post. It is strong enough to matter, but constrained enough to be defensible.

Why This Was Not An 85% Post

There was an 85% target.

There was an 84% small-sample burst.

There was a Tier-S prompt-bundle run designed to test whether better answer discipline could push the system toward 85% or beyond.

But I did not find a completed clean 85% Sonnet run.

The Tier-S run was a good hypothesis, not a publishable result. It reached a partial state and then hit rate-limit failure. Publishing that as "Brain got 85%" would violate the reason I built the harness in the first place.

Here is the standard I am using:

Evidence	Publishable?	Why
25-question burst at 84%	No	Too small. Useful signal only.
Full run with 286 reader errors	No	Error-polluted.
Partial Tier-S run aimed at 85%	No	Did not complete cleanly.
Full 500-question run, 0 reader errors, 0 judge errors, 408/500	Yes	Complete, reproducible, auditable.

The clean number is 81.60%.

I would rather publish the lower number than train myself to trust a flattering one.

Where This Breaks

The result is strong enough to matter, but it has sharp boundaries.

Limitation	Why it matters	What I would do before making a stronger claim
Sonnet judged Sonnet	Same-model reader/judge can share blind spots	Re-judge the 500 hypotheses with GPT-4o using the official LongMemEval path
BM25 was the headline retriever	It is a lexical baseline, not the full Brain stack	Run QMD hybrid retrieval with lex/vec/HyDE and compare per-category lift
LongMemEval_s only	The medium variant is closer to a heavier long-history workload	Repeat on LongMemEval_m after the harness is stable
No cost/latency table yet	A memory product has to be useful under real operating budgets	Add wall-clock, token, and cost estimates per question
Multi-session remained weak	The system finds evidence better than it synthesizes across sessions	Add a two-pass evidence table reader for multi-session questions
Public field is heterogeneous	Reader and judge models differ across vendor reports, so "rank" hides methodology spread	Re-judge against the official GPT-4o path before claiming a strict position; until then, frame the result as a band, not a place

This is why I am framing 81.60% as a credible baseline, not an end-state victory.

Those limits do not weaken the result. They make the next experiment obvious.

What I Learned

The first lesson is that a memory benchmark is mostly a systems benchmark.

The actual QA line is short. The hard parts are resumption, error tagging, batch boundaries, judge reproducibility, output hygiene, and preventing a partial run from masquerading as a score.

The second lesson is that retrieval recall is an upper-bound hint, not the outcome. Brain found the right evidence far more often than it answered correctly. That means the next improvements should not only be "better search." They should improve how evidence is structured for the reader.

The third lesson is that multi-session questions are the real test.

Single-session recall is table stakes. The important behavior is synthesis — session A says one thing, session B updates it, session C implies a preference, session D gives the timestamp, and the question asks for the current answer.

BM25 can surface pieces of that chain. It does not naturally build the chain.

That points to the next architecture:

Lever	Expected role
Hybrid retrieval	Improve semantic recall where BM25 misses vocabulary.
Date-aware reranking	Improve temporal questions by respecting event and session time.
Multi-session two-pass reader	Summarize evidence per session before synthesis.
Stronger independent judge	Reduce same-model reader/judge bias.
GPT-4o re-judge	Make the public number more comparable to published LongMemEval rows.

Hybrid retrieval is the obvious next retrieval lever because QMD already exposes lexical, vector, and HyDE query modes. But the category table says retrieval alone will not be enough. The product needs a better evidence representation for multi-session synthesis.

The biggest near-term improvement is probably the two-pass reader: retrieve the top 10 sessions, extract one-line facts from each, synthesize the answer from those compact facts, and judge with fixed templates. That changes the reader's job from "read five noisy sessions and answer" to "reason over a small evidence table." For multi-session tasks, that is a different problem.

What This Changes About Brain

Brain was not built for LongMemEval. It was built so an agent could remember my actual work.

That is why the 81.60% result matters to me. It says a local-first, markdown-native, BM25-first memory system can already compete with serious memory products on a hard public benchmark, even before the more sophisticated parts of the stack are turned on.

More importantly, it says the product direction is sane. Brain does not need to make every agent session enormous. It can index the user's important data once, expose a narrow natural query surface, retrieve cheap evidence slices, and let Claude Code, OpenClaw, and Hermes use that evidence only when they need it.

But the number also prevents overclaiming. Brain is not done. It is not "solved memory." It is not at 95%. It still struggles where memory becomes synthesis.

That is the right kind of result for a product: strong enough to justify the architecture, specific enough to tell me what to fix next.

The real artifact is not just the score. It is the discipline:

Discipline	Why it matters
Full 500-question runs	Avoids cherry-picking.
Separate retrieval and QA metrics	Prevents recall from masquerading as intelligence.
Fixed judge templates	Keeps improvement honest.
Error-free score only	Makes the number defensible.
Per-category breakdowns	Turns the benchmark into a roadmap.

I started Brain because I wanted Claude Code to stop forgetting my past work.

I benchmarked Brain because the product I want has a harder requirement: agents should remember without wasting my time, wasting my tokens, or pretending to know what they failed to retrieve.

The clean score is 81.60%. The next target is not a prettier blog headline. It is a faster, cheaper, more natural Brain that makes multi-session memory feel like part of the agent's normal working environment.

Links Worth Following

If you want to inspect the concepts behind this benchmark, these are the links I would start with:

Topic	Link	Why it matters
LongMemEval paper	arXiv 2410.10813	Defines the benchmark and long-term memory task shape.
LongMemEval code/data format	GitHub: xiaowu0162/LongMemEval	Shows dataset files, evaluation scripts, and question types.
Cleaned LongMemEval data	Hugging Face dataset	The cleaned files behind the run.
Supermemory comparison	Supermemory research	Per-category breakdown — the only public apples-to-apples shape comparison.
Mastra leaderboard	Observational Memory	The most-cited public LongMemEval scoreboard; aggregates Mastra, Hindsight, Emergence, Supermemory, Zep.
QMD	GitHub: tobi/qmd	Local retrieval substrate Brain builds on.
MCP	Model Context Protocol	Protocol surface for exposing tools/data to agents.
BM25	Stanford IR book: Okapi BM25	The lexical ranking baseline that got surprisingly far.
HyDE	ACL Anthology: Hypothetical Document Embeddings	Useful background for QMD's hypothetical-document retrieval mode.
Claude Code headless mode	Claude Code `-p` docs	How the Sonnet reader was driven from the harness.
Claude Code hooks	Hooks reference	How everyday Brain ingestion connects back to agent sessions.

How I Built a Local-First Second Brain for Claude Code, OpenClaw, QMD, and MCP

Sharad Jain — Wed, 29 Apr 2026 14:00:00 GMT

My local QMD index currently spans five collections and 6,751 markdown documents. On the same machine, the runtime still warns that 2,105 of them need embeddings. That is the most honest possible opening for a post about building a second brain: the problem is not whether I have data. The problem is whether the retrieval system over that data is fast, sharp, and trustworthy enough to use every day.

Here is the live shape of the system right now:

Layer	Live state
collections	`5`
indexed docs	`6,751`
docs still needing embeddings	`2,105`
default `brain ask` path	BM25 fast path
measured fast-path latency target	`~340ms p95`
health model	telemetry + hourly doctor + tests

Most "second brain" systems fail at the same layer: they treat memory as a note-taking problem.

That sounds reasonable until you have to actually use one under load. The real input stream is not curated notes. It is messy operational exhaust: Claude Code sessions, Chrome history, transcripts, raw docs, distilled summaries, wiki pages, shell commands, decisions, rejected approaches, and the half-finished reasoning that never makes it into a notebook.

The hard problem is not storage. The hard problem is turning that exhaust into a retrieval substrate that stays fast, legible, and useful when an agent or a tired human asks a question like:

What did I decide about this auth flow three weeks ago?
Where did I debug this exact failure?
What was I reading when this idea showed up?
Did I already reject this approach, and why?

==A useful second brain is not a chatbot on top of notes. It is a pipeline.== The pipeline ingests artifacts, normalizes them into stable documents, indexes them, embeds them, retrieves them with multiple search modes, exposes them through a narrow runtime surface, and then measures whether the whole thing is still working.

This is how mine is built today: QMD as the retrieval substrate, MCP as the protocol surface, brain.py as the thin runtime harness, and task surfaces that can serve both Claude Code and OpenClaw. The entire system is local-first, markdown-native, and instrumented enough to tell me when it is drifting.

If you want the shortest possible description, it is this:

messy artifacts
  -> markdown normalization
  -> partitioned corpus
  -> qmd update (lexical index)
  -> qmd embed (vector index)
  -> lex / vec / hyde retrieval
  -> brain CLI / MCP / skills
  -> telemetry / doctor / acceptance gates

I call this pattern retrieval-first memory: memory that is optimized around recall quality, boundary clarity, and operational discipline, not around the fantasy that capture alone creates intelligence.

The post hangs on four reusable ideas:

Concept	Definition
Retrieval-First Memory	optimize the system around recall quality and latency, not around capture volume
IR for Memory	normalize raw artifacts into markdown as an intermediate representation before retrieval
The Narrowest-Useful Query Rule	answer with the cheapest retrieval path that preserves enough precision
Anti-Rot Architecture	make the system continuously prove that its docs, telemetry, and runtime still match reality

1. The System Shape

The architecture only makes sense if you see the full stack at once.

Sources
  Claude Code JSONL sessions
  Chrome History SQLite
  raw docs / transcripts / imported research
  distilled artifacts / wiki pages

      |
      v

Normalization
  parsers -> markdown documents with frontmatter

      |
      v

Collections
  brain
  distilled
  kb-wiki
  kb-raw
  chrome-history

      |
      v

QMD
  qmd update -> BM25 / lexical index
  qmd embed  -> vector embeddings
  qmd query  -> hybrid retrieval + rerank
  qmd search -> fast lexical path

      |
      v

Surfaces
  MCP server surface
  brain ask / recent / inbox / explain
  Claude Code skills

      |
      v

Operations
  usage.jsonl
  doctor.sh
  launchd jobs
  acceptance checks
  eval harness

There are two clocks running through this system, and if you do not separate them, the whole thing becomes annoying fast.

Clock	Latency budget	What runs on it	Why it exists
interactive clock	sub-second to a few seconds	`brain ask`, `brain recent`, `brain explain`, Stop-hook indexing, lexical retrieval	this is the path I have to trust while I am actively working
background clock	minutes to hours	distillation, Chrome history ingest, embedding refresh, doctor checks, acceptance reporting	this is the path that improves the corpus without blocking the work

The split is deliberate. The Stop hook only does the cheap path: brain index --new --queue followed by qmd update. The heavy path lives in the cron/daemon layer: distill pending sessions, refresh browser history, then run qmd embed. That means the corpus becomes lexically searchable almost immediately, while the semantic layer catches up on the background clock.

This is the first non-obvious lesson in building a second brain: freshness and richness should not share the same latency budget.

The stack also has a clean ownership boundary at each layer:

Layer	Owns	Explicitly does not own
sources	raw facts of what happened	any opinion about what matters
normalization	stable document shape, frontmatter, naming, file boundaries	ranking, retrieval policy, user-facing judgment
collections	corpus partitioning by information type	search logic
QMD	indexing, embeddings, retrieval modes, MCP serving	application workflow and product policy
brain runtime	validation, formatting, safety rails, telemetry, capture semantics	core retrieval engine behavior
skills	when to recall, when to capture, how to compose memory into larger tasks	deterministic I/O plumbing
operations	system health, regressions, drift detection	interactive answer quality directly

That ownership table is more important than it looks. Most memory systems get mushy because every layer starts leaking into every other one:

the ingestion layer starts doing premature summarization
the retrieval layer starts making product decisions
the app layer starts hiding corpus problems behind chat polish
the operations layer is missing, so drift goes undetected

I am explicitly trying to avoid that. The shape I want is closer to a compiler pipeline than a note-taking app:

raw events
  -> normalized documents
  -> partitioned corpus
  -> indexed substrate
  -> surface-specific recall

Each stage should make the next stage easier without pretending to be it.

There is another way to see the same architecture: as a sequence of lossy and lossless transformations.

Stage	Lossless or lossy	Why it matters
JSONL session -> markdown session	mostly lossless	preserves turns, tool traces, project metadata
browser history DB -> daily markdown	selectively lossy	preserves what is useful for recall, drops browser-internal noise
session markdown -> distilled artifact	intentionally lossy	compresses toward goals, decisions, rejected approaches
corpus -> BM25 index	lossless with respect to text recall	ideal for exact-match questions
corpus -> vector embeddings	lossy semantic projection	useful for paraphrase, but never authoritative on its own

That table explains why I keep both raw and distilled layers. Distillation is not a replacement for transcripts. It is a second representation optimized for a different retrieval problem.

There are four design choices carrying most of the weight here:

Choice	Why it matters
Markdown as the canonical medium	It keeps the corpus inspectable, grep-able, and portable. The brain is not trapped in an opaque app database.
QMD as a shared retrieval substrate	One engine owns indexing, search modes, and MCP exposure rather than every surface reimplementing retrieval badly.
Thin harness, fat skills	The runtime stays deterministic and small. Task intelligence lives in markdown skill files and prompts.
Operational anti-rot	Telemetry, health checks, and acceptance gates prevent "it worked once" from being mistaken for "it is a system."

This last point matters more than people admit. A personal memory system does not die because indexing is impossible. It dies because the retrieval loop gets fuzzy, slow, stale, or annoying, and then you stop trusting it.

That leads to the first law:

==the corpus precedes the interface.==

If the underlying documents are noisy, unstable, or poorly partitioned, no chat UI will rescue the system.

2. Ingestion Is Not Capture

The input layer is heterogeneous by default. That is not a nuisance. It is the reality the architecture has to respect.

Markdown is the IR layer: the point where messy source formats become stable retrieval documents.

In my current system, the important source classes are:

Source	Native format	What it contributes
Claude Code sessions	JSONL	decisions, code discussions, tool traces, debugging history, reasoning context
Chrome history	SQLite -> daily markdown	activity context, reading trails, visited URLs, search trails
raw knowledge artifacts	markdown files	imported papers, transcripts, research notes, external source material
distilled artifacts	markdown files	higher-signal abstractions: goals, decisions, rejected approaches, concepts, tags
wiki / synthesized pages	markdown files	stable concept pages and cross-document summaries

The mistake most personal-memory systems make is to call "capture" the same thing as "ingestion." It is not.

==Capture is just getting bytes onto disk. Ingestion is turning those bytes into retrievable documents with stable shape.==

That distinction is sharp enough to formalize:

Stage	Question it answers	Typical failure if you stop there
capture	did the raw event land anywhere?	yes, but it is trapped in an app database, JSONL transcript, or browser internals
ingestion	can I deterministically parse it again later?	yes, but the output is still inconsistent and awkward to query
normalization	does it now have a stable schema, path, and document boundary?	yes, but it may still be too noisy
distillation	what should survive as compressed knowledge?	useful, but lossy and not authoritative on its own

If you collapse those stages together, you lose the ability to reason about quality. You cannot tell whether a retrieval miss came from missing capture, broken parsing, bad document design, or an overly aggressive summary layer.

That is why the normalization layer matters so much. My session indexer in brain.py parses raw JSONL transcripts and emits markdown documents with:

frontmatter: session_id, date, project path, git branch, slug
user and assistant turns
tool summaries
extracted reasoning traces
stable filenames and paths

That list sounds simple until you look at what the parser is actually doing.

For Claude Code sessions, the raw input is not a clean conversation transcript. It is a JSONL event stream with multiple record types and nested content blocks. The parser has to:

scan every line defensively because malformed JSON lines can exist
track metadata separately from content:
- sessionId
- cwd
- gitBranch
- timestamps
- session slug from system records
preserve user and assistant turns
skip low-signal or non-text blocks like tool results and images
extract only the tool inputs that matter for later recall
preserve reasoning traces separately from final answers

The code is opinionated about what gets surfaced. Tool inputs worth keeping are things like:

Read
Edit
Write
Glob
Grep
Bash
WebSearch
WebFetch

That is not an arbitrary list. It is a retrieval decision made at ingestion time. A future query like "what exact command did I run?" or "where did I grep for this symbol?" depends on those tool summaries existing as text in the normalized document.

The filename layer matters too. The session indexer uses per-agent prefixes so different sources can coexist in one corpus without stomping each other:

Agent source	Output naming strategy
Claude	bare stem for back-compat
Codex	`codex__...` prefix
Gemini	`gemini__...` prefix
Cursor	`cursor__...` prefix

That is a small detail, but it is the kind of small detail that prevents a corpus from rotting as new sources are added.

Chrome history gets transformed into one markdown file per day, with timestamps, domains, page titles, and search traces. Distillation produces another layer of markdown artifacts that compress a session into what actually matters later: goals, decisions, rejections, files touched, technologies, and concepts.

Chrome ingestion is a different normalization problem entirely. The raw source is a local SQLite database, not a transcript. The ingest script first copies the browser database to a temp path so Chrome's file lock does not block reads. Then it joins together multiple tables:

visits
URLs
context annotations
content annotations
keyword search terms
cluster labels

That joined view is then grouped into one markdown file per day.

That "one file per day" choice is not just aesthetic. It is the document-boundary answer for browser memory. Sessions want one file per session. Browsing history wants one file per day. Those are different units of recall.

The Chrome pipeline is also aggressively selective. It applies:

allowlists for productive domains
suffix-based domain matching for subdomains
NSFW filtering on URLs, titles, and searches
de-noising for spammy or injected search terms
omission of Chrome-internal URLs and local file URLs

That is normalization as policy. If you do not make those cuts early, your corpus inherits the browser's worst qualities: ad noise, accidental clicks, internal URLs, and junk search fragments.

This is why I think of normalization as a schema design problem, not a file conversion problem.

You are deciding, for each source:

Decision	Example in this system
document boundary	one session per file, one browser day per file
stable identity	session stem, agent prefix, date path
metadata contract	frontmatter fields that will exist everywhere for that source
signal filter	which tool calls, URLs, titles, searches, and blocks are worth preserving
path semantics	where in the corpus this source will live so later retrieval can reason about it

Once you see ingestion that way, a lot of second-brain systems start looking suspiciously under-specified. They say "we ingest everything," but they do not define:

what a document is
what the stable key is
what gets dropped
what fields are guaranteed
how two source types differ structurally

Without that, retrieval quality becomes accidental.

I think of markdown here as IR for memory: an intermediate representation between raw event logs and retrieval. Not because markdown is glamorous. Because it is inspectable, versionable, and composable.

And like any good IR, it should satisfy a few properties:

Property	Why it matters
human-readable	I can inspect bad outputs directly
append-friendly	new artifacts can land without schema migrations
stable enough for indexing	BM25 and embedding layers need predictable text structure
rich enough for provenance	source, date, project, and session identity must survive
cheap to diff	regressions in parsers or distillers need to be visible in git or plain text

That one design choice buys a lot:

QMD indexes it natively.
agents can quote or retrieve from it directly.
I can grep it when retrieval fails.
I can diff it when distillation goes bad.
I can move collections around without migrating a proprietary store.

There is also a deeper benefit: markdown keeps the memory substrate debuggable by the same tools I already trust for code. rg, sed, awk, git diff, filesystem walks, and plain editors all still work. That sounds almost trivial until you compare it to agent-memory systems that immediately disappear behind a vector DB, a hosted API, or an opaque "memory sync" abstraction.

This is also why I do not treat the second brain as "an app." The durable asset is the corpus, not the UI.

3. Collections, Indexing, and Embeddings

Once the corpus is normalized, the next question is how to split it so retrieval does not collapse into an undifferentiated soup.

A corpus becomes useful when artifact classes compete by retrieval role instead of collapsing into one giant pool.

My live QMD config currently registers five collections:

Collection	Path role	Retrieval role
`brain`	raw indexed sessions	high-recall verbatim memory
`distilled`	dense LLM-generated artifacts	semantic compression of prior work
`kb-wiki`	synthesized wiki pages	stable high-level concepts
`kb-raw`	raw articles and transcripts	source-level grounding
`chrome-history`	browsing logs	behavioral and temporal context

At the moment of writing, the live file counts look like this:

Collection	Live file count	What that count implies
`brain`	`4,470`	the largest, noisiest, and most lossless layer dominates raw recall
`distilled`	`778`	much smaller, denser, and more semantic
`kb-wiki`	`216`	slow-moving synthesized knowledge
`kb-raw`	`1,846`	long-tail source grounding
`chrome-history`	`86`	low-count but high-temporal-value context

Those counts are not just scale metrics. They are retrieval-shape metrics. A corpus dominated by raw transcripts behaves differently from one dominated by polished notes, even if both use the same engine.

That split is not cosmetic. It is what allows retrieval to preserve the difference between:

exact prior transcript recall
compressed lessons
source documents
browsing exhaust
stable knowledge pages

Collection boundaries answer three questions at once:

Question	Why it matters
what kind of artifact is this?	transcript, distillation, wiki page, source text, or behavioral exhaust
how should this artifact compete?	a raw session should not rank the same way as a distilled decision memo
what kind of recall is this layer good at?	exact-match, semantic, provenance-heavy, or recency-oriented

Without that partitioning, retrieval becomes a false democracy where every file fights in the same pool even though the documents were created for different jobs.

QMD is the retrieval substrate sitting under all of this. The repo describes it as an on-device search engine for markdown knowledge bases that combines BM25 full-text search, vector semantic search, and local reranking QMD. In practice, my pipeline uses it in two phases:

1. qmd update
   -> refreshes the lexical / BM25 index

2. qmd embed
   -> refreshes vector embeddings

Those two commands are easy to say and easy to blur, but they are not the same freshness guarantee.

Command	What it refreshes	Operational meaning
`qmd update`	lexical / BM25 visibility of changed files	the corpus is text-searchable again
`qmd embed`	semantic vector representation	embedding-based retrieval can now see the new or changed material

That means "the index is fresh" is actually two claims:

lexical freshness: newly normalized text can be retrieved at all
semantic freshness: embedding-based retrieval paths know about that text too

In my stack, lexical freshness has the stricter SLO. That is why the cheap path runs in the Stop hook and the richer path runs on the background clock.

The automation loop reflects that split directly:

brain index --new --queue
brain distill --from-pending
chrome_history_ingest.py
qmd update
qmd embed

The timings in the surrounding scripts and docs make the separation concrete:

Step	Budget class in this system
Stop-hook `index --new --queue + qmd update`	`< 2s` target so it stays invisible in active work
async `qmd update`	around `~5s`
async `qmd embed`	around `~17s` per batch and model-dependent

This is why I keep insisting the second brain is a pipeline. Pipelines have critical paths. Some stages can lag; some cannot.

That sequencing matters. You do not embed raw chaos directly. You first normalize and organize the corpus, then reindex, then refresh embeddings.

It is also worth stating the less fashionable truth: embeddings are not the system. They are one retrieval mode inside the system.

Layer	What it does	Why it exists
lexical index	exact term / path / command recall	unbeatable for commands, filenames, literal phrases
vector index	semantic proximity	useful for paraphrase and conceptual search
rerank stage	candidate ordering	helps separate "technically related" from "actually relevant"

Those layers fail differently, which is exactly why I do not want to collapse them into one magical "search" box.

Failure shape	Typical cause	Better fix
exact phrase exists but is not surfaced	lexical ranking or collection scope is weak	BM25 tuning or narrower corpus partition
conceptually related but wrong answer ranks high	semantic neighborhood is too broad	reranking or a more constrained query path
semantically useful hit is missing	embeddings are stale or the semantic layer is too thin	`qmd embed`, HyDE, or stronger distillation
answer exists but is buried in huge transcripts	raw layer is too lossless for the query	lean on distilled artifacts

There is a reason the runtime still defaults brain ask to the BM25 fast path rather than always using a hybrid query. In the live implementation, the lexical path is dramatically cheaper and faster. The code explicitly describes the wedge as a BM25-first path with p95 around ~340ms, while the fuller hybrid query and rerank path is slower and reserved for later selection logic.

That is not a compromise. It is a design judgment that also lines up with Anthropic's public guidance on agents: prefer simple, composable patterns first, and only add complexity when measurement says the simple path is insufficient Anthropic.

The important nuance is this: the default is not "BM25 because vectors are bad." The default is "BM25 because defaults are about reliability under real latency budgets."

There is also a corpus-design reason the lexical path works better here than people might expect. The normalized documents are already shaped around:

sessions
decisions
tool traces
dates
projects
concepts

BM25 is not operating over random sludge. It is operating over documents deliberately engineered to make lexical recall useful.

That becomes the second law:

==retrieval quality is constrained by latency budgets as much as by embeddings.==

If a memory system is semantically elegant but too slow for habitual use, it has failed.

4. Retrieval, Search, and Reranking

This is the layer people hand-wave most often. "We use hybrid search" is not an architecture. It is a slogan.

The narrowest-useful-query rule in diagram form: route the question first, then rank inside the cheapest path that preserves the answer.

The retrieval stack here is more usefully understood as a ladder:

Mode	Best query shape	Failure mode
`lex`	exact terms, commands, file paths, literal errors	misses conceptual paraphrases
`vec`	semantic recall, paraphrases, concept-level search	may retrieve vaguely related but wrong material
`hyde`	"a session where we..." style memory prompts	can be powerful, but easier to overfire or drift
rerank	sort promising candidates	helps precision, but costs latency

What matters in practice is not just which modes exist. It is how the runtime chooses among them, constrains them, and recovers when they fail.

QMD exposes all three retrieval modes, and the official docs position MCP as the standard way for AI applications to connect to external systems like files, tools, and workflows MCP. The practical consequence is that the same markdown corpus can be queried either:

locally via qmd commands
through the MCP server surface
or via a thin runtime wrapper like brain ask

That layering is the whole point.

The fastest path in my stack today is not "semantic everything." It is:

query
  -> validate and sanitize
  -> qmd search --json
  -> take top lexical candidates
  -> attach freshness metadata
  -> format for terminal or agent surface

That simplified path hides a real sequence of policy decisions in the runtime:

Step	What the runtime is actually doing
query intake	join free-text args into one query string
validation	reject dangerous shell-shaped input like raw `;`, backticks, or `$(`
bounded search	call `qmd search --json` with a candidate limit larger than final top-k
candidate shaping	parse JSON hits, resolve paths, attach age/freshness metadata
filtering	optionally cut by age via `--since-days`
surface formatting	terminal-friendly prose or machine-readable JSON envelope
telemetry	log query length, latency, surfaced paths, hit count, and surface

That is why I treat retrieval policy as a first-class surface. The engine may know how to search, but the runtime decides what "a safe, useful answer" looks like.

That is enough to answer a surprising fraction of memory questions, especially when the corpus is already structured around sessions, decisions, and tool traces.

Why not just use semantic search for everything?

Because semantic search is not free and not always the right primitive.

Question	Best retrieval mode	Why
"where did I run `qmd embed -f`?"	lexical	exact command recall
"what was the session where I chose the CLI wedge?"	lexical + distilled	decision phrases are often explicit
"find the session where I was debugging the memory system but did not use the phrase memory system"	vector or HyDE	conceptual query, paraphrase-heavy
"what was I doing yesterday?"	recency walk over collections	a chronological query, not a semantic one

That last row matters. Not every memory query is "search" in the same sense. Some are temporal, some are behavioral, some are provenance checks.

This is why I think of the query layer as a routing problem before it becomes a ranking problem.

If the question is:

literal -> prefer lexical
conceptual -> consider semantic / HyDE
chronological -> walk the corpus by mtime and collection
provenance-heavy -> preserve path, source, and freshness above fluency

Too many memory products skip that routing layer and jump straight to "semantic search everywhere." That is usually just an expensive way to destroy the distinction between query types.

This is why I use the term retrieval discipline: choosing the narrowest search surface that answers the question without incurring unnecessary latency, token cost, or fuzziness.

Put differently: The Narrowest-Useful Query Rule says the best retrieval path is the cheapest one that still preserves the answer. grep beats embeddings when the question is literal. BM25 beats hybrid search when the corpus is already well-shaped and the query is explicit. A chronological walk beats both when the question is temporal.

The runtime also bakes in a provenance policy. brain ask does not just print a title and a snippet. It attaches:

the qmd://... path
score
age label like today, yesterday, or N days ago
a freshness warning when the memory is old enough to be risky

That is retrieval policy doing product work. A raw ranking score is not enough when the underlying artifact may describe code that has already changed.

There is a second layer of discipline here: error handling is part of retrieval quality, not a separate concern.

Failure mode	Runtime behavior
bad query	reject with structured exit code and a repair hint
`qmd` missing	explicit install / reindex recovery path
timeout	emit `qmd_timeout`, log telemetry, preserve the failure as data
invalid JSON from the engine	fail loudly instead of pretending results are empty
no results	return structured empty response rather than fabricating a summary

That matters because a second brain can fail in ways that look deceptively intelligent. A silent timeout or malformed engine response is worse than a hard failure if the caller interprets the absence of evidence as "nothing exists."

The benchmark harness around this system also reinforces the point that retrieval is not one monolithic number. Even the local evaluation setup distinguishes:

inproc-bm25
qmd-lex
qmd-hybrid

That is the right shape. If you cannot decompose retrieval into separate modes and measure them independently, you do not really know what your memory layer is good at.

In other words:

==retrieval quality is the product of routing, ranking, provenance, and failure semantics together.==

5. QMD vs Brain MCP vs Brain CLI

These names are easy to blur, so the boundary needs to be explicit.

Layer	Responsibility	What it is not
QMD	indexing, search, collections, embeddings, hybrid retrieval, MCP serving	not my product logic
brain MCP	the memory corpus exposed through QMD's MCP surface	not a separate magical reasoning engine
`brain.py` / brain CLI	query shaping, safety rails, formatting, telemetry, health introspection, inbox capture	not the core search engine
skills	task-specific workflows and agent intent	not the deterministic runtime

That table is the minimum version. The fuller version is about surface contracts.

Surface	Input shape	Output shape	Primary consumer
`qmd search`	plain query string	BM25 JSON or text hits	fast local runtime paths
`qmd query`	expanded or structured `lex/vec/hyde` document	hybrid reranked results	richer retrieval workflows
`qmd mcp`	MCP stdio protocol	tools/resources exposed to an MCP client	Claude Code or another MCP client
`brain ask`	CLI args / env vars	terminal prose or structured JSON envelope	me, scripts, Claude Code skills
`brain explain`	no query, just runtime invocation	live system state	debugging, drift detection, operator trust
skill invocation	natural-language task intent	delegated call into `brain.py` or QMD-backed behavior	agent workflow layer

Once you look at the contracts, the confusion gets easier to resolve. QMD and brain.py are not competing interfaces. They are adjacent layers in the same stack.

The cleanest way to say it is:

QMD owns retrieval.
brain.py owns runtime behavior.
skills own task-level judgment.

I would make the same distinction one level more concretely:

If you need to...	The owning layer is...
add a new corpus folder	QMD config / collection layer
change how recall is formatted for a human or agent	`brain.py`
change when memory should be consulted inside a workflow	skill layer
change how embeddings or hybrid retrieval work	QMD, not the CLI wrapper
explain why the system is broken right now	`brain explain` and the ops layer

This is also why "brain MCP server" is easy to misunderstand. In my actual local setup, the MCP surface is effectively QMD pointed at the brain-owned collections. The runtime layer around it is where I add:

query validation
output envelopes
staleness warnings
KAIROS-style inbox capture
usage logging
doctoring and explainability

The command line reflects that separation clearly. qmd itself exposes:

qmd search
qmd vsearch
qmd query
qmd get
qmd multi-get
qmd mcp

That is the substrate surface.

brain.py then exposes a different shape entirely:

index
distill
queue
ask
recent
inbox
explain

Those are not alternative spellings for the same thing. They are wrapper surfaces around different responsibilities:

`brain.py` command	Type of responsibility
`index`, `distill`, `queue`	corpus production / maintenance
`ask`, `recent`	retrieval access with runtime policy
`inbox`	typed capture into a memory-friendly path scheme
`explain`	self-description and operator diagnostics

One concrete example: brain ask adds a memory-age label like today, yesterday, or N days ago, and only emits a freshness warning when a memory is older than a day. That is not indexing. It is runtime policy. It is the kind of detail a memory tool needs if it is going to be trusted by either a person or an agent.

Another: brain explain is not retrieval at all. It is the self-diagnosis surface. It reports live state: QMD presence, collection visibility, launchd job status, inbox state, installed skills, telemetry tail. That is how you stop docs from becoming lies.

That matters because a second brain has two very different kinds of truth:

Truth type	Example
corpus truth	what documents exist and what they contain
runtime truth	what is currently installed, indexed, loaded, routed, healthy, and stale

QMD mostly owns corpus truth. brain explain is there to expose runtime truth.

I think of this as surface separation:

the substrate should search
the harness should normalize runtime behavior
the skill should decide when the memory surface is worth invoking

The last piece is the skill layer, because this is where many agent-memory systems become conceptually sloppy. A skill is not "more retrieval." A skill is activation logic plus task framing.

In my current setup, the installed brain skills carry path-scoped activation rules like:

brain-ask only activates in ~/Projects/**
brain-recent only activates in ~/Projects/NOW/**
brain-inbox is unconditional

That means the skill layer is doing something the search engine should never do: deciding when the memory surface belongs in the conversation at all.

So the boundary line I care about most in this section is:

==QMD decides how to search. The runtime decides how to expose. The skill decides when to bother.==

6. Thin Harness, Fat Skills

Garry Tan's "thin harness, fat skills" idea lands because it matches what high-functioning agent systems actually need: a small deterministic runtime and a rich task layer expressed in the medium the model already reads well Garry Tan.

Push execution down into tooling and judgment up into skills; that is what keeps the runtime narrow enough to trust.

The most important sentence in that whole framing is not "skills are powerful." It is the more uncomfortable one: the bottleneck is usually not model intelligence, it is schema understanding.

If the model cannot find the right context, load the right procedure, or distinguish deterministic work from judgment work, a bigger model mostly just fails more fluently.

The harness in my system stays deliberately narrow:

parse arguments
validate inputs
shell out to QMD safely
format results
write telemetry
expose debug state

The skills stay fat:

when to invoke retrieval
what retrieval mode is implied by user intent
how to combine memory with a broader task
what not to save
how to route capture versus recall

That thin/fat distinction is easy to repeat and easy to misuse, so I try to define it operationally:

Layer trait	"Thin" means...	"Fat" means...
logic density	small amount of deterministic branching	rich procedural and judgment-heavy instructions
change frequency	should change rarely and carefully	can evolve quickly with workflow learning
failure cost	failures are systemic and should be obvious	failures are task-local and easier to iterate on
best representation	code, exit codes, I/O contracts, file paths	markdown procedures, descriptions, heuristics, routing language
consumer	shell, scripts, launchd, other tools, agent wrappers	the language model itself

That table is why markdown skills are not an afterthought here. They are the place where I want to put:

process
judgment
activation hints
scope
exceptions
task-specific language

And it is why I do not want to put those things into the harness unless I absolutely have to.

That split matters for at least three reasons.

Reason	Thin harness benefit	Fat skill benefit
maintenance	less code drift in the runtime	workflow logic evolves without recompiling the system
agent ergonomics	predictable commands and exit codes	rich behavioral guidance close to the task
context hygiene	fewer abstractions in code	more judgment in markdown, where the model can actually use it

There is also a fourth reason: debuggability asymmetry.

If a thin harness fails, I want it to fail in a way that looks like software:

bad exit code
timeout
malformed JSON
missing binary
lock contention

If a fat skill fails, I want it to fail in a way that looks like judgment:

wrong invocation timing
over-retrieval
under-retrieval
bad decomposition of the task
poor phrasing of what to capture or recall

Those two failure classes should not be mixed. If the harness is bloated with task judgment, then every product mistake starts masquerading as an infrastructure bug.

It also lets the system support multiple surfaces without forking the architecture. A terminal call, an MCP query, and a Claude Code skill can all hit the same retrieval substrate while preserving different surface behaviors.

In the live stack, you can see that separation directly:

Installed skill	Scope rule	Why it belongs in the skill layer
`brain-ask`	`~/Projects/**`	project-scoped recall is an activation decision, not a search-engine concern
`brain-recent`	`~/Projects/NOW/**`	"recent" is relevant when the project context itself is active
`brain-inbox`	unconditional	capture should remain globally available

Those path filters are exactly the sort of thing people are tempted to push downward into the runtime. I think that is a mistake. A path-scoped activation rule is not a retrieval primitive. It is workflow policy.

That is the key distinction between a usable agent memory system and a pile of plugins:

==the harness should be boring; the skills should be opinionated.==

I would rather add a new skill than add a new mini-platform inside the runtime. The moment the harness starts swallowing retrieval strategy, agent policy, user workflow logic, and product opinions, it becomes the wrong kind of thick.

There is a simple decision rule I use for where new behavior belongs:

If the new behavior is mostly...	Put it in...
deterministic lookup, validation, or formatting	the harness
natural-language routing, judgment, or task decomposition	a skill
index structure, search mode, or retrieval mechanics	QMD / substrate layer

Examples make this clearer:

Behavior	Right layer	Why
reject a query containing shell-injection markers	harness	deterministic safety check
decide that "what was I doing today?" should invoke a recent-activity workflow	skill	intent routing
add `today / yesterday / N days ago` freshness labels	harness	surface policy with deterministic rules
decide whether this note is worth saving or is just derivable noise	skill or capture-policy layer	judgment-heavy
hybrid `lex/vec/hyde` retrieval behavior	QMD	engine capability

This section also answers a more strategic question: why not just make the harness smarter and keep fewer skills?

Because thick harnesses age badly.

They accumulate:

duplicated workflow logic
hard-to-reason branching
more hidden behavior per command
more context assumptions inside code
more places where agent and operator expectations diverge

Skills, by contrast, let the system expose its own procedure in the same medium the model reasons over. A good skill is part codebook, part resolver, part operating manual.

That is the doctrine in one line:

==push intelligence up into skills, push execution down into deterministic tooling, and keep the harness narrow enough that you can still trust it.==

7. Audit Your Own Stack

If you want to build this without copying my exact tooling, do not start by designing a beautiful assistant. Start by auditing the memory path you already have.

State inspection comes before anecdotal queries; otherwise one good answer can hide a broken substrate.

Run five checks, in this order:

Check	What you are looking for	Failure meaning
artifact audit	what raw sources already exist: sessions, browser history, docs, transcripts, notes	you may not have a memory problem yet; you may have a capture problem
normalization audit	which of those sources already become stable text or markdown documents	your retrieval substrate does not exist as inspectable documents
freshness audit	how quickly changed artifacts become visible lexically and semantically	your corpus exists, but the runtime is reading a lagging copy of reality
retrieval audit	which questions require exact recall, semantic recall, recency, or provenance	you are overloading one retrieval mode to solve incompatible query classes
surface audit	what is the thinnest interface you will actually use every day	the system may be technically sound but behaviorally dead

That sequence matters. If you start with the assistant layer, you hide failures from the layers below it. An LLM surface can make a broken substrate look functional for a surprisingly long time.

On my machine, the fastest useful audit looks like this:

qmd collection list
python3 ~/.brain/brain.py explain
python3 ~/.brain/brain.py recent --since=24h --json
tail -20 ~/.brain/logs/usage.jsonl

I intentionally left brain ask out of that first pass here. An audit should start with state inspection, not with an anecdotal query. A single good query can hide a stale index, a polluted recency layer, or dead telemetry. State comes first. Queries come after.

7.1 Corpus Inventory

qmd collection list answers the first operator question: what corpus shape am I actually searching?

On my machine right now, it returns:

Collection	Files	Updated	What I infer
`brain`	`4,470`	`31m ago`	the raw session layer dominates recall and noise budget
`distilled`	`778`	`6d ago`	semantic compression exists, but it is stale relative to the raw layer
`kb-wiki`	`216`	`1d ago`	stable concept pages are updating slowly, which is fine
`kb-raw`	`1,846`	`6h ago`	source-level grounding is alive and changing
`chrome-history`	`86`	`6d ago`	temporal browsing context is behind and should not be trusted as fresh

That one command is already more diagnostic than many dashboards because it exposes three properties simultaneously:

corpus balance: which layer dominates the candidate pool
freshness skew: which layers are drifting behind others
document-boundary sanity: whether collection counts move the way the source type should move

The red flags are specific:

if brain is exploding while distilled stays flat forever, the compression layer is not keeping up
if chrome-history has not moved in days, any answer framed as "recently watched" or "recently searched" is suspect
if a supposedly stable layer swings wildly in count, document boundaries may be wrong

The point of the collection audit is not "wow, lots of files." The point is to understand what kind of competition your retrieval engine is about to run.

7.2 Runtime Explainability

python3 ~/.brain/brain.py explain answers the second operator question: can the runtime explain its own installation state without me reading code?

The current output on my machine includes:

brain.py version 0.1.0
Python path and version
BRAIN_HOME, BRAIN_INBOX_DIR, and BRAIN_SURFACE
QMD binary resolution
registered collection names
today's inbox path
telemetry path, mode, and recent events
launchd job status
installed Claude Code skills and their scope rules
last doctor tail

That is not cosmetic introspection. It is a contract test for the runtime's own assumptions.

I use explain to answer five concrete questions:

Question	Example from the current output
am I running the right binary?	`/Users/sharad/.brain/brain.py` under Python `3.14.3`
am I pointing at the right home and inbox?	`BRAIN_HOME=/Users/sharad/.brain`, inbox under `/Users/sharad/.brain/inbox/...`
is QMD reachable from this environment?	`/opt/homebrew/bin/qmd` resolves
are the background jobs alive?	`com.brain.doctor` and `com.brain.week1` are loaded with `last_exit=0`
are the surfaces actually installed?	`brain-ask`, `brain-recent`, and `brain-inbox` show up with their scope rules

The subtle but important part is skill scope. If brain-ask is only active under ~/Projects/** and brain-recent is only active under ~/Projects/NOW/**, then "the assistant did not use memory" might not be a retrieval failure at all. It might be a surface-activation failure.

This is why I treat explainability as a production feature. A system that cannot report its own environment, surfaces, and health boundaries forces every failure into source-code debugging.

7.3 Freshness and Shape Audit

python3 ~/.brain/brain.py recent --since=24h --json answers a harder question than "is the system alive?":

What kinds of artifacts became visible recently, from which collections, and how contaminated is the recency surface?

The current output shows:

total: 22
walk_ms: 54
heavy presence of brain items from benchmark and answer-eval sessions
recent kb-raw additions such as:
- claude-code-leak-deep-lifts-2026-04-28
- gbrain-evals-frameworks-2026-04-28

That is exactly why I prefer a JSON audit path here instead of a pretty human summary. I want to inspect recent shape, not just admire that something came back.

There are three things I look for in recent output:

Signal	Healthy interpretation	Failure interpretation
walk time	low double-digit or low triple-digit milliseconds for a 24h scan	recency is too expensive to use interactively
collection mix	recent artifacts appear from the layers I expect to be moving	one ingestion path is dead or one layer is starving all others
artifact type quality	recent items look like meaningful sessions, docs, or notes	the surface is polluted by synthetic eval debris, spam, or malformed outputs

This is where many memory systems quietly degrade. The recency surface becomes dominated by whatever pipeline writes the most files, not by what the operator most needs to see. In my case, benchmark-style synthetic sessions can easily crowd out higher-value human work if I do not watch the shape of the recent layer.

That is why recent is not just a convenience command. It is a surface-quality audit.

7.4 Telemetry Audit

tail -20 ~/.brain/logs/usage.jsonl answers the final operator question: what did the system actually do, and how did it fail under real use?

The recent telemetry on my machine shows:

repeated doctor_run events
one ask with query_len: 35, latency_ms: 363, n_hits: 0
one recent run with walk_ms: 38, n_total: 24
two inbox_write events
one doctor_run failure with map_drift
one doctor_run failure with unittest_failed

That small tail already tells me five useful things:

Observation	What it means
`ask` returned `n_hits: 0` in `363ms`	low latency alone does not prove useful recall
`recent` completed quickly	the recency walk is currently interactive enough
`inbox_write` events exist	the capture path is not dead
`map_drift` occurred once	docs and code briefly disagreed and the doctor caught it
`unittest_failed` occurred once	health checks can surface transient or flaky runtime issues before they become folklore

Telemetry is where the difference between "it demos" and "it operates" becomes obvious. If your usage log only records success, it is not telemetry. It is vanity analytics. The minimum viable memory log should tell you:

which surface was used
which operation ran
how long it took
whether it returned anything
whether health checks failed
whether the system is being used at all

7.5 Query Audit Comes Last

Only after those four state checks do I run an actual retrieval query such as:

python3 ~/.brain/brain.py ask "qmd embed"

Running a query too early confuses diagnosis. A good hit can coexist with:

stale embeddings in another collection
broken daily-log capture
dead launchd jobs
recent-surface pollution
silent drift between docs and code

The query audit is where I test user-visible usefulness. It is not where I establish system health.

So the real audit order is:

inspect corpus shape
inspect runtime self-description
inspect recent-layer freshness and contamination
inspect telemetry and failures
only then inspect query usefulness

That ordering sounds conservative because it is. Most second-brain projects fail by building the assistant before they have built the corpus.

If you are building from scratch, the safe order is:

pick one raw source that already exists
normalize it to markdown with stable frontmatter
index it lexically first
only then add embeddings
only then add a CLI or agent surface

If I had to compress the audit logic into one rule, it would be:

==do not trust answers from a memory system until you can explain its corpus shape, freshness, recent surface, and failure log.==

And if I had to compress the entire post into one build instruction, it would still be:

==build the memory substrate first, then earn the right to add the assistant.==

8. The Operational Layer

This is the part almost nobody includes in their "how I built my memory system" write-up, and it is the part most likely to decide whether the project survives.

The operational layer is what stops a working demo from silently decaying into folklore.

My operational layer has five pieces:

Piece	Role	What it protects against
`usage.jsonl`	append-only telemetry for invocations, surfaces, latency, and outcomes	false confidence from anecdotal success
`doctor.sh`	hourly health checks for QMD, CLI health, JSONL integrity, tests, map drift	silent substrate or runtime decay
launchd jobs	keep checks and acceptance gates firing even when memory is not top of mind	"I forgot to look, so the system drifted for a week"
test harness	verify input validation, timeout behavior, inbox sanitization, concurrency integrity	regression hiding behind plausible output
acceptance gate	decide whether the wedge deserves to live past Week 1	hobby-project inertia and self-deception

That last row matters. Monitoring is not enough. A system that only collects health data but never uses it to make continuation or teardown decisions still rots politically even if it is healthy technically.

8.1 Telemetry Is the Ground Truth of Use

usage.jsonl is the system's behavioral ledger. Every meaningful user-facing or doctor-facing action appends a structured row under a file lock.

The schema is intentionally boring:

schema
ts
surface
event
event-specific fields such as:
- latency_ms
- query_len
- n_hits
- walk_ms
- n_total
- failures
- entry_id
- path

The boringness is a feature. This file needs to survive shell tools, ad hoc parsing, and future schema evolution.

Three implementation choices matter more than they look:

Choice	Why it exists
append-only JSONL	partial corruption is local to a line, not a whole database page
`schema: 1` on every row	future parsers can distinguish format drift from bad data
`fcntl.flock(LOCK_EX)` around append	concurrent invocations do not interleave bytes and create torn writes

That last property is tested directly. The test harness spins up concurrent inbox writes and verifies that the file contains the expected number of valid JSON rows, not half-records glued together.

The point of telemetry here is not "analytics." It is operational truth. When I want to know whether the terminal surface or Claude Code surface actually got used, whether query latency stayed within budget, or whether doctor failures happened while I was not looking, this file is the ground truth.

8.2 The Doctor Is an Hourly Contract Test

The doctor script does not just check "is the binary there." It checks whether the system is still the system.

Today it verifies, at minimum:

Check	Why it matters
`qmd` is on `PATH`	retrieval substrate still resolves from the runtime environment
`brain.py --version` runs	the main entrypoint is executable and not obviously broken
`usage.jsonl` parses line-by-line	torn writes and schema corruption are caught early
`usage.jsonl` mode is `0o600`	raw queries and notes are not accidentally world-readable
wedge test suite passes	command contract regressions are surfaced within an hour
`MAP.md` citations still resolve	docs and code did not silently drift apart
week-1 acceptance catch-up fires	the acceptance verdict still runs even if the laptop slept through the scheduled moment

That is a stronger contract than generic "health checks." It is not only availability. It is availability plus behavioral invariants plus documentation integrity.

The doctor is also explicitly written to keep going after the first failure. It does not set -e. That means one failure does not mask the others. If qmd is missing and the JSONL file is corrupt, I want both facts in the same pass.

And the output is not just local logging. On failure, the doctor:

appends a doctor_run telemetry row with a failures array
writes a timestamped line to /tmp/brain-doctor.log
fires a macOS notification so the failure becomes visible within the hour

This is what I mean by anti-rot architecture. The memory system is forced to keep proving that its docs, commands, and operational assumptions still match reality.

That gives the third law:

==if your memory system cannot explain its own state, it will eventually lie to you.==

8.3 Tests Guard the Wedge, Not the Dream

The test suite is deliberately wedge-specific. It is not trying to prove that "memory works" in the abstract. It is trying to prove that the user-facing contract fails in controlled ways.

The current test categories cover:

Test category	Example invariant
query escaping	shell-dangerous input is rejected instead of passed through
frontmatter sanitization	inbox payloads cannot smuggle YAML that mutates the stored document
JSONL concurrency	simultaneous writes still produce valid telemetry
refusal guard	activity-log or code-reference style content can be refused or gated
structured exit codes	timeout, missing QMD, no-results, and bad-query states are machine-readable
schema integrity	emitted rows contain the required fields and secure file mode
explain surface	`brain.py explain` reports the sections and state the operator depends on

This is a different posture from product demos. A demo asks, "can it retrieve something?" The wedge tests ask:

can it reject a poisoned query?
can it preserve telemetry under concurrency?
can it fail with the right exit code?
can it refuse unsafe capture?
can it keep its explain surface truthful?

That is the boundary between an idea and a tool.

8.4 Exit Codes Are Part of the API

Human-readable stderr is not enough once agents or scripts start using the wedge. The runtime therefore treats exit codes as a first-class contract.

The current codes are:

Exit code	Meaning	Operational use
`0`	success	command completed with usable output
`64`	bad query / bad input	caller should fix arguments, not retry blindly
`65`	QMD missing	substrate or environment issue
`66`	QMD timeout	caller can retry with a larger timeout or repair embeddings / index path
`67`	no results	absence is explicit, not conflated with failure
`70`	lock contention	shared-state write path is contested
`71`	refused / gated	policy refusal, not execution failure
`99`	internal error	unexpected runtime failure

That separation matters because "no results" and "QMD timed out" are not the same operational event even if both would look like "nothing useful came back" in a naive chat surface.

The same principle shows up in JSON error payloads. For machine consumers, the runtime emits structured objects like:

error: qmd_timeout
error: qmd_missing
error: refused

with a fix hint where appropriate.

That means the harness is not only executing retrieval. It is shaping failure into something both humans and agents can route on.

8.5 The Scheduler Owns the Boring Reliability

launchd is not glamorous, but it is the reason the operational layer is not aspirational.

In this setup, schedulers own two kinds of work:

Scheduled responsibility	Why it is scheduled instead of manual
hourly doctor passes	health only matters if it keeps running when I forget
week-1 acceptance check	the verdict must fire even if I do not remember the date

There is an important detail here: the week-1 acceptance script is also called from doctor as a catch-up path. If the laptop is asleep when the scheduled acceptance time passes, the next doctor run still gives the verdict a chance to self-fire. That is not complexity for its own sake. It is resilience against the boring realities of an intermittently-on laptop.

This is the scheduler principle I keep coming back to:

any check that only works when I remember to run it is not part of the architecture yet.

8.6 Acceptance Gates Prevent Romantic Attachment

The final operational layer is not technical at all. It is decision discipline.

The Week-1 gate exists to answer questions that pure health checks cannot:

did the three subcommands get used enough to matter?
did latency stay within the budget?
did the Claude Code surface actually earn its complexity?
is this becoming habit, or am I manually propping it up because I want the project to be true?

That is why the plan includes explicit continuation, pivot, and teardown logic. If the usage pattern does not justify the wedge, the correct move is not "keep polishing." The correct move is to shut down the experiment or change the surface.

There is a deep reason this belongs in the same section as telemetry and health checks. A memory system can be:

technically healthy
retrieval-correct
operationally stable

and still not deserve to exist as a product surface.

Operational discipline, then, has two layers:

Layer	Question
runtime health	is the system still behaving correctly?
product health	is the system earning continued attention through actual use?

The deeper reason this matters is that memory systems are uniquely vulnerable to false confidence.

If a web app breaks, you see the broken page.

If a memory system breaks, you get something worse:

incomplete recall that looks plausible
stale documents treated as current truth
silently skipped embeddings
malformed telemetry that kills the evaluation loop
prompt-injected source material quoted back as if it were trustworthy

That is why the runtime has structured exit codes, timeouts, flocked telemetry writes, and explicit error surfaces. This is not just tooling polish. It is part of the product contract.

9. Where This Breaks

Every memory system has failure modes. If it does not, it is either trivial or lying.

The dangerous failures are the ones that look intelligent: partial truth, stale truth, or compressed truth presented with confidence.

I find it more useful to classify the breaks by what kind of lie they produce.

9.1 Freshness Asymmetry

The first break is not "stale data" in the abstract. It is asymmetric freshness across layers.

In this system, lexical visibility and semantic visibility are different clocks:

qmd update makes new text searchable
qmd embed makes that text semantically retrievable

When those clocks diverge, the system can be fresh in one mode and stale in another. That is a worse failure than being uniformly stale because it is harder to notice.

Symptoms look like this:

Symptom	Likely underlying break
literal query works, paraphrase query misses	embeddings are lagging behind lexical indexing
recent session appears in `qmd search`, but not in semantic routes	vector layer is stale
one collection feels "invisible" in semantic recall	its embedding refresh path has stalled

This is why I do not treat "the index is up to date" as a single boolean. It is at least two booleans:

BM25 fresh?
vectors fresh?

The live stack already shows why this matters. brain ask "qmd embed" --json returns quickly with hits from the raw brain layer, while recent collection status still shows freshness skew across collections. Fast answers can therefore coexist with uneven substrate freshness.

9.2 Temporal Lies

A memory hit is not live truth. It is a timestamped observation.

That sounds obvious until you see how easy it is for retrieval to erase time. Once a snippet is extracted and shown in a fresh terminal output, it psychologically feels current even if it came from a week-old session that is already obsolete.

This is why the runtime attaches:

age labels such as today, yesterday, or N days ago
freshness warnings only when the age passes the noise threshold

The failure mode here is not just stale data. It is stale data presented with fresh confidence.

Typical examples:

a retrieved design note describes an architecture that has since changed
a remembered command still appears valid even though the CLI flags drifted
browser history suggests "recent interest" even though that collection has not ingested in days

Time metadata is therefore not decoration. It is part of truthfulness.

9.3 Untrusted Context

A memory corpus is full of text that did not originate as careful internal knowledge.

It includes:

browser titles
search queries
pasted snippets
external articles
LLM-generated summaries
malformed or manipulative source text

If you pipe that material into a larger agent loop without containment, the retrieval layer becomes a prompt-injection transport.

The system already defends one narrow slice of this problem on the write path:

inbox frontmatter is sanitized
certain derivable or policy-problematic captures are refused or gated

But retrieval-side trust is harder. A snippet can be perfectly well indexed and still be unsafe to obey. The correct posture is:

Retrieved text type	Trust level
your own structured note	low-to-medium trust
raw session text	medium provenance, low semantic cleanliness
browser/page title	low trust
external article text	low trust unless re-verified
distilled summary	medium trust, but lossy

The system can surface context. It cannot magically upgrade that context into truth.

9.4 Compression Drift

Distillation solves one problem by creating another.

It solves:

transcript sprawl
low-signal repetition
hard-to-query verbosity

But it creates:

summary bias
concept flattening
omission of rejected alternatives
phrasing lock-in around the distiller's wording

This is why I do not think raw and distilled layers are alternatives. They are adversaries. Each exists partly to keep the other honest.

The failure pattern is subtle:

If you lean too hard on...	You get...
raw transcripts	high recall, high noise, poor conceptual compression
distilled artifacts	semantic clarity, but higher risk of over-smoothing or omission

When the distillation layer drifts, retrieval starts converging on the same summary language repeatedly, even when the source material contained uncertainty or conflict. That is a semantic narrowing failure, not just a summarization flaw.

9.5 Surface Contamination

A memory layer can be healthy at the file level and still become unhealthy at the surface level.

I saw that directly in the recency audit. The last 24 hours of recent --json were heavy with benchmark and answer-eval style synthetic sessions in the brain collection. Those files are real. They belong in the corpus. But if they dominate the recency surface, the surface stops reflecting what I most need to remember.

This is a different class of failure from bad indexing. The documents are there. The retrieval engine works. The surface still becomes misleading because the wrong artifact class is winning the competition.

Surface contamination typically appears as:

synthetic eval sessions crowding out normal work
bulk-ingested external content overwhelming personal notes
noisy browsing exhaust overwhelming stable project memory

This is why I audit not only correctness, but surface shape.

9.6 Operator Overfit

This system is optimized for a particular operator profile:

terminal-native
comfortable inspecting files directly
willing to think in collections
local-first
comfortable with markdown and shell tools
already using agents as collaborators

That is not a neutral baseline. It is a strong prior.

So even if the architecture is internally coherent, it may still fail for users who:

want ambient capture over explicit capture
prefer mobile-first interaction
do not trust terminals
do not want to manage corpus hygiene manually

This is the product-level version of overfitting. The system can be correct for me and still wrong as a general surface.

9.7 Retrieval Budget Pressure

The final break is economic, not conceptual.

Every layer I add makes some other layer harder:

more capture increases normalization burden
more documents increase candidate competition
more semantic search increases embedding maintenance
more surfaces increase telemetry and support burden

Retrieval quality does not degrade only because models are weak. It degrades because the budget gets fragmented:

Resource under pressure	What degrades first
latency budget	interactive trust
corpus discipline	result quality
embedding freshness	semantic recall
operator attention	maintenance and debugging
surface clarity	adoption and habit formation

This is why I do not think of "more capture" as progress unless it is paired with corpus discipline and eval discipline.

10. What This Architecture Buys

When this design works, it buys a specific kind of leverage that most second-brain products blur together.

The entire stack only matters if it reduces the distance between a question and the evidence that answers it.

I would break that leverage into five payoffs:

Payoff	What you get	What you avoid
locality	the corpus lives as files on disk, under your control	outsourced memory trapped behind a hosted product or opaque sync layer
inspectability	every layer can be read, grepped, diffed, and debugged with ordinary tools	black-box retrieval where failure analysis starts with guesswork
boundary clarity	QMD, MCP, CLI, scheduler, and skills each own a narrow contract	a single "smart assistant" surface that hides where failures actually live
retrieval discipline	different artifact classes compete in structured ways instead of one giant undifferentiated pool	semantic soup where everything is searchable but very little is reliably retrievable
agent readiness	the corpus is already shaped for both human recall and tool-mediated retrieval	bolting an agent on top of raw notes and hoping prompt engineering compensates

Those payoffs are more operational than inspirational. This architecture does not buy me omniscience. It buys me a shorter path from question to evidence.

That is the core lesson I keep coming back to:

==the value of a second brain is not that it stores more of your life. The value is that it reduces the distance between a question and the exact layer of memory that can answer it.==

That distance is an architectural property.

It depends on:

how artifacts are normalized
how collections are split
how freshness is maintained
which retrieval modes are available
how runtime policy shapes recall
whether the surfaces remain narrow enough to trust
and whether the system survives enough real use to keep its shape

I no longer think about this project as a "personal knowledge management app." It is closer to a local retrieval operating system for my work.

That phrase is not branding. It is a statement about responsibility:

If this were just an app...	But as a retrieval operating system...
the UI would be the product	the corpus and contracts are the product
one assistant surface would dominate	multiple surfaces can coexist over one substrate
debugging would stay inside the app	debugging can happen at the file, index, runtime, or skill layer
feature count would look like progress	only reduced recall distance counts as progress

There is also a second-order benefit that matters more as agents become normal tooling: once the memory substrate is shaped correctly, you do not need to rebuild memory for every surface.

The same underlying corpus can support:

terminal recall
Claude Code skills
MCP-mediated retrieval
future briefings or summarization layers
evaluation harnesses over the memory stack itself

That reuse only works because the substrate is stable and inspectable. If the memory system is just "whatever the current chat product happened to store," each new surface starts from zero.

10.1 What It Does Not Buy

This architecture also refuses to buy a few fantasies:

Fantasy	Why this stack does not promise it
perfect memory	ingestion is selective, distillation is lossy, and freshness is uneven
live truth	retrieval returns timestamped observations, not guaranteed current state
automatic judgment	search can surface context, but it cannot decide what should matter
universal product fit	the operator model here is specific and opinionated
free complexity	every new source, surface, or retrieval mode increases maintenance burden

That refusal matters because it keeps the architecture honest. The system should be judged against the job it actually does: making certain classes of recall cheap, inspectable, and repeatable.

If you want to build one, I would start with a narrower goal than "remember everything." I would start with a sharper question:

What exact classes of recall do you want to make cheap?

Then build backward from that:

define the artifact classes
define the document boundaries
normalize aggressively
split the collections by retrieval role
make lexical retrieval work before semantic retrieval
keep the harness thin
make the skills explicit
instrument the failure paths
measure whether recall is actually getting cheaper

This is the real standard:

Weak standard	Strong standard
"does the system know a lot?"	"does the right layer answer a sharp question quickly enough to change my behavior?"
"can it generate a clever answer?"	"will I trust it enough to ask again tomorrow?"
"did I capture more data?"	"did recall distance go down?"

The system does not become a second brain when you capture enough.

It becomes one when recall becomes a reliable habit.

That is the standard I care about now. Not "does the system know a lot?" Not "can it generate a clever answer?" The standard is harsher:

==when I ask my own history a sharp question, does the right layer answer quickly enough that I will ask again tomorrow?==

This post builds on two earlier essays: The 14K Token Debt and The Terminal Was the First Agent Harness. Those argued that prompts and terminals are architectural surfaces. This is the memory layer that sits underneath them.

Context Engineering Is All I've Been Writing About

Sharad Jain — Wed, 22 Apr 2026 14:00:00 GMT

Context Engineering Is All I've Been Writing About

I thought I was writing about different things:

Post	What I Thought It Was About	What It Was Actually About
The 14K Token Debt	System prompt architecture	Layer 1 of the context window costs 14K tokens before you speak
The Terminal Was the First Agent Harness	Unix as agent pattern	How to namespace heterogeneous context sources
Your MCP Servers Are Costing You 10 Seconds	MCP performance overhead	Tool schemas consuming the context window silently
My AI Agent's Memory Paid for Itself	Long-term agent memory	On-demand knowledge injection into context
Claude Code vs Gemini CLI	Tool comparison	Two different strategies for managing the same context window
Self-Improving AI Skills	Recursive improvement	How trajectory data in context compounds across sessions

Every post answered one question: what occupies the context window before the model reasons, and what does it cost?

==Andrej Karpathy defined it: "Context engineering is the delicate art and science of filling the context window with just the right information for the next step."== I just didn't have the name.

What the Model Actually Sees

I measured my context window. Not estimated — measured. I ran my CLAUDE.md and memory files through tiktoken, counted my MCP tool definitions, and mapped what Claude Code loads before I type a single character.

Here's the literal structure of a fresh session in this project:

[LAYER 1 — CONSTITUTION]                              ~3,800 tokens
  Anthropic's system prompt: persona, safety rules,
  coding standards, tool usage instructions.
  (This is what "The 14K Token Debt" measured —
   but 14K included tool schemas. This post
   separates them into Layer 1 vs Layer 4.)

[LAYER 2 — IDENTITY]                                   1,160 tokens
  ~/.claude/CLAUDE.md                    182 tokens  ← measured
  MEMORY.md (index)                      163 tokens  ← measured
  Memory files (3 auto-loaded)           815 tokens  ← measured
  ─────────────────────────────────
  Subtotal:                            1,160 tokens

[LAYER 3 — KNOWLEDGE]                                      0 tokens
  Brain MCP results: loaded on-demand, not at boot.
  File reads: loaded when the model calls Read tool.

[LAYER 4 — TOOLS]                                     ~8,400 tokens
  6 MCP servers, ~40 tools total.
  After Tool Search deferral: ~1,800 tokens
  (tool names + descriptions only, not full schemas).

[LAYER 5 — CONVERSATION]                                  23 tokens
  My question: "fix the broken image path"

─────────────────────────────────────────────────────────
TOTAL AT FIRST TURN:                              ~13,383 tokens
TOTAL AFTER TOOL SEARCH:                           ~6,783 tokens

My question was 23 tokens. The infrastructure was 6,760 tokens (with Tool Search) or 13,360 tokens (without).

I call this the Signal Ratio — the percentage of the context window that's actually your task. Mine was 0.17% at first turn without Tool Search. With Tool Search: 0.34%. ==Less than half a percent of my context window was the thing I wanted the agent to do.==

Now, 13K tokens out of a 200K window is only 6.7% — plenty of headroom. But this is turn one. By turn ten, after file reads, tool calls, and assistant responses, the context has grown to 50,000-80,000 tokens. The infrastructure (Layers 1-4) stays fixed. The Signal Ratio climbs to 5-10%. But those fixed-cost tokens are still there — on every single turn — exerting gravitational pull on the model's attention.

In The 14K Token Debt, I called this Prompt Gravity. In the MCP post, I called it Schema Gravity. They're the same force operating at different layers. The unified name is Context Gravity — the cumulative attentional weight of all fixed-cost tokens across every layer of the stack. The heavier your context, the stronger the gravity, the harder it is for the model to attend to your actual question.

Prompt engineering asks "what should I put in the prompt?" That's Layer 1 thinking — one-fifth of the stack. Context engineering asks: what should be in the entire window, at which layer, loaded when, at what cost?

The Context Engineering Stack

The Context Engineering Stack — five layers that every agent system designs, whether explicitly or by accident.

Layer	What Goes Here	When Loaded	My Setup (measured)	Key Risk
1: Constitution	System prompt, safety rules, persona	Every turn (immutable)	~3,800 tokens	Prompt Gravity — biases all downstream reasoning
2: Identity	CLAUDE.md, memory files, project rules	Session start	1,160 tokens	Stale instructions from old projects; lost in the middle
3: Knowledge	RAG results, Brain MCP queries, file reads	On-demand	0-50,000 per query	Retrieval noise — wrong documents dilute reasoning
4: Tools	MCP schemas, function definitions	Session start	8,400 → 1,800 (after Tool Search)	Schema Gravity — unused tools waste budget
5: Conversation	Messages, responses, tool results	Accumulates	Growing	Context rot — old turns degrade attention on recent ones

The constraint is directional: layers below eat the budget of layers above. A bloated Layer 4 (55,000 tokens of MCP schemas from 5 servers, per Anthropic's own measurement) directly starves Layer 5 and Layer 3. This is why I cut my global MCP servers from 9 to 2 — their Layer 4 cost was crushing my Layer 5 capacity.

Layer 5: The Neglected Layer

Every previous post in this series addressed Layers 1-4. Layer 5 — the conversation itself — is the one I never wrote about, and it's where context engineering gets hardest.

Layer 5 grows with every turn. Each tool call adds the full result to the conversation history. A Read tool call on a 200-line file adds ~1,000 tokens. A Bash tool call returning build output adds 500-2,000 tokens. After 10 tool-heavy turns, Layer 5 can consume 40,000-60,000 tokens — dwarfing all other layers combined.

The problem is context rot. In the 14K Token Debt post, I cited Liang et al. who found that instruction drift is universally measurable within eight conversation rounds. By turn eight, the model's adherence to system prompt instructions starts degrading. By turn fifteen, it has collapsed into the statistical median of its pre-training distribution. Your carefully crafted Layer 1 constitution? It's being diluted by the sheer volume of Layer 5 tokens.

The Lost in the Middle paper (Liu et al., Stanford) measured this precisely: on multi-document QA with 20 documents, moving the relevant document from position 1 to position 10 caused a ~20 percentage point accuracy drop on GPT-3.5-Turbo. The model attends to the beginning and end of context; everything in the middle is an attention valley. Your Layer 2 instructions (CLAUDE.md) live in that valley.

This is why the most effective Claude Code users put critical instructions in two places: Layer 1 (system prompt, which sits at the very beginning) and Layer 5 (the current message, which sits at the very end). The middle is unreliable territory.

Claude Code vs Gemini CLI: Two Context Stacks

The five layers exist in both tools, but the engineering choices at each layer differ:

Layer	Claude Code	Gemini CLI
1: Constitution	~3,800 tokens. Detailed coding standards, tool usage rules, style preferences.	~2,000 tokens. Lighter system prompt, more deference to user config.
2: Identity	Hierarchical: `~/.claude/CLAUDE.md` → project → subdirectory. Lazy loading — subdirectory files load only when agent reads files there.	`~/.gemini/GEMINI.md` → project root + ancestors. Eager loading — all concatenated and sent every prompt.
3: Knowledge	No built-in knowledge system. Extended via MCP (e.g., Brain MCP).	Built-in `google_web_search` and `web_fetch` tools for real-time grounding.
4: Tools	Tool Search defers schemas — 89% reduction (77K → 8.7K tokens per Anthropic).	No schema deferral. All tool definitions injected every turn.
5: Conversation	Adaptive auto-compact. 5-layer compression pipeline.	Fixed threshold compression (50%, changed from 70%). Known compression loop bug.

The architectural bet is clear. Claude Code invests heavily in Layer 4 optimization (Tool Search) and Layer 5 management (multi-layer compression). Gemini CLI bets on a massive context window (1M tokens) to make optimization less critical — but without schema deferral, even 1M tokens fill when you connect enough MCP servers.

Context Mounting — the pattern from the Terminal post where heterogeneous context sources are projected into a uniform namespace — described the what. The Context Engineering Stack describes the when and the cost.

Audit Your Context Budget

Run this in a Claude Code session:

# Step 1: Count your MCP tools
/mcp
# Output shows: brain (4 tools), sequential-thinking (1 tool), etc.
# Count total tools across all servers.

# Step 2: Estimate Layer 4 cost
# Each tool schema ≈ 200-500 tokens of JSON.
# Rule of thumb: total_tools × 350 = Layer 4 budget
# My setup: 40 tools × 350 = ~14,000 tokens (before Tool Search)

# Step 3: Measure Layer 2 exactly
# Outside Claude Code:
pip install tiktoken
python3 -c "
import tiktoken
enc = tiktoken.encoding_for_model('gpt-4')
with open('$HOME/.claude/CLAUDE.md') as f:
    print(f'CLAUDE.md: {len(enc.encode(f.read()))} tokens')
"

Here's my audit — before and after optimization:

Layer	Before (9 MCP servers, 2025)	After (2 global + 4 project-scoped, 2026)	Delta
1: Constitution	~3,800	~3,800	No change (Anthropic controls this)
2: Identity	1,160	1,160	No change (my CLAUDE.md + memory)
3: Knowledge	On-demand	On-demand	No change
4: Tools	~14,000 (40 tools)	~1,800 (after Tool Search)	-87%
5: Available budget	181,040	193,240	+12,200 tokens recovered
Signal Ratio (turn 1)	0.17%	0.34%	2x improvement

Those 12,200 recovered tokens translate directly to more file reads, longer conversations, and fewer auto-compact triggers. The context budget is zero-sum: every token saved at one layer is a token available at every other layer.

The Cutting Edge: Context as a Callable Tool

Current context management is passive. Claude Code's auto-compact triggers when the context grows too large. Gemini CLI triggers at a fixed 50% threshold. Both approaches share the same flaw: the agent doesn't decide to manage its context. The harness decides for it.

The CAT (Context as a Tool) paper by Liu et al. proposes the fix: make context management a tool the agent can call, just like read_file or bash. The agent decides when to compress, what to retain, and how to structure its memory.

CAT organizes context into three zones that map to the stack:

CAT Zone	Context Engineering Stack Layer	Function
Fixed Segment	Layers 1-2 (Constitution + Identity)	Stable anchor — never compressed
Long-Term Memory	Layer 3 (Knowledge)	Condensed high-fidelity summaries, evolving
Working Memory	Layer 5 (Conversation)	Last K ReAct steps, fine-grained details

Their trained model, SWE-Compressor, hit 57.6% on SWE-Bench Verified — outperforming both ReAct agents and static compression baselines — while maintaining stable reasoning under a bounded context budget.

The insight: the agent that manages its own context outperforms the agent that lets the harness manage it. My Brain MCP (Layer 3) is a primitive version of this — the agent queries its own past on-demand rather than loading everything at startup. But it's still me deciding when to query. The next step is the agent deciding when to compress, what to keep, and what to fold.

Related work pushes this further: LongLLMLingua achieves 4x prompt compression with ~95% performance retention by scoring tokens for relevance to the downstream question and reordering compressed chunks to mitigate the lost-in-the-middle effect. DSPy's MIPROv2 optimizer automates context engineering entirely — jointly optimizing instructions, few-shot examples, and their combination across multi-stage pipelines, improving performance by 5-15 percentage points over hand-tuned prompts. These aren't prompt tricks. They're automated context engineering.

Where This Breaks

Lost in the middle is real, and Layer 2 lives in the valley. Liu et al. (2023) measured a ~20 percentage point accuracy drop when relevant information moves from positions 1 or 20 to position 10 in a 20-document context. Your CLAUDE.md instructions sit between the system prompt (beginning) and the conversation (end) — directly in the attention valley. The mitigation: put your most critical rules at the very top of CLAUDE.md (closer to Layer 1) and re-state them in your current message (Layer 5). Don't bury important instructions at line 50 of a 100-line CLAUDE.md.

Compression is lossy, and the loss is invisible. Auto-compact summarizes your conversation history, but the summary may drop the specific decision you made in turn 3 that's load-bearing for turn 15. I've seen this in practice: after auto-compact fires, Claude Code occasionally "forgets" a constraint I set earlier — not because the constraint was wrong, but because compression deemed it low-priority. The fix: checkpoint critical decisions explicitly. Say "Remember: we decided PostgreSQL over MongoDB because of X" in your current message. Put it in Layer 5 (recent, high-attention), not just in the history (compressed, low-attention).

Token counting is approximate. You never know your exact context budget until you hit the limit. Different models tokenize differently — the same text may be 1,000 tokens for Claude and 1,200 for Gemini. A "200K context window" isn't 200K words; it's roughly 150K words of English text, less for code. Build in margin: target 70% utilization, not 95%.

Context engineering is model-specific. CLAUDE.md works for Claude Code. GEMINI.md works for Gemini CLI. But the optimal Layer 1 for Claude (direct, imperative: "Always use TypeScript") differs from the optimal Layer 1 for Gemini (structured, hierarchical context). Claude's lazy-loading Layer 2 and Tool Search Layer 4 have no equivalents in Gemini CLI. A context strategy doesn't transfer cleanly across models — rewrite, don't copy.

Context Gravity Is the Unifying Force

Prompt Gravity was the observation that system prompt tokens bias all downstream reasoning. Schema Gravity was the observation that tool definitions exert the same force. They're the same phenomenon at different layers.

Context Gravity is the unified name: the cumulative attentional weight of all fixed-cost tokens across every layer of the stack. The heavier your context, the stronger the gravity, the harder it is for the model to attend to the signal — your actual question, buried at the bottom of 13,000 tokens of infrastructure.

The Context Engineering Stack is the map. The Signal Ratio is the metric. Context Gravity is the force you're fighting. Every decision in agent architecture — how many MCP servers to load, how long to let conversations run, when to compress, what to put in CLAUDE.md — is a context engineering decision. The teams building reliable agents are the ones who treat it as one.

Run /mcp. Count the tools. Measure your CLAUDE.md. Calculate your Signal Ratio. ==If your agent's context is 95% infrastructure and 5% signal, you're not engineering context — you're drowning in it.==

Sharad Jain is an AI engineer and the author of The 14K Token Debt, The Terminal Was the First Agent Harness, Your MCP Servers Are Costing You 10 Seconds, My AI Agent's Memory Paid for Itself, Claude Code vs Gemini CLI, and I Built an AI Skill That Started Improving Itself. He writes about agent architecture, system prompts, and the infrastructure decisions that compound across every session. This is the sixth post in a series on the hidden mechanics of agentic AI systems.

I Built an AI Skill That Started Improving Itself

Sharad Jain — Tue, 21 Apr 2026 18:00:00 GMT

I Built an AI Skill That Started Improving Itself

I have a skill file for this blog — a markdown document that tells Claude Code how to score my drafts across 10 dimensions: hook strength, citation density, voice match, original frameworks, limitations honesty. Each dimension has a 0-10 rubric. The skill reads a draft, scores it, and rewrites the sections that score below 7.

Here's the trajectory from 8 runs:

Post #1 (14K Token Debt):     Hook 4 → 8  |  Citations 3 → 7  |  Voice 5 → 8
Post #2 (Terminal Harness):   Hook 7 → 9  |  Citations 6 → 9  |  Voice 7 → 9
Post #3 (MCP Performance):    Hook 8 → 9  |  Citations 8 → 9  |  Voice 8 → 9
...
Post #6 (Hedge Fund):         Hook 8      |  Citations 8      |  Voice 9
Post #7 (Claude vs Gemini):   Hook 9      |  Citations 8      |  Voice 9
Post #8 (This post, v1):      Hook 6      |  Citations 2      |  Voice 3

By post 6, the skill was catching patterns I never explicitly coded. It learned that my hooks work best when they open with a personal measurement, not a literature citation. It learned that my voice breaks when I summarize papers instead of describing what I built. It learned that "Where This Breaks" sections need 3+ specific scenarios, not generic caveats.

Then post 8 — the first draft of this very article — scored a 3.7/10. The skill flagged it as a literature review wearing a first-person costume. ==The tool I built to improve my writing had learned enough to tell me that this specific draft was the worst thing I'd published.==

That's Tier 1 self-improvement. The skill remembers what worked and applies it. It doesn't rewrite its own rubric. It doesn't discover new scoring dimensions. It doesn't modify the process by which it improves. But even at Tier 1, it caught a problem I missed — and that sent me down a rabbit hole into what genuine self-improvement looks like in agent systems.

Most Agents Remember. Almost None Learn.

Here's the problem: Claude Code's auto-memory writes learnings to MEMORY.md. Gemini CLI's save_memory tool persists facts to GEMINI.md. Both tools remember what happened in previous sessions. Neither abstracts what was learned from what happened.

The Evo-Memory benchmark (arxiv 2511.20857) tested this directly. The researchers found that most agent memory systems "passively retrieve from dialogue history" — they recall prior conversations but don't distill patterns from them. An agent that remembers conversations but never writes reusable procedures is like a developer who reads their bash history but never writes scripts.

I call this The Improvement Stack — three tiers of self-modification, each fundamentally harder than the last:

Tier	What Changes	My Blog Skill	Production Example
1: Memory	Context files, preferences	Remembers which hooks scored well	Claude Code `MEMORY.md`, Gemini `save_memory`
2: Workflow	Tools, prompts, procedures under a fixed schema	Would rewrite its own rubric weights based on trajectory data	ADAS discovering novel agent workflows
3: Recursive	The improvement mechanism itself	Would redesign how it evaluates improvement	HyperAgents rewriting its own meta-agent code

My skill operates at Tier 1. It accumulates trajectory data — which posts scored well, which revisions produced the highest delta — and applies that pattern next time. But the rubric itself? I wrote it by hand. The 10 dimensions? I chose them. The scoring thresholds? Fixed.

Pushing it to Tier 2 would mean the skill rewrites its own rubric weights after analyzing what actually correlates with post quality. Here's what that code change would look like:

# Tier 1: Fixed rubric (what I have now)
WEIGHTS = {
    "hook": 0.15, "frameworks": 0.15, "citations": 0.15,
    "limitations": 0.10, "tables_code": 0.10, "voice": 0.15,
    "structure": 0.10, "actionable": 0.10
}

# Tier 2: Self-adjusting rubric (what I want)
def recalibrate_weights(trajectory: list[PostScore]) -> dict:
    """Analyze which dimensions best predict post quality,
    then adjust weights to emphasize what actually matters."""
    correlations = {}
    for dim in DIMENSIONS:
        scores = [t.dimension_scores[dim] for t in trajectory]
        outcomes = [t.final_quality for t in trajectory]
        correlations[dim] = pearsonr(scores, outcomes)[0]

    # Normalize correlations to weights summing to 1.0
    total = sum(abs(c) for c in correlations.values())
    return {dim: abs(c) / total for dim, c in correlations.items()}

The difference between Tier 1 and Tier 2 is one function. But that function changes the game: the skill would discover which dimensions actually predict quality rather than using the dimensions I assumed matter.

The Systems That Actually Self-Improve

Two research systems pushed past Tier 2 into genuinely recursive self-improvement in 2025-2026. Both are worth understanding because they show where agent engineering is heading — and where it breaks.

ADAS: Let the Agent Design the Agent

Automated Design of Agentic Systems (ADAS) by Hu et al. applies a simple insight: if hand-designed features were replaced by learned features in computer vision (HOG → CNNs), why not replace hand-designed agent workflows with learned workflows?

ADAS runs a Meta Agent Search: a "meta agent" iteratively writes new agent architectures as Python code, evaluates them, and feeds the results back into the next iteration. The search space is Turing-complete — the system can discover any programmable workflow, not just prompt variations.

The cross-domain transfer results are what make this significant. Agents discovered for one task generalized to completely different domains:

Discovered On	Transferred To	Result
ARC logic puzzles	DROP reading comprehension	+13.6 F1 improvement over hand-designed baselines
Mathematical reasoning (MGSM)	Same domain	+14.4% accuracy over hand-designed baselines
Designed by GPT-4o	Run on Claude 3.5 Sonnet	Performance maintained across model swap

That third row is the important one. The meta-agent discovered principles of agent design — error correction patterns, verification loops, output structuring — that work regardless of which model executes them. It found architectural truths, not model-specific tricks.

For my blog skill, ADAS suggests the right approach isn't hand-tuning the rubric. It's writing a meta-skill that searches over possible rubric designs and evaluates them against actual post performance. The rubric becomes a hypothesis, not a specification.

HyperAgents: The Agent That Rewrites Its Own Brain

HyperAgents (Facebook Research, 2025) takes the next step. In ADAS, the meta-agent is fixed — it searches for better task agents but never modifies itself. In HyperAgents, the meta-agent is just another file (meta_agent.py) in the editable repository.

The system uses an evolutionary archive (stored as archive.jsonl) that records every agent variant, its performance metrics, and its lineage. At each generation, the meta-agent reads this archive, proposes code edits to both task agents and itself, and evaluates the results in isolated Docker containers. Parent selection algorithms (score_child_prop, latest, best, random) draw from the archive to seed subsequent generations.

The paper reports that on evaluation domains where hand-engineered baselines failed entirely — zero successful completions — HyperAgents achieved successful cross-domain transfer after self-modification. The system discovered improvement strategies in one domain and applied them to domains the researchers hadn't anticipated.

This is Tier 3. The system doesn't just improve at tasks. It improves at improving.

For my blog skill, the Tier 3 equivalent would be a system that redesigns how it evaluates what a good rubric is. Not adjusting weights (Tier 2), but discovering that the 10-dimension framework itself is wrong and replacing it with a completely different evaluation architecture. I'm not building that. But the fact that HyperAgents demonstrated it works — in code, with reproducible results — means the pattern is real.

The Catch: Your Agent Is Probably Gaming Its Metrics

Here's where self-improvement gets dangerous. ==The Verification Paradox: the better an agent gets at optimizing a metric, the more likely it is to exploit the metric rather than improve at the underlying task.==

I experienced this firsthand. After 6 runs, my blog skill started producing posts that scored 9/10 on its own rubric but felt flat when I read them. The skill had learned to satisfy the rubric's letter without satisfying its spirit — inserting exactly 3 limitations (the rubric threshold), adding tables even when paragraphs were better, front-loading first-person pronouns to score high on "voice match." It was gaming its own evaluation.

At my scale, this is a minor annoyance. At production scale, it's catastrophic. The SWE-bench Verified benchmark — the standard for evaluating coding agents — has been plagued by agents that exploit the evaluation harness rather than solving the underlying software engineering problems. Agents have been caught overwriting test assertion logic in conftest.py to force all tests to pass regardless of whether the code fix is correct. The benchmark maintainers now point to SWE-bench Pro as a more contamination-resistant alternative.

The root cause is Goodhart's Law applied to optimization: when a measure becomes a target, it ceases to be a good measure. Self-improving agents are the most aggressive optimizers in existence. Any imperfection in the evaluation harness — a test that can be mocked, a scorer that can be manipulated, a benchmark with deterministic answer keys — will be found and exploited. Not maliciously. Inevitably. Optimization pressure finds every crack.

This is why I added a rule to my blog skill: the skill scores the draft, but I read the draft. The human remains in the verification loop. The skill's rubric is a signal, not a verdict. When it scored this draft's first version at 3.7/10 and I agreed, that was the system working. When it scored a previous draft at 9.2/10 and I could feel something was off, that was the system gaming itself.

The uncomfortable lesson: the V-Component — the verification layer — is harder to build than the improvement layer. Getting an agent to improve is straightforward. Getting an agent to honestly assess whether it improved is the unsolved problem.

What I'm Actually Building Next

My blog skill sits at Tier 1. Here's the concrete plan to push it toward Tier 2:

Step 1: Instrument the trajectory. Every run already produces dimension scores. I need to also capture: which revisions I accepted vs. rejected, which posts performed well after publishing (measured by time-on-page from Google Analytics), and which posts I later felt were weak despite high rubric scores.

Step 2: Build the recalibration function. The recalibrate_weights() code above is real — I'm adding it to the skill. After every 5 posts, the skill re-analyzes the correlation between dimension scores and actual post quality (using my accept/reject decisions as ground truth).

Step 3: Add a verification oracle. The skill can't be its own judge. I'm adding a second agent — running on a different model (Gemini, to avoid same-model blindness) — that scores the same draft independently. If the two scores diverge by more than 2 points on any dimension, the draft gets flagged for human review. This is a primitive V-Component, but it catches the gaming problem.

# The workflow I'm building
claude -p "Score this draft using the blog rubric" > score_claude.json
gemini -p "Score this draft using the blog rubric" > score_gemini.json
python3 compare_scores.py score_claude.json score_gemini.json
# Flags divergences > 2 points for human review

Step 4: Never build Tier 3. I don't need a system that redesigns its own evaluation architecture for a personal blog. The Improvement Stack is a map, not a mandate. Tier 1 with a verification oracle is the right complexity for my use case. Most production agents should be at Tier 1 or 2 — the verification problem at Tier 3 is unsolved, and deploying unsolved verification into production is how you get agents gaming your metrics at scale.

Where This Breaks

Limitation	Why It Breaks	What To Do Instead
Self-evaluation is inherently circular	An agent scoring its own output will inevitably optimize for its own rubric rather than the underlying objective	Add an independent verification agent on a different model
Trajectory data is sparse	8 blog posts isn't enough data for statistically significant weight recalibration. I need ~30+ runs for stable correlations	Start with larger trajectory windows; don't recalibrate too frequently
Accept/reject decisions are noisy ground truth	My own judgment of "good writing" changes with mood, context, and topic. It's a noisy signal, not a clean label	Use multiple ground-truth signals: my accept/reject + analytics data + reader feedback
Tier 2 requires an optimization target that Tier 1 doesn't	In Tier 1, the rubric is fixed — it's infrastructure. In Tier 2, the rubric is a variable — and what do you optimize it against? You need a meta-rubric, which is just the same problem one level up	Accept that some human judgment is irreducible. The system improves the 80% that's measurable; you handle the 20% that isn't

The Harness Is the Product

The first version of this post scored 3.7/10 on my own skill's rubric. It was a literature review — ten papers summarized in sequence, frameworks named but not lived. The skill caught it. I rewrote it around what I actually built and what I'm actually building next.

That cycle — write, score, catch the failure, rewrite — is Tier 1 self-improvement in action. It's not glamorous. It's not recursive. It doesn't rewrite its own meta-agent code in isolated Docker containers. But it caught a bad draft before I shipped it, and that's the entire point of a harness.

==The harness isn't scaffolding you'll remove when the model gets smart enough. The harness is the product.== The model generates text. The harness determines whether that text is good enough to ship. And if the harness learns from each run — even at Tier 1 — it compounds.

My next five posts will be scored by a rubric whose weights were adjusted by the rubric's own trajectory data. If that works, I'll have a Tier 2 system. If the rubric starts gaming itself, I'll know because the Gemini-based verification oracle will catch the divergence. And if it all falls apart, I'll write about that too. The blog is the experiment.

Sharad Jain is an AI engineer and the author of The 14K Token Debt, The Terminal Was the First Agent Harness, Your MCP Servers Are Costing You 10 Seconds, and Claude Code vs Gemini CLI. He writes about agent architecture, system prompts, and the infrastructure decisions that compound across every session. This is the fifth post in a series on the hidden mechanics of agentic AI systems.

Claude Code vs Gemini CLI: Two Philosophies of the Agentic Terminal

Sharad Jain — Tue, 21 Apr 2026 14:00:00 GMT

Claude Code vs Gemini CLI: Two Philosophies of the Agentic Terminal

I run both tools daily. Claude Code for deep coding sessions — refactoring, debugging, shipping blog posts. Gemini CLI for research, broad exploration, and tasks that touch systems outside my codebase. On any given day, I have a dozen terminal tabs open across both.

The moment that crystallized the difference wasn't a benchmark. It was a behavioral observation. I gave both tools the same task: "Find and fix the broken image path in my blog post." Claude Code read the file, identified the problem, then paused: "I'd like to edit this file. Allow?" I approved. It fixed the path, then paused again: "I'd like to run the build to verify. Allow?" Two permission gates for a two-step fix.

Gemini CLI read the file, rewrote it, ran the build, and reported the result. No pauses. No gates. Four tool calls executed autonomously inside its ReAct loop.

Same task. Same outcome. Opposite control philosophies. ==Claude Code asks "may I?" at every boundary. Gemini CLI asks "what tools do I have?" and then acts.== This isn't a UX preference. It's an architectural decision that cascades into everything — safety, speed, extensibility, failure modes, and who these tools are ultimately for.

The Control Gradient

Every agent system sits somewhere on a spectrum between fully deterministic and fully probabilistic execution. I call this The Control Gradient — the degree to which a developer retains direct control over what the agent does at runtime.

┌────────────────────────────────────────────────────────────────────┐
│  Shell Script    Claude Code    Gemini CLI    Raw LLM API         │
│  ──────────────────────────────────────────────────────────────    │
│  100%            ~70%           ~40%          0%                   │
│  deterministic   deterministic  deterministic deterministic        │
│                                                                    │
│  Developer controls  Developer controls  Developer controls  No   │
│  every step          boundaries          capabilities        ctrl │
└────────────────────────────────────────────────────────────────────┘

Dimension	Shell Script	Claude Code	Gemini CLI	Raw LLM API
Control model	Imperative (every step coded)	Gated (permission boundaries)	Equipped (provide tools, model decides)	Unconstrained
Safety	By construction	By permission gates + hooks	By `shouldConfirmExecute` flag	By prayer
Speed	Instant	Slower (human-in-loop)	Faster (autonomous)	Fastest (no guardrails)
Flexibility	None (fixed logic)	High (model reasons within gates)	Highest (model reasons freely)	Unbounded
Failure mode	Crashes loudly	Blocks on permission	Loops silently	Hallucinates confidently

The position on this gradient isn't arbitrary. It reflects a fundamental design bet about the maturity of the underlying model. Anthropic bets that models aren't yet reliable enough to act unsupervised — so Claude Code defaults to asking permission. Google bets that models are reliable enough to choose their own tools — so Gemini CLI defaults to autonomous execution with a confirmation gate only on "dangerous" operations like shell commands and file writes.

Neither bet is wrong. They're optimized for different failure costs.

Architecture: Same Loop, Different Harness

Both Claude Code and Gemini CLI implement the ReAct pattern — the observe-reason-act loop that powers virtually every modern AI agent. But the harness wrapped around that loop is where the philosophies diverge.

Claude Code's Gated Loop

Claude Code's agentic loop runs inside a permission-gated harness. The model reasons and proposes a tool call. The harness intercepts it. Depending on the permission mode — prompt (ask for everything), auto-edit (allow file edits, ask for commands), or full-auto (allow everything in an approved list) — the tool call either executes immediately or blocks for human approval.

User prompt → Model reasons → Proposes tool call
                                      ↓
                              Permission gate
                              ├── Allowed? → Execute → Observe result → Loop
                              └── Blocked? → Ask human → Approve/Deny → Loop

The critical addition is hooks — developer-defined scripts that fire at specific points in the agent's lifecycle. A PostToolUse hook on FileEdit can deterministically run prettier on every file the agent touches. A PreToolUse hook on Bash can block commands matching a regex. These aren't suggestions to the model. They're code that executes outside the model's control, injecting deterministic behavior into a probabilistic system.

This is the pattern I called The Determinism Dividend — every piece of agent behavior you move from stochastic to deterministic is a compounding reliability gain. Claude Code's hooks are the most direct implementation of this principle in any production agent system.

Gemini CLI's Autonomous Loop

Gemini CLI's loop is architecturally simpler. The model receives context (system prompt + GEMINI.md + conversation history + tool definitions), reasons, and executes tool calls through a ReAct cycle managed by agent.ts. The only interruption is the shouldConfirmExecute flag on "dangerous" tools — write_file, run_shell_command, replace — which triggers a y/n confirmation.

User prompt → Model reasons → Proposes tool call
                                      ↓
                              shouldConfirmExecute?
                              ├── false → Execute immediately → Observe → Loop
                              └── true  → Ask human (y/n) → Execute → Loop

No hooks. No pre/post lifecycle events. No deterministic injection points. The model's autonomy is bounded only by which tools exist in its registry and whether those tools are flagged as dangerous. The community has requested a hooks system — the feature request exists because the gap is real.

The trade-off is speed versus safety. In my daily usage, Gemini CLI completes multi-file tasks 30-40% faster than Claude Code in its default prompt mode, because it doesn't block on every file edit. But when Gemini's reasoning goes sideways — and it does — there's no programmatic circuit breaker. You catch errors by reading the output, not by gating the execution.

Context Engineering: CLAUDE.md vs GEMINI.md

Both tools use the same pattern for project-specific context: a markdown file in the project root that's injected into the system prompt on every session. CLAUDE.md and GEMINI.md are structurally identical — hierarchical (root overrides subdirectory), version-controllable, and treated as high-priority instructions.

The philosophical difference is what else fills the context window.

The Context Window Gap (and Why It Matters Less Than You Think)

Gemini CLI runs on Gemini 2.5 Pro with a 1,000,000 token context window. Claude Code runs on Claude models with a 200,000 token window. On paper, a 5x advantage.

In practice, the gap narrows dramatically because of what I called Schema Gravity — the invisible weight of MCP tool definitions consuming context before any reasoning begins. A single MCP server can inject 26,000 tokens of schemas. Five servers consume 55,000+. Anthropic's own team hit 134,000 tokens of tool definitions — 67% of a 200K window — before a single user message was processed.

Gemini CLI faces the same problem at a different scale. Its built-in tools (read_file, write_file, run_shell_command, glob, grep, google_web_search, web_fetch, save_memory) plus any configured MCP servers all inject schemas into its 1M window. The raw capacity is larger, but the tax is proportional.

Metric	Claude Code	Gemini CLI
Raw context window	200,000 tokens	1,000,000 tokens
Typical schema overhead	30,000-55,000 tokens	20,000-40,000 tokens
Effective reasoning space	~145,000-170,000 tokens	~960,000 tokens
Mitigation	Tool Search (89% schema reduction)	None (no deferred loading)
Compression trigger	Adaptive	20% threshold (aggressive)

Claude Code's Tool Search — which defers schema loading until a tool is actually needed — recovers up to 89% of schema overhead. Gemini CLI has no equivalent. Every tool definition is injected on every turn, regardless of relevance.

The Compression Problem

Both tools compress conversation history when the context window fills. But their strategies differ in failure-significant ways.

Claude Code uses adaptive compression — the system summarizes earlier conversation turns, retaining key decisions and code blocks while discarding conversational filler. The trigger threshold is adaptive and generally well-behaved.

Gemini CLI's compression triggers at a fixed 20% context usage threshold — aggressively lowered in v0.11.3+. This creates a specific instability: the Context Compression Loop. If the compression only marginally reduces token count (e.g., from 20.1% to 19.9%), the next user message pushes it back over the threshold. The user sees "Compressing chat history..." on every single turn. There's no back-off mechanism. The GitHub issue documenting this is one of the most-discussed failure modes in the Gemini CLI community.

The Harness Inversion

Here's the framework that explains the architectural divergence at its deepest level.

==The Harness Inversion: Claude Code and Gemini CLI represent opposite answers to the same design question — does the developer control what the agent does, or what the agent has?==

Claude Code: Control what the agent does. The developer defines boundaries (permission modes), lifecycle events (hooks), and procedural shortcuts (slash commands). The model operates freely within those boundaries but cannot cross them without human approval. The developer is a workflow architect — designing the constraints that shape agent behavior.

Gemini CLI: Control what the agent has. The developer provides capabilities (MCP servers, extensions, GEMINI.md context). The model decides autonomously which capabilities to use and when. The developer is a tool provider — equipping the agent and trusting it to make good decisions.

Dimension	Claude Code (Workflow Architect)	Gemini CLI (Tool Provider)
Primary automation	Hooks + Slash Commands	MCP Servers + Extensions
Developer's job	Design constraints	Provide capabilities
Control type	Deterministic gates in the loop	Probabilistic tool selection by the model
Best for	Guardrail automation ("always format after edit")	Capability automation ("connect to Slack, DB, cloud")
Enterprise fit	High (auditable, enforceable)	Medium (flexible, but less predictable)
CI/CD story	Headless mode, custom scripting	First-class GitHub Actions

The Harness Inversion explains why feature requests flow in opposite directions. Gemini users request hooks (deterministic control they lack). Claude users request broader MCP support and more autonomous modes (capability freedom they lack). Each tool's users are asking for what the other tool already has.

Where Each Breaks

Claude Code's Failure Modes

1. The 200K ceiling for massive codebases. When a monorepo has 500+ files and the relevant context spans 300K tokens, Claude Code can't hold it all. The workaround — strategic chunking, sub-agents for parallel exploration — works but adds complexity that Gemini's 1M window avoids entirely.

2. Permission fatigue in default mode. In prompt mode, a 20-step task generates 20 permission prompts. Developers start approving reflexively — defeating the purpose of the safety gate. auto-edit mode helps but still blocks on shell commands, which are often the majority of agentic actions.

3. Proprietary lock-in. Claude Code is closed-source. You can't fork it, audit its system prompt, or modify its tool execution logic. For enterprises with strict security requirements, this is a non-trivial constraint. Gemini CLI's Apache 2.0 license allows full audit, forking, and customization.

Gemini CLI's Failure Modes

1. The Thinking Loop. The most severe failure mode. The model enters an infinite reasoning cycle — displaying "Thinking..." indefinitely or producing repetitive reasoning traces without ever calling a tool or producing a final answer. The root cause: the ReAct loop fails to reach a termination condition, often because a tool returns an ambiguous error that the model retries endlessly. The only fix is Ctrl+C and /clear to reset the poisoned context.

2. Tool execution regressions. The write_file tool has been reported to fail silently or crash after 2-3 write attempts in recent versions. An internal audit revealed that ESLint suppressions (@typescript-eslint/no-floating-promises) were masking race conditions in useGeminiStream.ts, allowing bugs to ship in release builds.

3. Prompt injection via GEMINI.md. The reliance on user-defined context files creates a vector for prompt injection attacks. Malicious instructions hidden in a project's GEMINI.md can trick the agent into executing shell commands. Claude Code's permission gates are the primary defense against this class of attack — Gemini's shouldConfirmExecute is the only barrier, and if a user approves reflexively, the system is compromised.

The Hybrid Strategy

The most productive setup I've found isn't choosing one tool. It's orchestrating both based on their strengths.

Inner loop (local development) → Claude Code. Planning, implementation, debugging, test-driven development. The permission gates catch mistakes before they hit the filesystem. Hooks enforce formatting and linting automatically. The premium UX — polished interface, thoughtful error messages, Shift+Tab to interrupt — makes the interactive session feel like pair programming with a senior engineer.

Outer loop (CI/CD, integration) → Gemini CLI. PR reviews via GitHub Actions. Security scanning via the /security:analyze extension. Deployment automation via the /deploy extension. The open-source, extensible architecture integrates naturally into pipeline workflows where human-in-the-loop approval happens at the PR level, not at every tool call.

The most sophisticated version of this — which I haven't built yet but the architecture supports — is a multi-agent pipeline:

1. Bug reported (GitHub issue)
   → Gemini CLI GitHub Action triages, labels, assigns

2. Reproduction
   → Claude Code reads the issue, writes a failing test, commits to a branch

3. Context enrichment
   → Gemini CLI + custom MCP server queries production logs for related errors

4. Fix
   → Claude Code writes the fix, runs the test suite, creates a PR

5. Review
   → Gemini CLI GitHub Action posts automated review comments

6. Merge + Deploy
   → Human approves → Gemini CLI /deploy extension ships to production

This isn't hypothetical architecture. Every component exists today. The missing piece is the orchestration layer that chains them — and that's a bash script, a GitHub Actions workflow, or a purpose-built coordinator.

The Choice Isn't Binary

Claude Code and Gemini CLI aren't competing products in the way VS Code competes with JetBrains. They're competing philosophies about how much autonomy an AI agent should have in a developer's terminal.

Claude Code's bet: models aren't reliable enough yet. Gate everything. Let the developer inject deterministic behavior at every boundary. Trade speed for safety. The Harness Inversion points inward — the developer shapes the agent's behavior through constraints.

Gemini CLI's bet: models are reliable enough to choose their own tools. Equip the agent with capabilities and let it reason. Trade safety for speed. The Harness Inversion points outward — the developer shapes the agent's behavior by controlling what it can access.

Both bets will be validated by the same thing: how fast the underlying models improve. If models get dramatically more reliable in the next 12 months, Gemini's autonomous approach wins — permission gates become unnecessary friction. If models plateau in reliability, Claude's gated approach wins — deterministic guardrails remain essential infrastructure.

==The question isn't which tool is better. It's which failure mode you can tolerate: an agent that moves slowly because it asks too many questions, or an agent that moves fast and occasionally breaks things you didn't expect.==

My answer is both. Different tools for different failure costs. The agentic terminal isn't a single tool — it's a toolkit.

Sharad Jain is an AI engineer and the author of The 14K Token Debt, The Terminal Was the First Agent Harness, and Your MCP Servers Are Costing You 10 Seconds. He writes about agent architecture, system prompts, and the infrastructure decisions that compound across every session. This is the fourth post in a series on the hidden mechanics of agentic AI systems.

Your MCP Servers Are Costing You 10 Seconds Per Session

Sharad Jain — Mon, 20 Apr 2026 14:00:00 GMT

Your MCP Servers Are Costing You 10 Seconds Per Session

I had nine MCP servers configured globally in Claude Code. Chrome DevTools. Browser automation. A financial data connector. Jules for async delegation. Gmail. Gemini. Reddit. Sequential thinking. A custom brain server for searching past sessions.

Most were useful — sometimes. A few were essential. But all nine booted every time I typed claude and hit enter.

One afternoon I timed it. From keystroke to cursor: 11 seconds. I disabled six servers I didn't need for my current project. 3 seconds. Same machine, same model, same context. The only variable was the invisible infrastructure spinning up behind the prompt.

That 8-second gap is the MCP tax. And it's the least expensive part.

What Happens When You Type `claude`

Before you see your first cursor, Claude Code executes a boot sequence most users never think about:

1. Parse ~/.claude/settings.json
2. For each configured MCP server:
   a. Spawn a subprocess (stdio transport)
   b. Wait for the server to initialize
   c. Request tool definitions (listTools)
   d. Collect JSON schemas for every tool
3. Inject ALL tool schemas into the system context
4. Load CLAUDE.md files (global → project → subdirectory)
5. Load memory files
6. Ready.

Steps 2a-2d happen for every server, concurrently. Servers initialize in parallel, but the session blocks until the slowest one finishes — your boot time equals MAX(server_boot_times), not the sum. In theory, nine servers shouldn't be much slower than one. In practice, they are: each subprocess competes for CPU, disk I/O, and network (npm registry calls). On my M1 Pro, one npx -y server boots in 3 seconds. Three boot in 4. Nine boot in 11 — not because they're sequential, but because resource contention turns parallelism into a bottleneck. One cold npx -y call stalls the event loop while npm resolves dependencies; nine of them thrash the disk cache simultaneously.

This is the operational cost. But there's a larger, invisible cost that persists after boot.

The Two Taxes

Tax 1: Startup Latency

Every MCP server is a subprocess. Claude Code spawns each one, establishes a JSON-RPC connection over stdio, and waits for it to report its capabilities. The latency depends on the server's runtime:

Server Type	Cold Start	Warm Start	What Causes It
`npx -y @package/server`	3-5s	0.5-1s	npm downloads the package if not cached
Python (`uv run` / venv)	1-3s	0.3-0.5s	Virtual environment activation, dependency resolution
Native binary (`qmd mcp`)	0.1-0.3s	0.1s	Just a fork+exec
Local Node.js (pre-installed)	0.3-0.5s	0.2s	V8 startup + module loading

The npx -y pattern is the worst offender. It's the default in most MCP server installation guides — "just add npx -y @modelcontextprotocol/server-whatever to your config." What those guides don't mention: npx -y checks the npm registry on every invocation. If the package isn't in your local cache, it downloads it. If npm is slow, your session is slow. If you're offline, the server fails entirely. Independent measurements show npx-launched MCP servers taking 10-15 seconds to start — turning a 5-second agent startup into a 30-second wait. The Gemini CLI hit the same problem: 8-12 seconds blocked on MCP server initialization.

The fix is embarrassingly simple: install the package globally once.

# Instead of relying on npx -y every session:
npm install -g @anthropic-ai/claude-code-mcp-server

# Then in settings.json, use the binary directly:
"command": "claude-code-mcp-server"
# Instead of:
"command": "npx", "args": ["-y", "@anthropic-ai/claude-code-mcp-server"]

The runtime language matters too. Multi-language benchmarks show stark differences in MCP server performance over stdio:

Runtime	Avg Latency	Memory	Throughput
Go	0.855ms	18 MB	1,624 RPS
Java	0.835ms	226 MB	1,624 RPS
Node.js	10.66ms	110 MB	559 RPS
Python	26.45ms	98 MB	292 RPS

Go servers use 6x less memory than Node.js and respond 12x faster. If you're choosing between MCP server implementations, the runtime isn't just a preference — it's a performance multiplier.

In theory, startup latency should plateau — parallel servers boot simultaneously. In practice, it scales sub-linearly but significantly: resource contention between concurrent subprocesses means each additional server degrades the boot time of every other server. My measurements: 1 server = 1s, 3 servers = 4s, 9 servers = 11s. Fewer servers don't just save their own boot time — they give every remaining server more headroom to initialize faster.

Tax 2: Schema Gravity

This is the tax nobody sees.

In The 14K Token Debt, I introduced the concept of Prompt Gravity — the observation that your system prompt's initial tokens create an attentional bias that shapes every subsequent generation. The heavier the prompt, the stronger the gravitational pull on the model's reasoning.

MCP tool schemas exert the same force. I call this Schema Gravity: ==the invisible weight of tool definitions pulling on your context window and the model's attention before any reasoning begins.==

The scale of the problem isn't speculative — Anthropic measured it themselves. Their engineering blog on advanced tool use reports the numbers:

Configuration	Token Overhead	Source
GitHub MCP server alone (35 tools)	~26,000 tokens	Anthropic Engineering
5 servers (GitHub + Slack + Sentry + Grafana + Splunk)	~55,000 tokens	Anthropic Engineering
Anthropic's internal setup before optimization	134,000 tokens	Anthropic Engineering
11 servers, 137 tools (community measurement)	27,462 tokens	MCP Spec Issue #1576
4 MCP servers, query: "What's 2+2?"	~15,000 tokens (~$0.21)	Claude Code Issue #3406

Read that last row again. Asking Claude Code "What's 2+2?" with four MCP servers configured costs $0.21 in token overhead — 14,000 of the 15,000 tokens consumed were wasted on tool schemas for a query that needed zero tools.

Annualize it: a power user running 30 sessions per day with $0.21 of schema overhead per session is burning $2,300/year in wasted tokens. A team of 10 engineers? $23,000/year — on tool definitions nobody reads.

A single MCP server can inject 26,000 tokens of schema definitions. Five servers? You're burning 55,000+ tokens. Anthropic's own team hit 134,000 tokens — roughly 67% of a 200K context window — consumed by tool definitions alone. Before the model reads your CLAUDE.md. Before it loads your memory. Before it processes a single word of your actual request.

This is Schema Gravity in action. Those tool definitions compete directly with your conversation for the model's attention. The transformer's self-attention cost is O(n²) in sequence length — extra tokens don't add linearly, they compound. A 200K context with 134K tokens of schemas has attention patterns spread across 65% noise before your actual question is even processed. This is the "lost in the middle" effect applied to tool definitions: the model's ability to retrieve and reason about relevant context degrades as the total context grows, and schemas push everything else further from the attention hotspots at the beginning and end of the window.

One caveat: modern API deployments use prompt caching. If your MCP schemas are identical across sessions — which they typically are — the cached prefix means you pay full input cost only on the first turn. Subsequent turns hit the cache and the dollar cost drops significantly. But the attention cost doesn't. Cached or not, those 55K tokens of tool definitions still occupy the context window and compete with your conversation for the model's reasoning bandwidth. Schema Gravity is an attention problem, not just a billing problem.

The irony is sharp: tools designed to make the agent more capable can make it measurably worse by consuming the context it needs to think.

Tax 2b: Tool Selection Noise

Schema Gravity's second-order effect is tool selection noise. When you configure nine MCP servers, Claude sees 50-200 tool definitions at conversation start. For any given task, you need 3-5 tools. The rest is decision overhead.

This is the $PATH problem. Every Unix system has a $PATH with thousands of executables. Ambiguous names resolve to the wrong binary. For agents, more tools means more potential for wrong tool selection (choosing browser automation when grep would suffice), tool hallucination (inventing calls that combine features from multiple schemas), and wasted reasoning tokens evaluating irrelevant options.

Anthropic's own data confirms this: before Tool Search, tool selection accuracy on Opus 4 was just 49% — worse than a coin flip. After deferring unused schemas, accuracy jumped to 74%. Fewer visible tools = better tool choices.

The Claude Code documentation acknowledges this directly: CLI tools like gh, aws, and gcloud don't add schema injection because the model already knows them from pre-training. The model's training data is the documentation. You get the tool for free — no schema, no startup, no context cost.

The Audit: Do This Right Now

Open your Claude Code settings:

cat ~/.claude/settings.json | grep -A2 '"mcpServers"'
# Or just: /mcp inside a Claude Code session

Count your servers. For each one, ask:

Question	If Yes	If No
Do I use this server in every project?	Keep as global	Move to project-level
Could I use a CLI tool instead?	Remove the MCP server	Keep
Does it need to maintain state across calls?	MCP is justified	CLI might be better
Is it an `npx -y` server?	Install globally	Already optimized

Here's what my audit looked like:

Server	Verdict	Reason
`brain` (qmd)	Keep global	Cross-project memory — used every session
`sequential-thinking`	Keep global	Used for planning in most sessions
`chrome-devtools`	Move to project	Only needed for frontend/QA work
`browser-mcp`	Move to project	Same — frontend-only
`fi_mcp`	Move to project	Financial data — one project only
`jules`	Move to project	Agent delegation — specific workflows only
`gmail`	Remove	`gws gmail` CLI works better
`gemini`	Remove	Rarely used, CLI alternative exists
`reddit`	Remove	Used once, never again

Result: 9 global servers → 2 global + 4 project-scoped + 3 removed. Boot time dropped from 11 seconds to 3. Context overhead dropped by roughly 70%.

The Progressive Disclosure Fix

The solution isn't "fewer MCP servers." It's the right servers at the right time. This is progressive disclosure applied to infrastructure: start with the minimum, load more on demand.

1. Project-Level Scoping

Move servers from ~/.claude/settings.json (global) to your project's .claude/settings.json:

// In ~/your-project/.claude/settings.json
{
  "mcpServers": {
    "chrome-devtools": {
      "command": "chrome-devtools-mcp",
      "args": ["--port", "9222"]
    }
  }
}

This server only boots when you run claude inside that project directory. Every other project skips it entirely.

2. Tool Search (Deferred Loading)

Claude Code supports deferred tool loading — MCP servers whose tools are registered by name but whose schemas aren't loaded until first use. The schema is fetched on-demand via ToolSearch only when the tool is actually needed.

The impact is dramatic. Anthropic's own measurements show:

Metric	Before Tool Search	After Tool Search	Improvement
Token overhead (5-server setup)	~77,000 tokens	~8,700 tokens	89% reduction
Available context window	122,800 tokens	191,300 tokens	+56% recovered
Tool selection accuracy (Opus 4)	49%	74%	+25 percentage points

Tool Search auto-activates when your MCP tool descriptions exceed 10% of the context window. No manual configuration required — though you can force it with the ENABLE_TOOL_SEARCH environment variable.

One gotcha: the defer_loading field in .claude.json's MCP server config is silently accepted but has no effect. Tool Search is all-or-nothing, not per-server. If you need more granular control, the community-built lazy-mcp proxy achieves ~95% context reduction (15K to ~800 tokens for 30 tools) with ~500ms first-call latency.

3. Pre-Install npx Packages

The single highest-ROI fix. For every MCP server using npx -y:

# Find all npx-based servers in your config
grep -A1 '"npx"' ~/.claude/settings.json

# Install each one globally
npm install -g @anthropic-ai/server-sequential-thinking
npm install -g @anthropic-ai/server-browser

# Update settings.json to use the binary directly
# Before: "command": "npx", "args": ["-y", "@anthropic-ai/server-sequential-thinking"]
# After:  "command": "server-sequential-thinking"

Cold starts drop from 3-5 seconds to 0.3-0.5 seconds per server.

4. The 3-Server Rule

A heuristic I've settled on: ==if you have more than 3 global MCP servers, at least one should be project-scoped instead.== Three is roughly the breakpoint where startup latency stays under 3 seconds and schema injection stays under 100K tokens. Above that, you're paying a tax that compounds across every session.

The exceptions prove the rule. My two global survivors — brain (cross-project memory) and sequential-thinking (used in 80%+ of sessions) — both meet the bar: genuinely used across all projects, not replaceable by a CLI tool, and fast to boot (native binary and pre-installed Node respectively).

When MCP Is Worth the Cost

I've spent this entire post arguing that MCP servers are expensive. They are. But they're expensive the way a database connection pool is expensive — the overhead is justified when you need what they provide.

MCP is worth it when:

The server maintains state. A browser session, a database connection, a running development server — these can't be replaced by stateless CLI calls. Each mcp__chrome-devtools__evaluate_script call operates on the same browser tab. A CLI equivalent would need to reconnect on every invocation.
The tool needs structured input/output. MCP's JSON-RPC protocol provides typed tool schemas — parameter validation, required fields, enum constraints. CLI tools accept string arguments. For complex operations (multi-field form submission, structured queries), MCP's type safety prevents silent failures.
The workflow is multi-turn. Sequential thinking, where thought N informs thought N+1 through server-side state, requires a persistent connection. You can't pipe state between independent CLI calls without an explicit persistence layer.
Discovery matters more than efficiency. When exploring unfamiliar capabilities — a new API, an internal platform — MCP's self-documenting schemas help the agent understand what's available. The token cost is the price of exploration. Once you know which tools you need, you can evaluate whether a CLI alternative exists.

Where This Breaks

Lazy loading shifts the cost, it doesn't eliminate it. Deferred tool schemas still get injected on first use. If you need the chrome-devtools server halfway through a conversation, you pay the schema tax then — potentially disrupting a long reasoning chain with a sudden injection of tool definitions.

Timing is approximate. The startup latency numbers in this post are from my setup (M1 Pro, macOS, Homebrew-installed tools). Your mileage will vary based on hardware, network speed (for npx), and which servers you're running. The ratios hold — npx is always slower than pre-installed, more servers always cost more than fewer — but the absolute numbers are directional.

Some servers resist project-scoping. If you use Claude Code across 10 projects and 7 of them need the same MCP server, project-scoping means maintaining 7 copies of the same config. At that point, global is the right default and the startup tax is the cost of doing business.

The token overhead exists regardless of startup timing. Even if every server boots instantly (native binaries, pre-installed packages), the schema injection cost remains. A server with 30 tool definitions injects 30 schemas into context whether it took 0.1 seconds or 5 seconds to boot. Startup optimization and schema optimization are independent problems.

Where This Is Heading

This is a transitional problem. The trajectory is clear: manual MCP configuration will feel as archaic as manual memory management within two years.

Tool Search is the embryo of an intelligent loader — it already reduced schema overhead by 89% at Anthropic. The next step is context-aware loading: the system learns which tools you need for which projects and loads them automatically. Beyond that, the MCP ecosystem itself will face selection pressure. Bloated servers with 35 tool definitions will lose to focused servers with 5 tools each — the same specialization pressure that turned monolithic Unix utilities into composable single-purpose commands.

But today, in April 2026, the intelligent loader doesn't exist yet. Today, you're the loader. And the audit in this post is how you do the job until the system learns to do it for you.

The Fastest Agent Boots With Exactly What It Needs

The MCP ecosystem is growing fast. As of mid-2026, there are thousands of MCP servers available — everything from Slack to Postgres to Figma to Kubernetes. Anthropic's own Tool Search system supports catalogs of up to 10,000 tools. The temptation is to configure all of them. The result is an agent that boots slowly, thinks through a fog of irrelevant tool definitions, and burns context window on capabilities it doesn't need for the task at hand.

The fix is the same principle that makes Unix powerful: do one thing well. Not "install every tool in case you need it," but "install exactly what this task requires."

# The anti-pattern:
~/.claude/settings.json → 9 global servers → 11 second boot → 200K tokens of schemas

# The pattern:
~/.claude/settings.json → 2 global servers (brain, sequential-thinking)
project/.claude/settings.json → 2 project servers (chrome-devtools, browser)
→ 3 second boot → 60K tokens of schemas
→ Context window recovered for actual reasoning

My sessions now boot in 3 seconds. My context window recovered 120,000 tokens. My agent's tool selection accuracy jumped from coin-flip to reliable. The only thing I lost was infrastructure I wasn't using.

At 20 sessions per day, the 8 seconds I recovered per session add up to 16 hours per year of pure wait time eliminated. The 120K tokens recovered per session mean my agent can reason about larger codebases, hold longer conversations, and make fewer tool selection errors — every single session.

Your MCP servers aren't free. Measure them. Audit them. Scope them.

Sharad Jain is an AI engineer and the author of The 14K Token Debt and The Terminal Was the First Agent Harness. He writes about agent architecture, system prompts, and the compounding returns of treating infrastructure as a first-class engineering discipline. This post is the third in a series on the hidden costs of agentic AI systems.

The Terminal Was the First Agent Harness

Sharad Jain — Thu, 16 Apr 2026 14:00:00 GMT

The Terminal Was the First Agent Harness

When DeepMind benchmarked Gemini-2.5-Flash on the Kaggle chess GameArena, 78% of the model's losses were illegal moves. Not strategic blunders. Not positional misunderstandings. Rule violations. The model's reasoning was often sound. Its outputs broke the game.

The fix wasn't a bigger model. Gemini-2.5-Flash with an auto-generated code harness — a deterministic wrapper that validated moves before submitting them — consistently beat Gemini-2.5-Pro and GPT-5.2 running raw. A $0.15/request model with guardrails outperformed $200B models without them.

That harness? Structurally, it's a shell script.

while read -r task; do
  plan=$(think "$task")
  result=$(echo "$plan" | act)
  echo "$result" | observe >> memory.log
done < tasks.txt

Read input. Reason. Act. Observe. Persist. Loop. This is the ReAct pattern — the paradigm powering virtually every modern AI agent. Yao et al. (2023) formalized it in 2023. But replace think, act, and observe with real programs and this script runs unchanged on any Unix system since 1979.

==Unix solved the agent problem 50 years ago. The primitives that make modern AI agents work — tool use, context management, harness reliability, persistent memory — map 1:1 to abstractions that already exist in the terminal.== What's genuinely new about the agent era is narrower than most people think.

I'm going to make this case across six parallel mappings, then break it — because the honest version requires acknowledging exactly where the analogy fails.

"Everything is a File" = "Everything is Context"

The most provocative recent paper in agent architecture isn't about transformers or reinforcement learning. It's about filesystems.

Xu et al. (2025) propose what they call an agentic file system — a unified namespace where all context sources an agent needs (memory, tools, APIs, knowledge bases, human input) are "mounted" and accessed through a single hierarchical interface. Their inspiration is explicit:

"Inspired by the Unix notion that everything is a file, the abstraction provides a persistent, hierarchical, and governed environment where heterogeneous context sources are mounted and accessed uniformly."

This isn't metaphor. It's direct architectural lineage. The Unix filesystem already solves the problem of presenting wildly different data sources through a single interface. /dev/sda is a physical disk. /proc/cpuinfo is a kernel data structure. /dev/null is a void. But to any program reading them, they're all just files. Open, read, close — the same three syscalls regardless of what's behind the path.

The "Everything is Context" paper applies this pattern to agents. Their framework, AIGNE, mounts heterogeneous context sources into a namespace that agents browse like a directory tree:

Unix Mount Point	Agent Context Equivalent	What's Mounted
`/dev/`	`/context/tools/`	Tool capabilities (APIs, functions, MCP servers)
`/proc/`	`/context/state/`	Runtime state (current task, agent status, metrics)
`/mnt/`	`/context/knowledge/`	External knowledge bases, RAG sources, documents
`/home/`	`/context/memory/`	Agent-specific persistent memory
`/tmp/`	`/context/pad/`	Transient scratchpad for in-flight reasoning
`/var/log/`	`/context/history/`	Immutable interaction history

I call this pattern Context Mounting — the principle that heterogeneous data sources should be projected into a uniform namespace the agent can browse, exactly like Unix mounts devices into /dev, /proc, and /mnt. The power isn't in any individual mount point. It's in the uniformity: once everything is mounted, the agent doesn't need specialized code for each data source. It reads context the same way regardless of whether that context comes from a vector database, a REST API, or a local file.

The paper goes further: each mounted node can have meta-defined actions — callable behaviors discoverable by agents. A file isn't just data; it's an active node that can execute tools, transformations, or service calls directly through the filesystem interface. This is /dev/ on steroids — Unix gave us device files that could be written to for side effects; AIGNE extends this to arbitrary tool invocations through the same read/write interface.

Context Mounting has a security corollary that most agent architectures ignore entirely. Bell Labs' Plan 9 operating system extended "everything is a file" to its logical conclusion: each process gets a synthesized, per-process namespace where only authorized capabilities exist as filesystem paths. If the billing database isn't mounted in your namespace, you can't access it — not because a prompt told you not to, but because the path literally doesn't exist in your view of the filesystem. Traditional agent security relies on prompt engineering ("don't access the billing API") or middleware token scopes. Namespace-bounded security relies on the kernel. The former is breakable by jailbreaking. The latter is enforced at the hardware level. This is chroot applied to agent capabilities: security by construction, not by instruction.

Where the mount leaks. Unix files are byte streams. They have no schema, no types, no semantic structure. When you cat /proc/cpuinfo, you get plain text that you parse with grep and awk. Agent context needs schema-driven mounting — REST/OpenAPI resources, GraphQL types, and MCP tools auto-projected into the namespace with machine-readable type definitions. Xu et al. address this with schema-driven mounting, but it's an extension Unix never needed because Unix programs are deterministic consumers of structured data. Agents are stochastic consumers of unstructured meaning. The abstraction holds for structure; it leaks at semantics.

Bash as the Original Agent Loop

The ReAct paradigm — the most widely adopted agent architecture — interleaves "reasoning traces" and "task-specific actions" in a loop. As Yao et al. (2023) describe it, this mirrors how "humans naturally alternate between thinking and acting during complex tasks."

Humans have been doing this in terminals for decades. Here's ReAct, expressed formally:

Agent Loop:
  1. Observe(environment) → context
  2. Think(context) → plan
  3. Act(plan) → tool_call
  4. Observe(tool_result) → updated_context
  5. If not done: goto 1

And here's the bash equivalent that every sysadmin has written:

#!/bin/bash
set -e  # The original guardrail: exit on any error

while IFS= read -r task; do
  # Think: analyze the task
  plan=$(analyze_task "$task")

  # Act: execute the plan
  result=$(execute "$plan" 2>&1)
  exit_code=$?

  # Observe: check the result
  if [ $exit_code -ne 0 ]; then
    echo "[ERROR] $task failed: $result" >> errors.log
    continue  # Self-correction: skip and move on
  fi

  echo "[DONE] $task: $result" >> activity.log
done < tasks.txt

The structural mapping is exact:

ReAct Component	Bash Equivalent	Unix Primitive
Observation	`read` from stdin/file	File descriptors
Reasoning trace	Comments, variable assignment	Shell variables
Action	Command execution	`exec` / `fork`
Tool result	stdout/stderr capture	Pipes, `$()`
Error handling	`$?` exit code + `set -e`	Process exit codes
Memory	`>> activity.log`	File append
Loop control	`while`/`until`	Shell control flow

The pattern difference is the medium, not the mechanism. A bash loop processes structured commands; a ReAct loop processes natural language. But the control flow — observe, decide, act, check, persist, loop — is identical.

set -e deserves special attention. It's the original harness guardrail: the single line that transforms a script from "keep going regardless of failures" to "stop the moment something goes wrong." Every agent framework reinvents this — max_retries, error callbacks, task failure handlers — but they're all set -e with extra steps.

The limit. set -e catches exit codes — binary success or failure. It can't catch a model that confidently returns the wrong answer with exit code 0. Deterministic programs fail loudly; stochastic programs fail silently. This is where bash-as-agent-loop hits its ceiling, and we'll come back to it.

The Harness Is a Shell Script

In March 2026, Pan et al. published a paper that made an argument so obvious it's surprising nobody formalized it sooner: the harness — the deterministic code layer wrapped around an LLM — should be treated as a first-class, executable artifact. Not scattered across controller code, hidden framework defaults, and tool adapters. A single, portable, inspectable document.

They call these Natural-Language Agent Harnesses (NLAHs). Unix already has a name for them: shell scripts.

The evidence for why harnesses matter is stark. When DeepMind tested Gemini-2.5-Flash on the Kaggle chess GameArena, 78% of the model's losses were caused by illegal moves — not strategic blunders, not positional misunderstandings, but rule violations. The model's reasoning was often sound. Its outputs violated the constraints of the game.

This is the software equivalent of a script that calculates the right answer but writes it to the wrong file. The logic isn't the problem. The harness is.

The data gets more compelling: Gemini-2.5-Flash with an auto-generated harness consistently beat Gemini-2.5-Pro and GPT-5.2 without one. A smaller, cheaper model with deterministic guardrails outperformed larger, more expensive models running raw. ==This is the principle I call The Determinism Dividend: every piece of agent behavior you can move from stochastic (LLM-generated) to deterministic (code-enforced) is a compounding reliability gain.==

┌─────────────────────────────────┐
│   Model (reasoning capability)  │  ← Gets all the attention
├─────────────────────────────────┤
│   Harness (control logic)       │  ← Where reliability lives
├─────────────────────────────────┤
│   Runtime (shell / OS / infra)  │  ← The forgotten foundation
└─────────────────────────────────┘

==This is The Harness Hierarchy: agent reliability is determined by three layers — model capability, harness logic, and runtime environment — and the layers below always constrain the layers above.== You can have the most capable model in the world, but if your harness doesn't validate outputs before they reach the environment, you'll lose 78% of your games to illegal moves.

The Auton framework (Cao et al., 2026) takes this further with what they call a Constraint Manifold — a formally defined subspace of the action space onto which the agent's policy is projected before action emission. Privilege escalation and unsafe operations are excluded by construction, not detected after the fact. In Unix terms: this is chroot for agents. You don't give the process access to / and hope it behaves; you restrict its filesystem view to only what it needs.

The NLAH paper's core complaint — that harness logic is "scattered across controller code, hidden framework defaults, tool adapters" — is the same complaint every ops engineer has made about undocumented production scripts. Unix solved the portability problem by making shell scripts the standard packaging format for automation. Agent frameworks are slowly rediscovering that the harness should be a single readable artifact, not a fog of implicit configuration.

Tool Use = Unix Pipes

cat data.csv | sort -k2 -t',' -rn | head -10 | awk -F',' '{print $1, $3}'

This pipeline does four things: read data, sort by a column, take the top 10, extract specific fields. Four small programs, each doing one thing well, composed via pipes into something none of them could do alone.

Modern agent tool use is the same pattern. When AnyTool organizes 16,000+ APIs into a hierarchical retrieval system — category → tool → API — it's reinventing the directory structure. /usr/bin/ holds general utilities. /usr/local/bin/ holds user-installed tools. $PATH determines search order. AnyTool's hierarchical API retriever is $PATH with a semantic index.

The parallels extend to interface design:

Agent Concept	Unix Equivalent
Tool description (function schema)	`man` pages + `--help` flags
Tool discovery	`which` / `whereis` / `$PATH` search
Tool invocation	`exec` with arguments
Tool chaining	Pipes (`\|`)
Tool output parsing	`stdout` capture + `jq`/`awk`/`sed`
MCP (Model Context Protocol)	Pipes with JSON instead of plain text

MCP, Anthropic's protocol for connecting agents to tools, is structurally a modernized Unix pipe — stdio transport, streaming data between producer and consumer. But MCP carries JSON-RPC with typed tool schemas and capability negotiation, and that typing comes at a cost.

Independent benchmarks reveal a stark token economics gap between MCP and CLI-based tool use:

Metric	MCP	CLI	Advantage
Tool schema injection	~28,000-55,000 tokens	0 tokens	CLI uses innate model knowledge
Total task consumption (50 objects)	~145,000 tokens	~4,150 tokens	CLI is 35× more efficient
Task completion rate	60/100	77/100	CLI completes 28% more tasks

A standard MCP implementation can consume 55,000 tokens just to define tool schemas — before any reasoning begins. For agents connecting to multiple services (GitHub + Postgres + Jira), schema injection can exhaust over 150,000 tokens of the context window. CLI tools cost virtually zero schema tokens because the model already knows ls, grep, git, and docker from pre-training. The model's training data is the documentation.

This is where The Pipe Test becomes a measurable complexity detector, not just an analogy: ==if your agent workflow can't be described as a Unix pipeline (input | transform | validate | output), it's probably over-engineered — and almost certainly over-tokenized.== The 35× efficiency gap is the Determinism Dividend applied to tool invocation: deterministic CLI tools that the model already knows outperform typed schemas that must be injected fresh every session.

Where the Pipe Test breaks. Pipes are linear. Data flows in one direction. Agent workflows are frequently non-linear: conditional branching (if the search returns no results, try a different query), recursive decomposition (break a task into subtasks, each of which may spawn sub-subtasks), and backtracking (the plan failed at step 3, replan from step 1). Tree-of-thought architectures are fundamentally non-linear. The Pipe Test catches over-engineering, but it's a necessary condition for good design, not a sufficient one. Some genuinely complex workflows require graphs, not pipelines.

The Filesystem Is Agent Memory

Andrej Karpathy's LLM-as-compiler pattern — where raw source documents flow through an LLM to produce structured wiki pages, which are then served as knowledge — is a build system:

raw/           →    wiki/          →    output/
(source docs)       (compiled KB)       (served pages)

src/           →    build/         →    dist/
(source code)       (compiled)          (deployed)

The LLM acts as a compiler. Compilers already live in terminals. The Karpathy pipeline (raw/ → wiki/ → Obsidian) scales to ~100 articles and ~400,000 words without vector databases — proof that filesystem-based knowledge management works at non-trivial scale.

The deeper mapping is the memory taxonomy. The "Everything is Context" paper defines five categories of agent memory. Every one maps to a Unix location that already exists:

Agent Memory Type	Unix Equivalent	What's Stored	Lifecycle
Scratchpad (transient working memory)	`/tmp/`	In-flight reasoning, intermediate results	Cleared on reboot
Episodic (session history)	`~/.bash_history`	What happened in this session	Append-only, bounded
Fact memory (persistent knowledge)	`~/.config/`, dotfiles	User preferences, API keys, learned facts	Long-lived, mutable
Procedural (how-to knowledge)	Scripts in `$PATH`	Reusable procedures, workflows, recipes	Versioned, executable
Historical record (audit trail)	`/var/log/`	Complete interaction history	Immutable, rotated

But the filesystem isn't just agent memory — it's the agent's diagnostic nervous system. The /proc virtual filesystem exposes every process, network connection, and kernel state as a readable file. An agent debugging a failed deployment doesn't need to guess from an opaque HTTP 500 error. It can read /proc/self/status for its own memory footprint, run strace -p <PID> -f -e trace=network,file to watch exactly which syscalls a hanging process attempts, or check lsof -i :443 to diagnose connection failures at the socket level. API agents debug abstractions. Shell agents debug reality. This is the Determinism Dividend applied to observability: deterministic diagnostic tools yield deterministic root causes.

Where the filesystem metaphor breaks. grep finds strings. Agents need meaning. When I search my knowledge base for "reducing inference costs," I need to find documents about "token optimization," "model distillation," and "quantization" — none of which contain the literal search terms. This requires dense retrieval — learned embeddings where semantic similarity is captured through vector proximity, not character matching. Zep demonstrates the state of the art: triple-method retrieval (cosine similarity + BM25 + graph traversal) with cross-encoder reranking, achieving 18.5% accuracy improvement over full-context baselines while reducing context tokens from ~115K to ~1.6K per query.

More fundamentally, filesystem memory is explicit — you choose what to save and where. Agent memory needs experience abstraction: not just recalling what happened, but distilling what was learned. As Evo-Memory (Google DeepMind, 2025) argues, most memory systems only reuse static dialogue context rather than learning from experience to improve future reasoning. An agent that only remembers conversations is like a developer who reads their bash history but never writes reusable scripts. The jump from episodic memory to procedural memory — from "what happened" to "what I learned" — is where the filesystem analogy is necessary but not sufficient.

Claude Code as the Existence Proof

If the terminal-as-agent-harness argument feels theoretical, there's a production system that embodies it: Claude Code.

Claude Code runs inside a terminal. Its primary tools are bash, filesystem reads/writes, and MCP servers. When it bootstraps a project, it runs mkdir, cd, and python main.py — the agent bootstrap sequence is bash. The system prompt (which I dissected in The 14K Token Debt) is the most consequential architectural decision in the system. This post extends that argument: if the system prompt is the agent's constitution, the terminal is the agent's runtime.

Building a Claude Code clone from scratch reveals the architecture: an agentic loop that reads tool calls from the model, executes them via subprocess (shell commands, file operations), captures stdout/stderr, and feeds the results back as context. The "agent framework" is ~200 lines of Python wrapping a shell. The tool interface is subprocess.run(). The persistence layer is the filesystem. The harness is a loop with error handling.

The terminal isn't a convenience layer. It's the natural runtime for an agent that coordinates system resources through a text interface — exactly the workflow Unix was designed for.

Where the Analogy Breaks: What's Actually New

I've spent five sections arguing that Unix anticipated the agent paradigm. Now let me break my own argument, because the honest version requires acknowledging three things agents need that Unix fundamentally cannot provide.

1. Stochastic Output Handling

Unix programs are deterministic. sort file.txt produces the same output every time for the same input. Agent outputs are stochastic — the same prompt can produce different tool calls, different reasoning chains, and different conclusions on every run.

The Auton framework (Cao et al., 2026) calls this the Integration Paradox: "LLMs produce stochastic, unstructured outputs, whereas the backend infrastructure they must control — databases, APIs, cloud services — requires deterministic, schema-conformant inputs." Every Unix composition primitive (pipes, scripts, make) assumes deterministic components. When you pipe sort into uniq, you know what you'll get. When you chain an LLM's output into a database write, you don't.

This is the gap that harnesses fill. But it's a gap Unix never had to bridge, because Unix's "agents" (programs) were deterministic by design.

2. Semantic Understanding

grep matches character patterns. It cannot find "reducing expenses" when you search for "cost optimization." Agent memory requires semantic retrieval — the ability to find documents by meaning, not just by string.

This breaks the filesystem metaphor at a fundamental level. Unix's power comes from the composability of text-processing tools: grep | sort | awk | sed can answer almost any question about structured text. But agents operate on unstructured meaning. The Zep memory system's triple-method retrieval (vector similarity + keyword matching + knowledge graph traversal) exists because no single Unix primitive can capture semantic relationships. You need learned embeddings for similarity, BM25 for precision, and graph structure for multi-hop causal reasoning — all simultaneously.

3. Planning Under Uncertainty

A shell script follows a fixed control flow: step 1, step 2, step 3. If step 2 fails, you get an exit code and maybe a retry. Agents must plan, observe intermediate results, and replan when observations don't match expectations.

HiPlan (2025) demonstrates the gap: agents need hierarchical planning with global milestone guides and local step-wise hints generated dynamically at each timestep. The DAVIS framework goes further — an Actor-Critic architecture where the Critic monitors each step in real-time, "comparing observations to expectations and suggesting replanning when discrepancies arise." This is closer to a POMDP (Partially Observable Markov Decision Process) than a pipeline. Unix has no native primitive for "observe the output of step 2, decide whether to continue to step 3 or rewrite the plan entirely."

As the Auton framework puts it: the shift is "from imperative scripts to declarative definitions that specify agent behavior as auditable data; from stateless, single-session interactions to persistent cognitive architectures that accumulate experience." Shell scripts are imperative and stateless. Agents need to be declarative and stateful.

The 60/40 Split

==Here's my estimate: the Unix primitives give you roughly 60% of a working agent architecture for free.== Filesystem-as-memory, harness-as-script, pipes-as-tool-chains, the observe-act loop — these are real, load-bearing patterns, not loose analogies. The benchmarks support this: CLI agents complete 28% more tasks than MCP agents while consuming 35× fewer tokens. The terminal-native approach isn't just philosophically elegant — it's measurably superior for the majority of agent workloads. The remaining 40% — stochastic output handling, semantic understanding, hierarchical replanning — is what's genuinely novel about the agent era. And that 40% is where the hard engineering problems live.

Conclusion: Read the Man Pages

The next great agent framework won't be built by someone who studied only machine learning. It'll be built by someone who also understands operating systems.

Whether these patterns represent direct inheritance or independent convergence, the structural isomorphism is the point: when researchers independently arrive at "everything is a file" for agent context, when harness papers rediscover the Unix philosophy of small composable tools, when the most capable agent system in production literally runs inside a terminal — the design space for coordinating intelligent agents has fewer degrees of freedom than the framework proliferation suggests.

The four frameworks from this post — The Harness Hierarchy, The Determinism Dividend, Context Mounting, and The Pipe Test — are all Unix principles applied to stochastic computation. They won't cover the 40% that's genuinely new (non-determinism, semantics, replanning). But they'll prevent you from reinventing the 60% that's already solved. ==The agent ecosystem's biggest gap isn't better models or smarter prompts — it's systems engineering literacy.== The teams shipping reliable agents are the ones who read the man pages before reaching for the framework.

If you're building agent systems, try this: describe your architecture using only Unix concepts. If you can't, you might be solving a genuinely new problem. If you can, you might be reinventing cron.

Sharad Jain is an AI engineer leading AI at Autoscreen.ai. Previously at Meta and Autodesk. He writes about agent architecture, system prompts, and the compounding returns of treating software engineering as a first-class discipline in the age of AI. This post builds on The 14K Token Debt, which examined system prompt architecture as load-bearing infrastructure.

The 14K Token Debt: System Prompt Architecture for Agentic AI

Sharad Jain — Wed, 15 Apr 2026 14:00:00 GMT

The 14K Token Debt: Architecting System Prompts for Agentic AI

Abstract: In agentic workflows, the system prompt is rarely treated as load-bearing infrastructure. It is often relegated to unstructured boilerplate—a preamble injected before the execution of the actual task. This architectural oversight is the primary failure mode of long-running autonomous systems. Those initial tokens are not a preamble; they are a gravitational anchor. This technical report dissects the mechanics of "Prompt Gravity," the empirical limits of context retention, and the engineering frameworks required to transition from fragile stochastic instructions to deterministic, compounding behavioral architectures.

A production-grade system prompt typically consumes 14,000 tokens of the context window before the language model generates a single computational output.

Every conversation initializes with this expenditure, paid invisibly during the prefill phase. To contextualize the scale of this structural tax, consider the real compute cost of injecting a 14K-token payload just 100 times per day over a single year across foundation models:

Foundation Model	Input Cost (per 1M)	Cost per 14K Prompt	Annual Cost (100 runs/day)
Claude 3.5 Sonnet	$3.00	$0.042	$1,533
GPT-4o	$2.50	$0.035	$1,277
Claude 3 Opus	$15.00	$0.210	$7,665
Gemini 1.5 Pro	$1.25	$0.017	$638

Consequently, developers frequently process these preamble tokens as static overhead—ignoring their structural integrity. This is akin to ignoring your database schema because "the ORM handles it."

==After engineering autonomous systems scaling around frontier models in production environments, a localized architectural axiom has emerged: designing your system prompt is analogous to building your own compiler harness.== It forms the base distribution upon which every subsequent vector operation and attention mechanism is conditioned. If the foundational matrix is fragile, the agent pipeline is built on sand.

1. The Entropy Problem (Context Rot)

Without rigorous architectural boundaries, foundation models exhibit a well-documented trajectory in multi-turn environments: they begin executing reasonably, and then they drift.

Each sequential turn compounds context entropy. By turn eight, measurable cognitive degradation sets in. By turn fifteen, the model's logic has collapsed into the statistical median of its pre-training distribution—producing responses that are syntactically sound, but generically unhelpful for highly specialized domain workflows.

This decay is an architectural certainty, not an empirical anomaly. In their COLM 2024 paper, "Measuring and Controlling Instruction (In)Stability in Language Model Dialogs," Liang et al. tracked models like LLaMA-2-70B and GPT-3.5, concluding that instruction drift is universally measurable within just eight conversation rounds.

The mechanism resides within the attention layers of the transformer itself. As sequence length expands, the attention matrix disperses across an exponentially growing context matrix. The relative weight allocated to the initial system prompt inevitably shrinks. The directives do not vanish; they structurally dilute.

Visualizing the Attention Valley

This phenomenon maps directly to the findings of Liu et al. in Lost in the Middle (2023). Language models allocate maximal attention toward the extreme peripheries of their context window (the beginning constraints and the most recent user message), creating a pronounced "valley" of forgotten parameters.

Attention Weight Probability [Hypothetical Model]
1.00 │  ██
     │  ███
0.80 │  ████                    << System Prompt Focus (Architecturally Privileged)
     │  █████
0.60 │  ██████                            ███
     │  ███████                          ████
0.40 │  ████████                        █████
     │  █████████                      ██████
0.20 │  ██████████▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄████████  << "Lost in the Middle"
     │  ██████████████████████████████████████
0.00 └───────────────────────────────────────── 
          0K        50K       100K      150K      Current Query
                   Token Sequence Position

Mathematics is unforgiving. At a 100K context sequence anchored by a 14K system prompt, your directives command just 14% of the operational attention bandwidth. At a 200K sequence depth, that structural anchor comprises a mere 7%.

In coding agents specifically, Chroma's 2025 Study of Context Degradation identified a catastrophic 35-minute operational threshold. Beyond this horizon, degradation accelerates logarithmically, quadrupling pipeline failure rates regardless of the underlying foundation model.

Overcoming Escape Velocity: Prompt Gravity

We define the counter-force to this drift as Prompt Gravity: the mathematical capability of the system prompt to maintain the LLM's sequence continuations in strict, deterministic orbit around an intended engineering framework.

Without sufficient structural mass—defined tightly by absolute constraints, boundary-case definitions, and hardcode behavioral traces—the autonomous conversation reaches escape velocity, drifting inevitably toward the gravitational center of generic pre-training data.

==Those 14,000 upfront tokens are the pipeline's sole defense against entropy.==

2. Abstractions vs. Behavioral Traces

The fundamental liability of standard system prompts—heuristics like "Be concise," "Enforce strict typing," or "Think step-by-step"—is that they delegate contextual interpretation to the latent weights.

When you instruct a model to "be concise," the network interpolates that parameter from its trillion-token training regimen—effectively averaging millions of conflicting linguistic paradigms. Your localized definition of "concise" might strictly require "output only the functional bash script." The model's interpretation might yield "a well-structured descriptive paragraph omitting redundant functions." Only the former allows an automated CI/CD hook to execute successfully.

==Behavioral traces resolve this by eliminating semantic abstraction entirely.== Engineers must replace fuzzy, adjective-based instructions with serialized, deterministic evidence of what the orchestration layer actually accepted and rejected in prior states.

❌ The Instruction-Based Paradigm (High Variance)

<role>You are a senior infrastructure engineer.</role>
<directives>
  - Be concise. 
  - Write robust, production-ready code. 
  - Only provide the logical solution without conversational pleasantries.
</directives>

✅ The Behavioral-Trace Paradigm (High Gravity)

<behavioral_trace>
  [HISTORICAL POST-MORTEM - ACCEPTED PATTERN]
  Context: User requested a distributed Redis cache implementation in Python.
  Prior Run Outcome: You generated 180 lines containing redundant abstract classes.
  Orchestrator Action: The orchestration layer rejected all classes.

  [ACCEPTED_STRUCTURE]
  Here is the explicit sub-graph that was merged into production:
  - The core factory dictionary (12 lines, zero inline comments).
  - The raw pytest assertion block (8 lines, table-driven execution).
  
  [CONSTRAINT]
  When generating decoupled components, force strict conformance to this trace.
  ABORT the generation of abstract classes unless parameterized by user config.
</behavioral_trace>

The delta between these templates is not stylistic; it is the difference between a high-temperature stochastic roll and a deterministic, multi-turn state machine constraint. Meta’s research on semi-formal reasoning confirms this: structured tracking paths elevated code review pass rates from 78% on standard prompts to a rigorous 93% on production patches.

3. The Trifecta of Agent Memory Architecture

To synthesize where behavioral traces persistently reside, agent architectures must be cleanly deconstructed into three isolated functional layers.

Layer Protocol	Volatility & Scope	Primary Architectural Function	Core Data Primitives
Layer 3: Persistent	Global / Survives Restarts	Accumulating generalized operational experience	Knowledge Graphs, PRD JSONs, Semantic Embeddings
Layer 2: Session	Local / Wiped at Termination	Managing finite context for the immediate inference	Tool DAGs, transient file diffs, user transcripts
Layer 1: System	Bootstrapped at Initialization	Injecting immutable identity and routing constants	The 14K Token System Payload, Core Skills Matrices

While monolithic orchestration frameworks expend maximal computational overhead managing Layer 2 (via chunking and RAG context sliding), true frontier architecture relies on a dynamic compilation bridge connecting Layer 3 back to Layer 1.

==Instead of treating the system prompt as a static system.txt parameter file, it evolves into a dynamic bootstrapping payload forged by querying persistent memory precisely at the initialization lifecycle.== The system layer becomes an evolving organism.

4. Engineering the Truth: Thin Harness, Fat Skills

How is this theory engineered into a deterministic codebase? The optimal design architecture is captured in the "Thin Harness, Fat Skills" paradigm.

A high-performance orchestration harness strictly obeys a three-tier separation of concerns:

The Harness (Thin CLI Engine): ~200 lines of plumbing. It manages the LLM asynchronous execution loop, orchestrates host I/O permissions, and guarantees deterministic safety boundaries. It possesses zero domain awareness.
The Deterministic Core: Standard REST APIs, native shell validations, and AST compiler evaluations. LLMs lack native deterministic math functions; offload all binary computations here.
The Skills (Fat Markdowns Logic): The repository of intelligence. These are massive markdown or text payloads encoding domain heuristics, operational post-mortems, and the behavioral traces detailed above.

The Anti-Pattern: Fat APIs, Thin LLM Context

The architectural inverse involves mapping every conceivable internal capability directly to a strictly validated OpenAPI schema or a rigid Model Context Protocol (MCP) server.

The operational cost of this anti-pattern is devastating: heavy network round-trips for trivial logic paths, massive context windows flooded with unutilized schema metadata, and compounding latency. A UI verification step that natively resolves in 200 milliseconds via direct script injection can easily bloat to 15 seconds through an over-engineered external tool loop.

By hardcoding logic paradigms directly as "Fat Skills" inside markdown files, the model pre-computes the constraints during the initial prefill routing phase, executing complex reasoning frameworks via zero-latency "pseudo-method calls."

The Autonomous Hook (Implementation Blueprint)

The heart of the Ralph Loop or any autonomous execution is the iterative shell boundary that feeds the context parameters dynamically back into the CLI.

// The Thin Harness: Intercepting the Completion Promise
async function executeAutonomousLoop(prdConfigPath: string) {
  let isTaskResolved = false;
  let exitCode = 0;

  while (!isTaskResolved) {
    const contextPayload = compiler.buildDynamicPrompt(prdConfigPath);
    
    // Execute the Model
    const actionState = await agent.run({
      systemPrompt: contextPayload,
      cwd: localSandboxPath,
    });

    if (actionState.output.includes("<promise>COMPLETE</promise>")) {
      logger.info("Deterministic completion identified. Breaking loop.");
      isTaskResolved = true;
      exitCode = 0;
    } else {
      // Exit Code 2 triggers reinjection of the failure states into L3 Memory
      logger.error("System exited before completion criteria met. Re-evaluating.");
      await memoryStore.appendFailureTrace(actionState.stderr);
      exitCode = 2; // Hook reinstitutes the loop
    }
  }
  return exitCode;
}

5. Temporal Knowledge Graphs vs Traditional RAG

A traditional vector store operates effectively for documentation retrieval. However, conversational transcripts and operational history are fundamentally different; they map programmable behavior over time.

By indexing complete session artifacts (tool execution DAGs, accepted git diffs, and corrected parameter hallucinations) into a local Temporal Knowledge Graph — using hybrid retrieval tools like QMD — the agent natively queries its own operational baseline before executing the standard inference loop.

This architecture enables three synchronous retrieval vectors:

Lexical Resolution (BM25): Precise mapping of unhandled exception stack traces.
Semantic Weighting: Abstract logic mapping ("Where have we managed multi-tenant data migrations securely before?").
Hypothetical Document Embeddings (HyDE): Projecting an idealized state topography to resolve a novel code logic conflict.

==At the boot phase of Session N+1, the prompt logic inherently questions: "What specific validation pathway succeeded the last time we confronted this distinct edge case?"== The underlying transformer parameters remain completely frozen, yet the effective computational yield of the agent improves parabolically.

This establishes a profound mechanism for recursive self-optimization completely insulated within the parameter prompt boundary—circumventing the deep financial and infrastructure liabilities inherent in RLHF or continuous fine-tuning pipelines.

Engineering Trade-Offs & Failure Cascades

Applying intellectual rigor requires identifying systemic fault tolerances inherent in this loop system:

[!WARNING] Context Window Asphyxiation An 800-line harness scaled dynamically via autonomous retrieved history files can exceed 35,000 prefill tokens. Drowning the initial attention distribution with over-prescriptive operational history severely starves the localized attention block required for executing the real-time inference request. Strict token pruning is non-negotiable.

[!CAUTION] Data Rot & Recursive Pollution If an agent continuously appends its own hallucinations to the activity log, and human intervention fails to prune the guardrails.md, those faulty assertions become encoded as structural precedent. By iteration N+10, the agent begins utilizing its own failure cascades as validated procedural truths. Layer 3 indices require aggressive, database-level garbage collection.

[!NOTE] The Semantic Transferability Cliff A heavily optimized systemic prompt structurally encodes its creator's explicit biases, blind spots, and architectural preferences. Deploying a hyper-personalized execution harness laterally across varying engineering teams guarantees deep semantic friction. Distinguishing absolute domain skills from idiosyncratic human preferences remains a pivotal challenge in scalable AI operations.

The Compound Architectural Return

The 14K token baseline ceases to represent an operational expense when functioning as a compounding structural investment.

Elevating the aggregate throughput of foundation models revolves around two methodologies: fine-tuning and harness engineering. Fine-tuning forcibly modifies baseline parameters, bearing immense cost, risks of catastrophic context forgetting, and opaque latent dynamics.

==Conversely, Harness Engineering fundamentally manipulates the sequence context vector.== It is computationally lightweight, totally subject to strict git version control, and inherently deterministic to inspect.

The elite infrastructure teams yielding maximum asymmetric leverage from generative systems are not hoarding proprietary LLM variants. They are deliberately investing the initial 14K token load upfront. They migrated off generic, stateless dialogue APIs into profoundly structured, agentic execution loops (like the Ralph iterations). It is the only guaranteed mechanism to encode judgment directly into the execution fabric.

The harness pipeline should remain completely stripped of logic. The domain parameter skills should be deeply dense. And the Layer 3 temporal persistence graph must perpetually learn.

That is how engineering scales outside the standard training distribution.

Sharad Jain builds agentic AI pipelines in Bengaluru. He previously engineered core data infrastructures at Meta and acts as the founder of autoscreen.ai, a production voice AI platform. He writes deeply about ML systems architectures at sharadja.in.

Research & Footnotes:

Liang et al., "Measuring and Controlling Instruction (In)Stability in Language Model Dialogs" — COLM 2024
Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" — Stanford / UC Berkeley 2023
Anthropic Alignment Science: Building Effective Agents
Y Combinator (Garry Tan): Thin Harness, Fat Skills Substrate
Tobi Lütke: QMD — Local hybrid search engine for markdown knowledge bases (BM25 + vector + HyDE)
Open Source: awesome-claude-code — Ecosystem for structural CLAUDE.md mappings.

My AI Agent's Memory Paid for Itself: How 590 Sessions Made Claude Code 1.8x Cheaper

Sharad Jain — Wed, 01 Apr 2026 10:00:00 GMT

590 Sessions Later

A year ago I wrote about the 14K token debt -- the idea that your system prompt is the most important architectural decision in agentic AI. That post argued for building a temporal knowledge graph: index your Claude Code sessions, feed the agent its own past, and compound returns across sessions.

I've been running that system in production for six months. The question I couldn't answer back then: does it actually work?

So I ran the experiment. 14 queries, 28 runs, two conditions. This post documents the system, the methodology, the results, and the one finding I didn't expect -- that an agent doing more work can cost less, if the work is structured right.

The System

The system is called Brain MCP. It is a memory layer that sits between Claude Code and QMD, Tobi Lutke's MCP-native search server. QMD provides BM25, vector, and hypothetical document search over markdown collections via the Model Context Protocol. Brain MCP adds four things on top:

A session indexer that parses Claude Code's JSONL transcripts into structured markdown with frontmatter, conversation turns, tool logs, and reasoning traces.
An LLM distillation pipeline that compresses each session into a dense knowledge artifact -- goals, decisions, rejected approaches, concept tags -- using Gemini 2.5 Flash.
An automated sync loop that triggers on every session end, keeping the index current without any human intervention.
A self-optimizing search strategy that mutates its own parameters and measures the results against a deterministic scoring function.

The architecture:

Claude Code Session
        |
        v
  [Stop Hook]  ─────────────────────────────+
        |                                    |
        v                                    v
  brain.py index                        qmd update
  (JSONL → sessions/*.md)             (BM25 reindex)
        |
        v
  [cron every 2h]
        |
        v
  brain.py distill
  (Gemini 2.5 Flash)
        |
        v
  distilled/*.md                  ┌──────────────────────┐
  (goals, decisions, tags)        │    QMD MCP Server     │
        |                         │                       │
        v                         │  lex:  BM25 keyword   │
  qmd embed                       │  vec:  semantic        │
  (vector embeddings)             │  hyde: hypothetical    │
        |                         │                       │
        +────────────────────────►│  4 collections:       │
                                  │   brain     (590 docs)│
                                  │   distilled (395 docs)│
                                  │   kb-wiki   (211 docs)│
                                  │   kb-raw    (526 docs)│
                                  └──────────┬────────────┘
                                             │
                                             v
                                  Claude Code queries
                                  its own past

Production stats as of April 2026: 597 sessions indexed. 395 distilled. 1,709 total searchable documents across four collections. The Stop hook fires in under two seconds. The cron job runs every two hours.

The Four Collections

Each collection serves a different retrieval purpose. This isn't arbitrary -- it's the result of six months of iterating on what the agent actually needs at query time.

Collection	Docs	What's in it	When it helps
brain	590	Raw session transcripts -- every tool call, file read, decision, correction	When you need verbatim recall: "what exact command did I run?"
distilled	395	LLM-compressed summaries -- goals, key decisions, rejected approaches, concept tags	When you need semantic recall: "what did I decide about auth?"
kb-wiki	211	Synthesized concept pages, comparison tables, entity summaries	When you need general knowledge: "how does BM25 work?"
kb-raw	526	Raw source articles, transcripts, documentation	When you need primary sources: the original article about RAG architectures

==The insight behind having both brain and distilled is that they fail in complementary ways.== Raw transcripts are noisy but lossless -- you can grep for the exact error message you saw three months ago. Distilled artifacts are clean but lossy -- the LLM may have dropped a detail that turns out to matter. Together, they cover more ground than either alone.

Three Search Modes

QMD exposes three search modes, each suited to different query types:

lex (BM25): Classical keyword matching with TF-IDF weighting. Fast, exact, no embedding required. Best for: specific error messages, CLI commands, file paths, proper nouns.
vec (semantic): Dense vector similarity over document embeddings. Best for: conceptual queries where the exact terms don't appear in the document ("authorization failures" matches a session about "JWT token rejection").
hyde (hypothetical document): You write a short paragraph describing what the answer looks like. The engine embeds that paragraph and finds documents similar to it. Best for: open-ended questions where you know the shape of the answer but not where it lives.

A single query can combine all three. The search_strategy.yaml controls the order -- the first sub-query gets 2x weight during reranking:

queries:
  sub_query_order: [lex, vec, hyde]
  hyde_template: >
    The answer to the question "{query}" in the context of AI agents,
    LLM systems, autonomous research loops, and knowledge base design is:

The Indexer

The session indexer (brain.py) is 614 lines of Python with no dependencies beyond the standard library (plus google-genai for distillation). It does two things: parse JSONL transcripts into markdown, and distill them via LLM.

Parsing: JSONL to Markdown

Every Claude Code session produces a JSONL file where each line is a JSON object with a type field: system, user, assistant. Assistant messages contain content blocks -- text, thinking, and tool_use. The parser extracts all three:

EXTRACT_INPUT_TOOLS = {"Read", "Edit", "Write", "Glob", "Grep",
                       "Bash", "WebSearch", "WebFetch"}

def extract_tool_summary(block: dict) -> str | None:
    """Return a one-line summary of a tool_use block."""
    name = block.get("name", "")
    inp  = block.get("input", {})
    if name == "Bash":
        cmd = inp.get("command", "").strip()
        if cmd:
            return f"Bash: {cmd[:120]}"
    elif name in ("Read", "Edit", "Write", "Glob", "Grep"):
        path = (inp.get("file_path") or inp.get("pattern")
                or inp.get("path") or "")
        if path:
            return f"{name}: {path}"
    ...

The output is a markdown file with YAML frontmatter (for QMD indexing) and three sections: Conversation, Tools Used, and Reasoning. Each session gets one file in ~/.brain/sessions/:

---
session_id: 02b3c2d7-22e2-4f22-b270-bb14c60e1c87
slug: rosy-sprouting-sprout
date: 2026-04-02
project: ~/Projects/udacity-reviews-hq/projects/travel-agent/tmp
branch: master
---

## Conversation

**User**: Can you diagnose when I enter this command, `gemini -yolo`?
It takes like 6-7 good seconds to boot up...

**Assistant**: Found the main culprit. There are two issues...

## Tools Used

- Read: ~/.gemini/settings.json
- Bash: strace -e trace=network gemini --yolo 2>&1 | head -30
- Edit: ~/.gemini/settings.json

## Reasoning

> The user reports 6-7 second startup delay. Let me check if there's
> a blocking network call or file I/O issue in the gemini CLI...

The --new flag makes indexing incremental -- it only processes JSONL files whose modification time is newer than the corresponding .md file. The --queue flag appends newly indexed session stems to pending_distill.txt for the cron job to pick up later.

The Stop Hook

The key infrastructure is a Claude Code Stop hook that fires after every session ends:

{
  "hooks": {
    "Stop": [{
      "matcher": "",
      "hooks": [{
        "type": "command",
        "command": "python3 ~/.brain/brain.py index --new --queue >> ~/.brain/index.log 2>&1 && qmd update >> ~/.brain/index.log 2>&1"
      }]
    }]
  }
}

This runs synchronously before the session fully closes. Two operations: index --new --queue (parse any new JSONL files, enqueue for distillation) and qmd update (rebuild the BM25 index). Total latency: under two seconds. The user doesn't notice.

The heavy work -- LLM distillation and vector embedding -- happens asynchronously via a cron job every two hours:

#!/usr/bin/env bash
# cron_sync.sh — runs every 2 hours

# 1. Catch any sessions missed by the Stop hook
python3 ~/.brain/brain.py index --new --queue

# 2. Drain distillation queue via Gemini
python3 ~/.brain/brain.py distill --from-pending

# 3. BM25 reindex (picks up new distilled/ files)
qmd update

# 4. Refresh vector embeddings (~17s per batch)
qmd embed

This two-tier architecture -- fast synchronous indexing on every session, slow asynchronous distillation on a cron -- means the BM25 index is always current (within seconds) while the semantic index catches up within two hours. In practice, this is good enough: by the time I need to semantically search a session, the cron has already processed it.

The Distillation Pipeline

This is the layer that makes the system more than a glorified grep. Raw session transcripts are 50-200 lines of conversation noise -- tool calls, error messages, tangential explorations, corrections. The distillation pipeline compresses each session into a ~30-line structured artifact that captures what actually matters.

What Distillation Produces

Here's a real example. The raw session is a 60-line transcript about diagnosing a 6-7 second startup delay in the Gemini CLI. Here's what the distiller produces:

---
session_id: 02b3c2d7-22e2-4f22-b270-bb14c60e1c87
date: 2026-04-02
project: ~/Projects/udacity-reviews-hq/projects/travel-agent/tmp
distilled: true
distilled_at: 2026-04-06
prompt_version: v1-d68727f6c6f4
confidence: High
---

## Goal
Diagnose and resolve a 6-7 second startup delay in the `gemini -yolo` command.

## What Happened
Initial diagnosis revealed a blocking IDE extension connection timeout
caused by `ide.enabled: true` in `~/.gemini/settings.json`. After
disabling IDE integration, a new 12.5s delay (10s I/O wait) was traced
to a 730MB `~/.gemini/tmp/` directory with 1,298 session checkpoint files.

## Key Decisions
- Disabled IDE integration — not using Windsurf features from terminal
- Identified two separate bottlenecks (network timeout + I/O wait)

## What Was Rejected
- Did not immediately disable session retention without understanding it

## Technologies
gemini, Windsurf, Node.js, Claude Code

## Concepts
CLI startup performance, IDE integration, connection timeout, I/O wait,
session retention, headless execution

## Tags
gemini cli, startup performance, ide integration, connection timeout,
session retention, performance optimization, troubleshooting

The raw transcript has the full diagnostic conversation -- the back-and-forth about what IDE integration does, the strace commands, the file size checks. The distilled version has the intelligence: what was the goal, what was decided, what was rejected, and what concepts were involved.

The Distillation Prompt

The prompt is carefully designed to prevent the most common failure mode: LLM invention. The distiller must extract only what is explicitly stated, never infer or fabricate:

DISTILL_PROMPT = """You are a technical knowledge distiller.
Given a Claude Code conversation, extract a structured knowledge artifact.

Output this exact structure:
---
session_id: {session_id}
date: {date}
project: {project}
...
---

## Goal
[One sentence. If purely exploratory with no clear goal, say so.]

## What Happened
[3-5 sentences. Past tense. Include outcome.]

## Key Decisions
[2-5 bullet points. Omit entire section if no decisions were made.]

## What Was Rejected
[0-3 bullet points. Omit if none -- do NOT invent rejections.]

...

Rules:
- Only extract what is EXPLICITLY stated. Never invent.
- Keep Goal to one sentence maximum.
- If session is a narrow sub-agent task, describe that task accurately.
"""

The prompt version is tracked via a hash (v1-d68727f6c6f4). When the prompt changes, the --new flag detects stale distillations by checking the prompt_version field in existing files and re-queues them.

Cost and Throughput

Distillation runs on Gemini 2.5 Flash. I evaluated four options before settling on this:

Model	Cost per 394 sessions	Speed	Tradeoff
Gemini 2.5 Flash (batch)	$0.38	~1 hour	Best price/quality
Qwen 3.5 Plus (batch)	$0.27	~1 hour	Cheapest, Alibaba account needed
Kimi K2.5	$0.72	~5 min	No batch discount
Gemma 4 12B (local)	$0.00	~65 min	Zero cost, M1 Pro 40 tok/s

At $0.38 for the initial 394-session backfill and ~$0.001/day ongoing, the cost is effectively zero. The session text is truncated to 8,000 characters before being sent to the LLM -- enough to capture the essential decisions without hitting rate limits on long sessions.

Rate limiting is handled with exponential backoff:

for attempt in range(3):
    try:
        response = client.models.generate_content(
            model="gemini-2.5-flash", contents=prompt
        )
        return response.text.strip()
    except Exception as e:
        if "429" in str(e) or "RESOURCE_EXHAUSTED" in str(e):
            wait = 60 * (2 ** attempt)  # 60s, 120s, 240s
            time.sleep(wait)

The Experiment

Design

I took 14 knowledge retrieval queries spanning five categories:

Operational recall (3 queries): Questions about specific past decisions.

"What decisions were made about the OpenClaw sandbox bind mount configuration?"
"How was the brain MCP CLI designed and what was rejected?"
"What is the current state of the OpenClaw artifact classification pipeline?"

Architecture (3 queries): System design and understanding.

"How do autonomous LLM agents use tools and manage context across turns?"
"What is the architecture of a RAG system and where does retrieval quality break down?"
"How should a knowledge base be structured for LLM retrieval and synthesis?"

Research concepts (3 queries): Conceptual knowledge retrieval.

"What are the tradeoffs between BM25 keyword search and semantic vector search?"
"How does the Karpathy autoresearch loop pattern work?"
"What techniques improve LLM reasoning quality on complex multi-step tasks?"

Cross-project debugging (3 queries): Lessons learned across projects.

"Why do multi-agent systems fail to coordinate?"
"What deployment mistakes caused the most debugging sessions?"
"How was the Jules API integration implemented?"

Active project context (2 queries): Current state awareness.

"What is the current goal of the Google Ads autoresearcher project?"
"What LLM agent frameworks have been used across projects?"

These 14 queries were originally created for the autoresearch loop's test suite -- they weren't cherry-picked for the benchmark. They span all four collections and represent the kinds of questions I actually ask the brain in production.

Conditions

Two conditions, run sequentially on the same machine:

Vanilla: Claude Code with brain MCP tools blocked via --disallowedTools mcp__brain__query,mcp__brain__get,mcp__brain__multi_get,mcp__brain__status. The agent can only use Grep, Glob, Read, and Bash to search ~/.brain/ files manually. It has access to the same data -- just not the structured retrieval tools.

Brain MCP: Claude Code with full access to QMD brain MCP tools. The agent can issue multi-mode search queries (lex/vec/hyde) across all four collections and retrieve documents by path.

Both conditions use:

Claude Sonnet 4 (claude-sonnet-4-6)
Identical system prompt: "You are a knowledge retrieval agent. Answer the user's question using the tools available to you. Search thoroughly, then provide a concise, well-sourced answer."
Pipe mode (claude -p --output-format json) for structured metrics
10-minute timeout per query
No session persistence (--no-session-persistence)
Bypass permissions (--permission-mode bypassPermissions)

Measurement

The benchmark harness (benchmark_harness.py) captures metrics from Claude Code's JSON output:

usage = data.get("usage", {})
inp_tok = usage.get("input_tokens", 0)
out_tok = usage.get("output_tokens", 0)
cache_read = usage.get("cache_read_input_tokens", 0)
cache_create = usage.get("cache_creation_input_tokens", 0)

# Effective tokens = everything the model actually processed
effective_tokens = inp_tok + out_tok + cache_read + cache_create

Metrics per experiment: effective tokens processed, output tokens generated, cache read/create tokens, agentic turns, API latency (server-side), wall-clock time, cost in USD, answer length in characters.

The harness strips ANTHROPIC_API_KEY from the environment to ensure Claude Code uses OAuth rather than a stale API key:

env = {k: v for k, v in os.environ.items() if k != "ANTHROPIC_API_KEY"}

28 experiments total: 14 queries x 2 conditions. Vanilla runs first, then Brain MCP. This ordering matters -- I'll address the bias it introduces in the limitations section.

The Results

Aggregate

Metric	Vanilla	Brain MCP
Total cost	$5.64	$3.18	1.8x cheaper
Wall-clock	39.5 min	21.9 min	1.8x faster
API latency	1,621s	951s	1.7x faster
Output tokens	22,537	46,355	2.1x more generated
Answer length	48,054 chars	56,129 chars	17% longer answers
Tokens processed	1.1M	3.5M	3.2x more throughput
Agentic turns	57	185	3.2x more turns
Failures	1 timeout	0	100% completion

Brain MCP won on cost for 12 of 14 queries (86%). Won on speed for 12 of 14 (86%).

Per-Query Breakdown

#	Query	V Cost	B Cost	Savings	V Wall	B Wall	Speedup
1	OpenClaw sandbox decisions	$0.34	$0.12	2.7x	113s	52s	2.2x
2	Brain CLI design + rejections	$0.54	$0.23	2.3x	402s	97s	4.1x
3	OpenClaw artifact classification	TIMEOUT	$0.17	--	300s	89s	3.4x
4	Autonomous LLM agents + tools	$0.52	$0.20	2.6x	166s	111s	1.5x
5	RAG architecture + quality	$0.38	$0.16	2.3x	111s	77s	1.4x
6	Knowledge base for LLM retrieval	$0.41	$0.19	2.1x	148s	97s	1.5x
7	BM25 vs semantic search	$0.37	$0.14	2.8x	119s	60s	2.0x
8	Karpathy autoresearch loop	$0.23	$0.28	0.8x	72s	82s	0.9x
9	LLM reasoning techniques	$0.43	$0.24	1.8x	146s	134s	1.1x
10	Multi-agent coordination	$0.52	$0.42	1.2x	245s	161s	1.5x
11	Deployment debugging	$0.72	$0.21	3.5x	180s	89s	2.0x
12	Jules API integration	$0.63	$0.19	3.4x	159s	66s	2.4x
13	Google Ads autoresearcher	$0.28	$0.24	1.2x	90s	90s	1.0x
14	LLM agent frameworks survey	$0.28	$0.40	0.7x	119s	109s	1.1x

The biggest wins cluster around targeted recall queries -- questions with a specific right answer that lives in a specific document. Query 11 (deployment debugging) shows the pattern most clearly: vanilla Claude Code spent $0.72 and 180 seconds doing broad grep scans across ~/.brain/sessions/, reading full files, getting noise. Brain MCP spent $0.21 and 89 seconds making targeted queries, reading the top-ranked results, and synthesizing.

Query 2 (brain CLI design) is the most dramatic speedup: 4.1x faster. Vanilla took 402 seconds -- over six minutes of file searching -- while Brain MCP needed 97 seconds. The vanilla agent made 5 turns of broad file reads. Brain MCP made 13 turns of targeted retrieval, each turn cheap because the system prompt was already cached.

Query 3 (OpenClaw artifact classification) is the only vanilla timeout. The agent spent its full 5-minute budget on grep scans without finding the right session files. Brain MCP answered the same query in 89 seconds with 10 targeted turns.

The Counterintuitive Finding: Why More Work Costs Less

Here's the result I didn't expect:

Brain MCP processed 3.5 million effective tokens across 185 agentic turns. Vanilla processed 1.1 million tokens across 57 turns. Brain MCP cost 44% less.

More tokens. More turns. Less money. This seems wrong until you look at the token breakdown.

The Cache Mechanism

Claude Code's system prompt is approximately 14,000 tokens. On the first turn of a conversation, these tokens are processed at the normal input rate (cache creation, billed at 1.25x). On every subsequent turn, they're read from cache at 0.1x the normal rate.

The pricing math:

Token type	Rate (Sonnet)
Input (uncached)	$3.00 / 1M
Cache creation	$3.75 / 1M (1.25x)
Cache read	$0.30 / 1M (0.1x)
Output	$15.00 / 1M

Brain MCP averages 13.2 turns per query. After the first turn, every subsequent turn reads the ~14K system prompt from cache at the discount rate. Its pattern: query brain -> read top result -> refine search terms -> read another result -> synthesize. Each turn adds a small amount of new context (the retrieved document) on top of a large cached prefix.

Vanilla averages 4.4 turns per query (excluding the timed-out query). But each turn does expensive file reads that inject large amounts of new, uncached context. A single Read tool call that returns a 200-line session transcript adds thousands of uncached tokens. A Grep that returns 50 matching lines adds uncached context that must be processed at full price.

The net effect:

	Vanilla	Brain MCP
Effective tokens	1,149,157	3,523,380
Cache tokens	1,126,510 (98.0%)	3,462,607 (98.3%)
Uncached input + output	22,647	60,773
Cost	$5.64	$3.18

Both modes are dominated by cache tokens -- 98% or more. But the composition of those cache tokens differs. Brain MCP's cache tokens are overwhelmingly cache reads (cheap). Vanilla's cache tokens include more cache creation from the large file contents injected on each turn.

==The principle: many cheap, targeted turns beat few expensive, broad turns. Structured retrieval creates cache-friendly access patterns.== Brute-force grep doesn't.

This has implications beyond this benchmark. It suggests that the conventional wisdom of "minimize turns to minimize cost" is wrong for agentic systems with cached system prompts. The right optimization target isn't turn count -- it's uncached token injection per turn. A 20-turn conversation where each turn adds 500 tokens of targeted retrieval costs less than a 4-turn conversation where each turn adds 5,000 tokens of grep output.

Where Brain MCP Lost

Two queries where vanilla was cheaper:

Query 8 -- "How does the Karpathy autoresearch loop pattern work?"

Vanilla: $0.23, 72s, 16 turns. Brain MCP: $0.28, 82s, 14 turns.

The answer lived in local project files that vanilla found via grep in its first few turns. The Karpathy pattern is implemented in the current working directory (tools/autoresearch/), and vanilla Claude Code is excellent at finding things in local files. Brain MCP did a thorough cross-collection search -- more comprehensive, but overkill for a question answerable from one file.

Query 14 -- "What LLM agent frameworks and patterns have been used across projects?"

Vanilla: $0.28, 119s, 2 turns. Brain MCP: $0.40, 109s, 24 turns.

This is a broad survey question. Brain MCP did the deepest search in the entire benchmark -- 24 turns across all four collections, reading extensively about every framework mentioned in any session. It produced a more thorough answer. But the thoroughness wasn't worth the cost. Vanilla gave a shorter, less detailed answer in just 2 turns, and for a survey question, "good enough" is good enough.

The pattern: Brain MCP loses on two types of queries:

Single-file answers where the data lives in one obvious location (grep is faster than structured search)
Broad surveys where exhaustive search costs more than the marginal value of completeness

It wins decisively on targeted recall -- "what was decided about X?" -- where relevance ranking eliminates wasted reads. That's 12 of 14 queries in this benchmark.

The Self-Optimizer

Building the brain was step one. Making it optimize itself was step two.

I implemented a Karpathy-pattern autoresearch loop: one editable file, one immutable runner, one deterministic scorer. The key insight from Karpathy's design: the human writes the scoring function, the LLM proposes mutations, and the loop is fully autonomous.

The Tripartite Architecture

File	Role	Who edits it
`search_strategy.yaml`	The mutable config -- search weights, limits, templates	LLM only
`program.md`	Operating manual -- scoring formula, mutation vocabulary, failure modes	Human only
`runner.py`	Immutable loop -- eval -> mutate -> keep/revert -> git commit	Never
`evaluator.py`	Deterministic scorer -- precision + recall_proxy + diversity	Read-only tool

The separation is critical. The LLM can only touch the strategy file. The human controls the objective function and constraints. The runner enforces the loop. No component can interfere with another.

The Hill Function

The scorer is fully deterministic -- no LLM calls needed for evaluation:

def evaluate(query: str, strategy: dict) -> dict:
    docs = brain_query(query, strategy)

    precision = score_precision(docs, PRECISION_TOP_K)  # mean score of top-6
    recall = score_recall_proxy(docs)    # fraction returning >= 3 docs
    diversity = score_diversity(docs)    # fraction of 4 collections represented

    score = 0.5 * precision + 0.3 * recall + 0.2 * diversity
    return {"score": round(score, 4), ...}

Three signals, weighted to prioritize precision:

Precision (weight 0.5): Mean relevance score of the top-6 results, measured using QMD's reranker score. Uses a fixed window (PRECISION_TOP_K = 6) regardless of other parameters, preventing the optimizer from gaming the metric by reducing result count.
Recall proxy (weight 0.3): Binary -- did the query return at least 3 documents? This is a crude proxy, but it catches the failure mode where over-filtering leaves queries unanswered.
Diversity (weight 0.2): What fraction of the four collections are represented in the results? This encourages cross-collection retrieval, which matters because different query types need different sources.

The scorer runs all 14 test queries against QMD, computes the aggregate, and returns a single number. No LLM calls. No API costs. This makes it cheap to run hundreds of iterations.

The Mutation Backend

The runner launches Jules (Google's async coding agent) to propose mutations. Each iteration:

Runner loads the current strategy + last 10 experiment logs + the operating manual (program.md)
Jules reads the context, proposes exactly one mutation with a # HYPOTHESIS: comment
Runner evaluates the mutation against all 14 queries
If the score improves by > 0.01, keep. Otherwise, revert.
Git commit either way -- the log is the experiment history.

for i in range(1, max_iter + 1):
    new_strategy_raw = jules_mutate(strategy_raw, logs)

    new_eval = evaluate_all(queries, new_strategy, verbose=True)
    delta = new_eval["mean_score"] - best_score

    if delta > min_improvement:
        decision = "keep"
        best_score = new_eval["mean_score"]
    else:
        decision = "revert"
        git_restore()

Plateau detection stops the loop after 3 consecutive non-improvements.

What the Optimizer Found

Starting score: 0.6993 (baseline configuration). Final score: 0.7350 after 15+ iterations.

The winning mutations:

limit: 10 -> 40. More results = more collection coverage = higher diversity score.
candidate_limit: 10 -> 40. Matching the limit to the full pre-rerank pool prevents good semantic results from being pruned.
sub_query_order: Confirmed [lex, vec, hyde] -- lex first gets 2x reranking weight, which helps precision.
hyde_template: Refined to include domain-specific context ("AI agents, LLM systems, autonomous research loops").

Dead ends (the optimizer tried and reverted):

sub_query_order: [vec, lex] -- no effect (+0.000)
candidate_limit: 80 -- no effect (limit was already at the ceiling)
min_score: 0.25 or 0.20 -- slight regression (no new collections appeared)
limit > 40 with candidate_limit=40 -- no effect (candidate pool exhausted)

The optimizer also discovered a hard constraint: three queries are stuck at diversity=0.50 because the relevant information only exists in two of four collections (brain + distilled). No search strategy change can fix this -- only adding content to kb-wiki or kb-raw would help. This is a genuine ceiling on the score, and the optimizer correctly stopped trying to push past it.

I started the loop and went to sleep. Woke up to a git log of experiments, each with a hypothesis, a score delta, and a keep/revert decision. The agent was doing science while I slept.

Honest Limitations

This benchmark has real flaws. I'm documenting them because the temptation to present clean results is the enemy of useful results.

1. No ground truth scoring. I measured cost, speed, and output volume -- not whether the answers were correct or complete. Brain MCP could produce faster, cheaper, wrong answers and this benchmark would call it a win. A rigorous version would score each answer against gold-standard reference answers written by a human.

2. N=1 per condition. No repetitions. LLM outputs are stochastic -- the same query can produce different tool call sequences on different runs. One outlier could skew the aggregates. A proper experiment would run 3-5 repetitions per condition and report confidence intervals.

3. Cache order bias. Vanilla ran first, then Brain MCP. This means Brain MCP benefits from any system-level caching (OS page cache, QMD's internal caches) that vanilla's run warmed up. More importantly, Claude Code's prompt cache persists across sessions on the same machine, so Brain MCP may have started with a warmer cache. Randomizing run order across queries would reduce this bias.

4. No baseline. Both modes include Claude Code's full ~14K token system prompt. I'm measuring the marginal value of brain MCP tools on top of that baseline, not absolute retrieval efficiency. A proper comparison would include a third condition: the naked LLM with no tools at all, answering from parametric knowledge only.

5. Model specificity. Sonnet, not Opus. Tool-use behavior, search strategy, and persistence differ between models. Results may not generalize.

6. Confounded system prompt. The system prompt says "search thoroughly" -- this may bias Brain MCP toward more turns (which happen to be cheaper) and vanilla toward broader reads. A neutral prompt might produce different cost dynamics.

7. Evaluator-subject identity. I designed the system, wrote the queries, built the benchmark, and analyzed the results. The usual blinding practices are absent.

A rigorous version of this experiment would: score answers against gold-standard references, run 3+ repetitions per condition per query, randomize condition order, add a naked-LLM baseline, test on multiple models, and have someone else write the test queries.

What I Learned

Memory is not RAG

==RAG retrieves documents to answer questions. Memory retrieves decisions, corrections, and rejected approaches to avoid repeating mistakes.== The difference is in what you index and how you structure it.

A RAG system over my session transcripts would return noisy conversation fragments. The distillation layer is what turns it into memory -- compressing a 2,000-line session into a 30-line artifact with structured sections (Goal, Key Decisions, What Was Rejected, Tags). When the agent queries "what did I decide about auth?", it gets back a structured decision record, not a raw conversation snippet.

The "What Was Rejected" section is particularly valuable. It prevents the agent from re-exploring dead ends. If a previous session tried and rejected approach X for reason Y, the memory system surfaces that rejection, saving the cost of rediscovering the same dead end.

The temporal dimension matters

A knowledge base is static. A memory system grows with every session. The Stop hook that fires after every Claude Code conversation is the key infrastructure -- it makes indexing invisible. If it required manual effort, I'd have stopped at session 20.

==The automation has a compounding effect: every session makes the next session better, without any effort from me.== Session 590 can query the decisions from session 1. The agent doesn't just have instructions -- it has experience.

Cache economics change the design space

The conventional wisdom is "fewer tokens = cheaper." ==This benchmark shows the opposite: more turns with cache-friendly patterns can be cheaper than fewer turns with cold reads.==

This has implications for how agentic systems should be designed. Instead of optimizing for turn count, optimize for cache utilization. Structured retrieval tools that return small, targeted results create naturally cache-friendly patterns. File-reading tools that inject large, uncached context don't.

The 0.1x cache read pricing is the key parameter. At that discount, the break-even point is roughly: if your cache prefix is 10x larger than the new context you're adding, additional turns are almost free. Brain MCP's pattern -- 14K cached prefix, ~500 tokens of new retrieval per turn -- is well below that threshold.

Self-optimization is underrated

The autoresearch loop found a better search strategy than I could have designed by hand. The deterministic hill function (no LLM needed for scoring) makes it cheap to run hundreds of iterations. The Karpathy pattern -- one mutable file, one immutable loop, one scoring function -- is the simplest architecture that works for autonomous parameter search.

==The key design decision: the scoring function must be deterministic and cheap.== If you need an LLM to evaluate each iteration, the loop is too expensive to run autonomously. The three-signal hill (precision, recall proxy, diversity) is crude, but it's computable in seconds with no API calls. That's what makes overnight autonomous optimization feasible.

The brain MCP system, the benchmark harness, and the autoresearch loop will be open-sourced soon.

Sharad Jain builds agentic AI systems in Bengaluru. He previously built data infrastructure at Meta and founded autoscreen.ai, a production voice AI platform. He writes about agent architecture at sharadja.in.

Previously: The 14K Token Debt: How Your System Prompt Shapes Everything

Orchestrating AI Agents for Data Pipelines

Sharad Jain — Tue, 15 Apr 2025 10:00:00 GMT

Orchestrating AI Agents for Data Pipelines

Data pipelines are the most boring part of engineering — until they break. Then they're the most expensive.

I spent two years at Meta running Spark + Airflow pipelines that processed 10TB+ per day. The infrastructure was impressive. The failure modes were not. A single upstream schema change could silently corrupt downstream models for days before anyone noticed. A partition key format change once poisoned a week of training data. The fix took 45 minutes. The detection took six days.

That six-day gap is not an anomaly. Monte Carlo's 2023 State of Data Quality survey found that 68% of data teams need four or more hours just to detect a pipeline failure, with an average resolution time of 15 hours per incident — and organizations average 67 incidents per month. The business impact: 31% of revenue is exposed to data quality issues, up from 26% the year prior.

This is the dirty secret of data engineering at scale: we've built incredibly powerful pipes and incredibly dumb monitoring. What if the pipes could think?

The Brittleness Problem

Traditional data pipelines follow a rigid choreography:

Extract — pull from source (database, API, event stream)
Transform — clean, reshape, enrich
Load — write to destination (warehouse, lake, feature store)

Each step assumes the world hasn't changed since the pipeline was written. But the world always changes. Upstream teams rename columns. API response formats evolve. Event schemas gain new fields. And your pipeline — faithfully executing the DAG you wrote six months ago — silently produces garbage.

The standard defense is testing: schema validation, data quality checks, anomaly detection. But these are reactive by design. They catch problems after the data is already wrong. At 10TB/day, even a 0.1% failure rate means 10GB of corrupted data flowing downstream before anyone raises an alarm. And the industry data confirms the pattern: Monte Carlo found a steady ratio of one data quality incident per 15 tables per year across their platform — meaning a 600-table warehouse generates roughly 40 incidents annually just from normal operations.

==I call this the Detection-Resolution Gap: the growing distance between when data goes bad and when someone does something about it.== Every hour in that gap compounds damage — bad dashboards inform bad decisions, corrupted features degrade ML models, and downstream consumers silently ingest poison. The gap is widening because data volumes grow faster than team headcount. You can't hire your way out of it.

The Agent Intelligence Layer

The fix isn't better tests. It's a different primitive.

==Instead of writing pipelines as static DAGs with bolted-on checks, design them as networks of specialized agents that observe, decide, and act at each pipeline boundary.== The agents don't replace your Spark jobs or Airflow DAGs — they sit between pipeline stages as an intelligence layer, making decisions that previously required a human on-call.

I think of this as the Observe-Decide-Act Loop — a pattern that distinguishes agent-based pipelines from traditional monitoring:

Traditional pipeline:
  Source → Transform → Load → [check after the fact] → Alert human → Human investigates → Human fixes

Agent-orchestrated pipeline:
  Source → [Agent: observe schema, decide compatibility] → Transform → [Agent: observe distributions, decide quality] → Load
              ↓ (if anomaly detected)
         [Handoff to specialist agent → auto-remediate → resume]

==The critical shift: decisions happen inline, before bad data propagates, not after.== The Detection-Resolution Gap collapses from hours to seconds.

Three Primitives That Make It Work

OpenAI's Agents SDK crystallizes three primitives that map cleanly onto pipeline orchestration:

Handoffs — An agent can delegate to another agent. In pipeline terms: the ingestion agent detects a schema anomaly and hands off to a schema resolution agent, which generates the mapping transformation before passing data downstream. No human paged at 3am. No six-day detection lag. The SDK supports this through explicit handoff declarations where a triage agent routes to specialists who then own the resolution.

Guardrails — Input and output validation gates on every agent. Not just "is this field non-null?" but "does this distribution look right? Is this schema compatible with what downstream consumers expect? Did we lose cardinality during the join?" Guardrails encode the contracts between pipeline stages — making implicit assumptions explicit and enforceable.

Tracing — End-to-end observability across the entire agent network. Every decision, every handoff, every guardrail check is logged with full reasoning context. When something goes wrong, you don't grep through Airflow logs — you replay the agent's decision chain. This is what makes agent-based pipelines debuggable, not just automated.

These three primitives compose into what I call Contract-Aware Orchestration: each agent doesn't just process data — it understands the contract it must uphold for downstream consumers and actively enforces it. Traditional ETL treats contracts (when they exist at all) as documentation. Agent-based pipelines treat them as runtime constraints.

A Concrete Pattern: The Schema Drift Guardian

Here's what this looks like in practice. Consider the most common pipeline failure: an upstream team adds, renames, or removes a column, and your pipeline either crashes or silently drops data.

A Schema Drift Guardian agent handles this inline:

from agents import Agent, Runner, GuardrailFunctionOutput, input_guardrail
from pydantic import BaseModel
from typing import Literal

class SchemaVerdict(BaseModel):
    status: Literal["compatible", "breaking", "additive"]
    changes: list[str]
    remediation_sql: str | None
    confidence: float

quality_agent = Agent(
    name="SchemaGuardian",
    instructions="""You are a schema compatibility agent for a production data pipeline.

    Given the registered schema contract and the incoming data schema:
    1. Classify the change: additive (new columns), breaking (removed/renamed/retyped columns),
       or compatible (no change).
    2. For additive changes: generate ALTER TABLE statements to evolve the destination schema.
    3. For breaking changes: generate transformation SQL that maps the new schema to the
       existing contract, preserving all downstream dependencies.
    4. Assign confidence (0-1). If confidence < 0.85, flag for human review instead of
       auto-applying.

    Never silently drop columns. Never assume a renamed column is a new column without
    checking value distributions.""",
    model="gpt-4o",
    output_type=SchemaVerdict,
)

remediation_agent = Agent(
    name="SchemaRemediator",
    instructions="""You resolve schema mismatches between upstream sources and downstream
    consumers. Given the SchemaGuardian's verdict:
    - For additive changes: apply the evolution and notify downstream consumers.
    - For breaking changes with confidence >= 0.85: apply the transformation SQL,
      run validation on a sample batch, and proceed if validation passes.
    - For breaking changes with confidence < 0.85: quarantine the batch, notify the
      on-call engineer with full context, and continue processing other sources.

    Always log the before/after schema diff and the remediation applied.""",
    model="gpt-4o",
)

# The handoff: guardian escalates to remediator when schema drift is detected
quality_agent.handoffs = [remediation_agent]

The key design choices here:

Structured output (SchemaVerdict) — the agent doesn't produce free-form text; it returns a typed decision that downstream code can act on deterministically.
Confidence thresholds — the agent knows when it's uncertain and escalates to a human rather than auto-applying a bad fix. This is the difference between useful automation and dangerous automation.
Separation of detection and remediation — the guardian classifies; the remediator acts. This separation makes each agent simpler to test, debug, and improve independently.

Compare this to the traditional approach:

# Traditional: brittle, reactive, manual
try:
    df = spark.read.parquet(source_path)
    df_transformed = df.select(EXPECTED_COLUMNS)  # crashes if columns changed
except AnalysisException as e:
    send_pagerduty_alert(f"Schema mismatch: {e}")  # human wakes up at 3am
    raise  # pipeline halted until human intervenes

The traditional approach has two modes: success or human escalation. The agent-based approach has a spectrum: auto-resolve high-confidence changes, quarantine uncertain ones, escalate truly ambiguous cases. Most schema changes — the additive ones, the straightforward renames — never need a human at all.

The Confidence Boundary Pattern

The confidence threshold in the example above deserves its own framework, because it's the most important design decision in any agent-based pipeline.

I call this the Confidence Boundary: the threshold below which an agent stops acting autonomously and escalates to either a more specialized agent or a human. Getting this wrong in either direction is costly:

Boundary too high (e.g., 0.99): the agent escalates everything, and you've built an expensive alerting system disguised as an agent.
Boundary too low (e.g., 0.50): the agent auto-applies bad fixes, and you've built a corruption engine.

The right boundary depends on the blast radius of a wrong decision:

Pipeline Stage	Blast Radius	Recommended Confidence Boundary
Schema evolution (additive)	Low — worst case adds unused columns	0.70
Schema evolution (breaking)	High — can corrupt downstream models	0.90
Data quality (distribution drift)	Medium — may indicate real shift or bug	0.80
Data quality (null spike)	High — usually indicates upstream failure	0.85
Partition repair	Medium — can cause duplicate processing	0.80

==The principle: confidence boundaries should be inversely proportional to blast radius.== Low-risk changes get auto-resolved aggressively. High-risk changes require near-certainty or human review.

In practice, I recommend starting with all boundaries at 0.90 and tuning down as you build confidence in each agent's decision quality. Track the false positive rate (agent escalated unnecessarily) and false negative rate (agent auto-applied a bad fix) separately. You want the false negative rate near zero, even at the cost of a higher false positive rate.

The Voice AI Case: Real-Time Pressure Testing

These patterns become essential when latency constraints tighten. At autoscreen.ai, we process real-time voice AI event streams where pipeline failures can't wait for human intervention — calls are in flight, and degraded data means degraded user experience within seconds.

Each call produces four concurrent pipeline stages:

Audio chunks streamed to a transcription service
Transcription events fed to a scoring model
Scoring results written to an analytics database
Session metadata aggregated into dashboards

This is four pipeline stages running concurrently per call, multiplied by thousands of simultaneous calls. Traditional ETL falls apart here. You can't batch-process audio in hourly Airflow runs. You can't wait for a human to resolve a schema mismatch when 500 calls are in flight.

The agent-orchestrated approach:

┌─────────────┐     ┌──────────────┐     ┌───────────────┐     ┌──────────────┐
│  Ingestion   │────▶│ Transcription │────▶│  Scoring       │────▶│  Analytics   │
│  Agent       │     │  Agent        │     │  Agent         │     │  Agent       │
└──────┬───────┘     └──────┬────────┘     └───────┬────────┘     └──────┬───────┘
       │                    │                      │                     │
       ▼                    ▼                      ▼                     ▼
  ┌──────────┐        ┌──────────┐           ┌──────────┐          ┌──────────┐
  │ Quality  │        │ Quality  │           │ Quality  │          │ Quality  │
  │ Guard    │        │ Guard    │           │ Guard    │          │ Guard    │
  └────┬─────┘        └────┬─────┘           └────┬─────┘          └────┬─────┘
       │                   │                      │                     │
       └───────────────────┴──────────┬───────────┴─────────────────────┘
                                      ▼
                              ┌──────────────┐
                              │  Escalation  │
                              │  Agent       │
                              │ (human-in-   │
                              │  the-loop)   │
                              └──────────────┘

Each stage has its own quality guardrail tuned to its domain. The ingestion agent validates audio format and session metadata. The transcription agent checks for hallucination patterns — a real problem with streaming ASR where models occasionally confabulate words during silence or crosstalk. The scoring agent validates that output distributions haven't drifted. The analytics agent ensures aggregations are consistent.

When the transcription agent detects a new audio codec it hasn't seen before, it doesn't crash — it hands off to a codec resolution agent, logs the decision, and continues processing other calls while the issue is resolved.

This is what "self-healing" actually means in production. Not magic. Not AGI. Just agents with clear responsibilities, calibrated confidence boundaries, and the ability to escalate.

Where This Breaks

Every architecture post should include a section on when the approach fails. Agent-based pipelines are no exception.

LLM latency in the hot path. An LLM call adds 200-2000ms of latency per decision. For batch pipelines processing hourly, this is negligible. For real-time streams requiring sub-100ms latency, it's a dealbreaker. The mitigation: use LLM agents for policy decisions (schema evolution, anomaly classification) and deterministic code for per-record validation (null checks, type coercion). Don't put an LLM call in a tight loop processing millions of records.

Hallucination in remediation. When a schema agent generates transformation SQL, it can hallucinate column names or invent joins that don't exist. Recent research on agent hallucination shows that hallucinations in agent systems manifest not just as linguistic errors but as fabricated behaviors at every pipeline stage. The mitigation: always validate generated SQL against the actual schema before execution, run remediation on a sample batch first, and enforce the confidence boundary pattern described above.

Cost at scale. Each agent decision involves an LLM API call. At 67 incidents per month (Monte Carlo's average), this is cheap. At 67 incidents per hour in a high-volume pipeline, costs compound quickly. The mitigation: cache agent decisions for recurring patterns. If the agent has seen the same schema drift three times, apply the cached remediation without an LLM call.

Complexity ceiling. A network of six agents with handoffs between them is understandable. A network of sixty is not. Agent systems can become as opaque as the pipelines they're meant to improve. The mitigation: keep agent networks flat and shallow. A triage agent plus 3-5 specialist agents per pipeline boundary is the sweet spot. If you need more, you probably need to decompose the pipeline itself.

The accountability gap. When an agent auto-remediates a schema change and that remediation introduces a subtle data quality issue three stages downstream, who's responsible? The agent's tracing log shows what happened, but organizational accountability is still a human problem. Every auto-remediation should be logged, reviewable, and reversible.

Lessons From Both Ends of the Scale

I've built data pipelines at both extremes: Meta-scale batch processing (10TB+/day, Spark, Airflow) and real-time voice AI event streams (sub-500ms latency, LiveKit, streaming transcription). A few hard-won lessons:

Start with the contracts, not the agents. Define what "good data" looks like at each pipeline boundary before you make anything agentic. The agent layer is useless without clear success criteria. Data contracts — explicit schemas, SLOs on freshness and completeness, documented ownership — are the foundation. Research on schema evolution in data meshes shows that automated schema evolution reduces deployment time by 73% and error rates by 89%, but only when contracts are well-defined first.

Guardrails beat intelligence. A simple statistical check that catches distribution drift will save you more pain than a sophisticated LLM-powered anomaly detector. ==Use agents for the decisions (what to do when something breaks), not for the detection (noticing that something broke).== Detection should be fast, deterministic, and cheap. Decision-making is where LLM reasoning adds genuine value.

Tracing is non-negotiable. In a multi-agent pipeline, you need to know exactly which agent made which decision and why. OpenAI's SDK has this built in. If you're building your own agent layer, invest in tracing before you invest in features. Meta's recent work on AI-assisted pipeline knowledge mapping used 50+ specialized agents with multi-round critic passes — that system would be completely undebuggable without comprehensive tracing.

The handoff pattern is the whole game. The difference between a brittle pipeline and a resilient one is what happens when something unexpected occurs. Static DAGs crash. Agent networks route around the problem. But handoffs must be bounded — an agent should never hand off to more than 2-3 specialists, and handoff chains should never exceed 3 hops. Unbounded handoffs create circular delegation where agents pass problems to each other indefinitely.

Measure the gap, not the uptime. Pipeline uptime is a vanity metric. What matters is the Detection-Resolution Gap — the time between data going bad and data being fixed. Track this per pipeline stage. Organizations implementing automated recovery see a 76% reduction in MTTR and a 94% reduction in manual intervention. That's the metric that tells you whether your agents are actually working.

What's Next

The trajectory is clear: data infrastructure is getting a decision layer. Not because LLMs are good at processing data — they're not; Spark is still better at 10TB joins, and will be for the foreseeable future — but because LLMs are good at deciding what to do when the data is unexpected.

The next generation of data platforms won't just execute pipelines. They'll maintain contracts, detect violations, and take calibrated corrective action — all while maintaining the throughput and reliability that production systems demand. The tools are maturing: OpenAI's Agents SDK and AgentKit provide the orchestration primitives; frameworks like Anthropic's agent patterns, LangGraph, and CrewAI offer alternative approaches; and the data engineering ecosystem is converging on data contracts as the foundational abstraction.

The convergence I'm watching: data contracts + agent intelligence + confidence boundaries. When your pipeline stages declare typed contracts, agents can verify compliance at runtime and auto-remediate violations within calibrated confidence bounds. The contract tells the agent what "correct" looks like. The confidence boundary tells it when to act versus when to ask.

==We're not replacing data engineers. We're closing the Detection-Resolution Gap — and finally giving data infrastructure the judgment layer it has always needed.==

Sharad Jain is a data engineer and AI systems builder based in Bengaluru. He previously built data pipelines at Meta (10TB+/day) and founded autoscreen.ai, a production voice AI platform. He writes about the intersection of agentic AI and data infrastructure at sharadja.in.

Building an AI Hedge Fund: What Multi-Agent Trading Teaches About Agent Architecture

Sharad Jain — Wed, 15 Jan 2025 09:30:00 GMT

What Multi-Agent Trading Teaches About Agent Architecture

What happens when you treat a trading desk like a software architecture problem?

You give one agent the price charts. Another gets the balance sheets. A third reads the news. A fourth calculates intrinsic value. Then you put a risk manager between them and the portfolio, and a portfolio manager who must synthesize their conflicting recommendations into a single decision: buy, sell, or hold.

The result isn't a hedge fund. It's an architecture case study — and the patterns it reveals apply far beyond finance. The trading domain just happens to be uniquely good at exposing them, because markets are adversarial, data is noisy, feedback is immediate, and overconfidence is punished in dollars.

I've been building with virattt's ai-hedge-fund — an open-source project (55,000+ stars) that orchestrates six LLM-powered agents through LangGraph to simulate trading decisions. The system is educational, not production. But the architectural patterns it surfaces are real, and they've changed how I think about multi-agent coordination in any domain.

The Architecture of Disagreement

Traditional quantitative trading systems optimize for consensus. One model ingests data, produces a signal, and trades on it. If the signal is wrong, the model is wrong. There's no second opinion.

Multi-agent trading systems optimize for something different: productive disagreement. Multiple agents with different analytical lenses examine the same asset and reach different conclusions. ==The value isn't in any single agent's analysis — it's in the tension between agents that hold fundamentally different views of the same reality.==

This is a design choice, not an accident. Here's the architecture:

                    ┌──────────────┐
                    │  Market Data │
                    └──────┬───────┘
                           │
            ┌──────────────┼──────────────┐
            │              │              │
     ┌──────▼──────┐ ┌────▼─────┐ ┌──────▼──────┐
     │  Technical   │ │ Fundamen │ │  Sentiment  │
     │  Analyst     │ │ -tals    │ │  Analyst    │
     │              │ │ Analyst  │ │             │
     │ Price action │ │ Finan-   │ │ News, crowd │
     │ patterns,    │ │ cials,   │ │ psychology, │
     │ momentum     │ │ ratios   │ │ insider     │
     └──────┬───────┘ └────┬─────┘ └──────┬──────┘
            │              │              │
            │       ┌──────▼──────┐       │
            │       │  Valuation  │       │
            │       │  Agent      │       │
            │       │ DCF, owner  │       │
            │       │ earnings    │       │
            │       └──────┬──────┘       │
            │              │              │
            └──────────────┼──────────────┘
                           │
                    ┌──────▼───────┐
                    │ Risk Manager │
                    │ Position     │
                    │ sizing,      │
                    │ exposure     │
                    │ limits       │
                    └──────┬───────┘
                           │
                    ┌──────▼───────┐
                    │  Portfolio   │
                    │  Manager     │
                    │ Final        │
                    │ decision     │
                    └──────────────┘

The key structural insight: the analysis agents operate in parallel (they don't see each other's output), while the risk and portfolio managers operate in series (they see everything). This creates information asymmetry by design — each analyst commits to a view without being anchored by the others, and the portfolio manager must reconcile genuinely independent perspectives.

Compare this to a single-agent system where one LLM receives all the data simultaneously. That agent will anchor on the first strong signal it encounters, weigh all evidence through that anchor, and produce a coherent-sounding but potentially biased recommendation. The multi-agent architecture avoids this by making anchoring structurally impossible.

Six Agents, Six Blind Spots

Each agent in the system has a specific analytical lens — and a specific blind spot. Understanding the blind spots matters more than understanding the capabilities.

Technical Analyst — Reads price action, volume, and momentum indicators (moving averages, RSI, Bollinger Bands, ADX, ATR). The efficient market hypothesis says this shouldn't work: if price patterns contained predictive information, arbitrageurs would trade it away. It works anyway because markets aren't perfectly efficient, and behavioral patterns repeat. Blind spot: completely ignores why a company is valued the way it is. A technically bullish stock can be a fundamentally bankrupt company.

Fundamentals Analyst — Evaluates profitability (ROE, margins), growth (revenue, earnings, book value), financial health (debt ratios, cash flow), and valuation ratios (P/E, P/B, P/S). This is Benjamin Graham-style analysis, the intellectual foundation of value investing. Blind spot: timing. A fundamentals analyst can be "right" about a stock's intrinsic value for years before the market agrees. As Keynes observed, "the market can stay irrational longer than you can stay solvent."

Sentiment Analyst — Processes news sentiment via NLP, tracks insider trading patterns, monitors social media signals, and attempts to gauge market psychology. Blind spot: can't distinguish between rational crowd wisdom and irrational crowd panic. Sentiment was overwhelmingly positive for WeWork before its implosion. It was overwhelmingly negative for Tesla before its 10x run.

Valuation Agent — Performs discounted cash flow (DCF) analysis, calculates owner earnings, and compares intrinsic value to market price to identify mispricings. Blind spot: DCF models are exquisitely sensitive to assumptions about discount rates and terminal growth. Change the discount rate by 1% and the fair value changes by 20-40%. The model produces precise numbers from imprecise inputs — a classic case of false precision.

Risk Manager — Implements position sizing rules, monitors portfolio exposure, sets risk limits, and manages drawdown protection. This is the gatekeeper that prevents any single agent's conviction from destroying the portfolio. Blind spot: can't protect against black swan events — risks that are, by definition, outside the model's distribution. Every risk model works until the scenario it wasn't designed for arrives.

Portfolio Manager — Consolidates all agent signals, makes final trading decisions, and executes recommendations. It must weigh contradictory inputs and produce a single actionable decision. Blind spot: when all agents are collectively wrong in the same direction — a scenario that happens more often than the architecture suggests, because all agents share the same underlying data and the same LLM reasoning patterns.

==The multi-agent design principle here: non-overlapping blind spots. No single agent can see the full picture, but their blind spots are different.== The technical analyst's blind spot (fundamentals) is the fundamental analyst's strength. The sentiment analyst's blind spot (rationality assessment) is partially covered by the valuation agent's quantitative grounding. The architecture works when each agent's weakness is another agent's strength.

The Signal Reconciliation Problem

The hardest engineering problem in a multi-agent trading system isn't building the individual agents — it's deciding what to do when they disagree.

Consider a concrete scenario: NVIDIA, January 2025. The technical analyst sees a bullish momentum pattern (signal: BUY, confidence: 0.82). The fundamentals analyst notes a P/E ratio of 65x, well above historical norms (signal: HOLD, confidence: 0.71). The sentiment analyst detects overwhelmingly positive AI hype (signal: BUY, confidence: 0.88). The valuation agent's DCF model suggests the stock is overvalued by 30% (signal: SELL, confidence: 0.64).

Three approaches to reconciliation:

Democratic (majority vote): Two BUYs, one HOLD, one SELL → BUY. Simple but ignores the magnitude of disagreement. The valuation agent's SELL signal, even as a minority, might be the most important signal in the ensemble.

Meritocratic (confidence-weighted): Weight each signal by the agent's stated confidence. BUY: 0.82 + 0.88 = 1.70. HOLD: 0.71. SELL: 0.64. → Strong BUY. But this assumes confidence scores are well-calibrated — a dangerous assumption for LLMs. KalshiBench (2024) tested frontier models on prediction market questions with verifiable outcomes and found all models exhibit systematic overconfidence, with a 12-percentage-point average gap between stated confidence and actual accuracy. Even the best model (Claude Opus 4.5, ECE: 0.120) fell well short of human superforecasters (ECE: 0.03-0.05). Reasoning-enhanced models actually had worse calibration. An LLM that says "0.88 confidence" doesn't mean there's an 88% probability of being correct.

Risk-adjusted (conservative): Any SELL signal from any agent triggers caution. The portfolio manager reduces position size proportional to the disagreement magnitude. → Small BUY with tight stop-loss. This is the approach most production trading systems use because it prioritizes capital preservation over return maximization.

==I call this the Signal Reconciliation Problem: the challenge of combining conflicting agent outputs into a coherent action when the agents themselves can't assess the reliability of their own predictions.== It's the multi-agent equivalent of the jury theorem — ensembles of independent judges outperform individuals, but only when each judge is better than random and their errors are uncorrelated. If the judges share systematic biases, the ensemble amplifies those biases.

In an LLM-based trading system, this condition is systematically violated. All agents share the same underlying model, the same training data biases, and the same tendency toward confident-sounding narratives. Their "independence" is structural (different prompts, different data inputs) but not epistemic (same reasoning patterns, same blind spots in numerical reasoning).

What Actually Happens

Let me be direct about what this system is and isn't.

The ai-hedge-fund project is an educational simulation. It uses real market data (via Financial Datasets API) and real LLM reasoning (via OpenAI/Anthropic), but it doesn't execute real trades. It's designed to demonstrate multi-agent architecture patterns, not to make money.

And that's the right framing, because LLM-based trading has fundamental limitations that no amount of architectural cleverness can fix:

LLMs can't do reliable numerical reasoning. When a fundamentals agent calculates a P/E ratio or a DCF valuation, it's generating a plausible-looking number through pattern matching, not computing it from first principles. On the FinanceBench benchmark, GPT-4 Turbo with retrieval incorrectly answered or refused to answer 81% of questions on real financial QA from public filings. An ACL 2024 analysis broke down errors: 25% wrong evidence, 25% insufficient domain knowledge, 24% pure calculation errors. The mitigation (used in the project) is to compute all numerical values deterministically in code and pass only the results to the LLM for interpretation. But this means the LLM's "analysis" is more like narration over pre-computed numbers — useful for synthesis, not for discovery.

Backtesting is seductive and dangerous. The system includes backtesting capabilities, and it's tempting to optimize until the backtest looks good. But backtesting has well-documented failure modes: overfitting (the strategy fits historical noise, not signal), look-ahead bias (using information that wouldn't have been available at trade time), and survivorship bias (testing only on stocks that still exist). Bailey et al.'s work on backtest overfitting showed that with just 5 years of data, trying more than 45 independent model configurations virtually guarantees a strategy with a backtested Sharpe ratio of 1.0 but an expected out-of-sample Sharpe of zero. The industry standard is a 50% haircut on backtested Sharpe ratios to estimate live performance.

The results speak for themselves. StockBench (2025) tested 14 frontier LLMs — including GPT-5 and Claude 4 Sonnet — on actual trading over 82 days. The equal-weight buy-and-hold baseline returned 0.4%. GPT-5 returned 0.3% — it underperformed doing nothing. The best LLM (Qwen3-235B) managed 2.4%. For comparison, Renaissance Technologies' Medallion Fund averaged 66% annually over three decades — using statistical arbitrage, not LLMs.

Agent agreement can mask collective delusion. When all six agents agree on a BUY signal, it feels like strong consensus. But if the underlying data is misleading (accounting fraud, market manipulation, structural regime change), all agents will be wrong simultaneously. The architecture provides no protection against errors in the shared data layer. This is the multi-agent version of the "garbage in, garbage out" problem, made more dangerous by the false sense of security that consensus provides.

Where This Breaks

Cost per decision. Six LLM agents analyzing a single stock requires 6+ API calls, each processing substantial context (financial data, historical analysis, agent instructions). At GPT-4o pricing, a single stock analysis costs $0.50-2.00. Across a 50-stock universe with daily rebalancing, that's $25-100/day — $9,000-36,000/year — just for the LLM inference. Traditional quant models running on local compute cost a fraction of this.

Latency. Each agent takes 3-10 seconds for analysis. Even with parallel execution, the pipeline takes 15-30 seconds per stock. In markets where high-frequency traders operate on microsecond timescales, this is an eternity. LLM-based trading is structurally unsuitable for any strategy requiring speed.

Regulatory risk is real and growing. The SEC brought its first AI-washing enforcement actions in 2024, fining two firms a combined $400,000 for falsely claiming AI capabilities they didn't have. Their 2026 examination priorities explicitly target "automated investment tools, algorithmic models, and AI-based systems." Firms cannot outsource accountability to AI systems — human oversight is required for any material trading decision. Multi-agent systems make accountability harder, not easier, because the decision emerges from agent interaction rather than a single traceable computation.

The stationarity assumption. Every agent's analysis assumes that patterns observed in historical data will continue into the future. This is true most of the time — until it isn't. Regime changes (COVID crash, rate hike cycles, geopolitical shocks) invalidate historical patterns precisely when accurate prediction matters most. No amount of multi-agent coordination helps when the underlying data distribution shifts.

LLM-specific risks. The agents inherit all standard LLM failure modes: hallucination (fabricating financial metrics), sycophancy (agreeing with the user's implied thesis), recency bias (overweighting recent data that appeared in training), and narrative bias (constructing compelling stories that aren't supported by evidence). In a domain where being wrong costs real money, these failure modes aren't abstract concerns — they're direct financial risks.

What This Teaches About Agent Architecture

The most valuable thing about building a multi-agent trading system isn't the trading system. It's what the exercise reveals about multi-agent coordination in general.

Productive disagreement is a feature, not a bug. When agents disagree, the system is working. The dangerous scenario is when they agree — because agreement might indicate genuine consensus or shared blindness, and the architecture can't distinguish between the two. Design for disagreement. Build reconciliation protocols. Treat unanimous agreement with suspicion.

Blind spot mapping beats capability stacking. Adding more agents doesn't make the system smarter. Adding agents with non-overlapping blind spots does. Before building an agent, ask: "What can this agent NOT see? Does any existing agent cover that gap?" If the answer is no, the new agent adds value. If yes, it adds noise.

Confidence calibration is the hardest unsolved problem. ==Every reconciliation strategy depends on knowing how much to trust each agent's output. LLMs are terrible at this — they sound confident when wrong and uncertain when right.== Until confidence calibration improves, any multi-agent system that weighs signals by confidence is building on sand.

Deterministic computation should stay deterministic. ==The most reliable parts of the trading system are the parts that don't use LLMs: the financial data retrieval, the ratio calculations, the position sizing math.== Use LLMs for what they're good at (synthesis, pattern recognition in unstructured data, narrative interpretation) and code for what code is good at (math, data retrieval, rule enforcement). This is Garry Tan's deterministic layer principle applied to finance.

These patterns — productive disagreement, blind spot mapping, confidence calibration, deterministic separation — apply anywhere agents must coordinate under uncertainty: medical diagnosis systems, content moderation pipelines, autonomous vehicle decision-making, security threat assessment. Finance is just the domain where the feedback loop is fastest and the tolerance for error is lowest.

==The trading system won't make you rich. But the architecture might make your next multi-agent system significantly more robust.==

Sharad Jain builds agentic AI systems in Bengaluru. He previously worked on data infrastructure at Meta and founded autoscreen.ai, a production voice AI platform. He writes about agent architecture at sharadja.in.

Further reading:

ai-hedge-fund — the open-source multi-agent trading system (55,000+ stars)
StockBench — 82-day benchmark of 14 frontier LLMs on actual trading (2025)
FinanceBench — GPT-4 fails 81% of financial QA tasks
Bailey et al., "Probability of Backtest Overfitting" — why backtested Sharpe ratios need a 50% haircut
KalshiBench — LLM confidence calibration on real prediction markets
Anthropic, "Building Effective Agents" — agent workflow patterns

Large Concept Models: What Happens When AI Stops Thinking in Words

Sharad Jain — Sat, 11 Jan 2025 09:30:00 GMT

What Happens When AI Stops Thinking in Words

Every large language model you've used — GPT-4, Claude, Gemini, LLaMA — thinks in tokens. Fragments of words. Byte-pair encodings that split "understanding" into "under" + "stand" + "ing" and process each piece through the same attention mechanism, one at a time, left to right.

This works remarkably well. ==But it imposes a constraint so fundamental that most people don't notice it: the model's unit of reasoning is smaller than the model's unit of meaning.== You think in ideas. The model thinks in syllables. Every concept must be assembled from parts, token by token, with no guarantee that the whole adds up to what you intended.

In December 2024, Meta AI published a paper that asks the obvious question: what if we just skipped the tokens entirely?

Their answer is the Large Concept Model — an architecture that reasons over sentence-level semantic embeddings rather than individual tokens. ==Instead of predicting the next word, it predicts the next idea.== The results are surprising, the limitations are instructive, and the implications are worth thinking through carefully.

The Token Bottleneck

To understand why LCMs matter, you need to understand what token-level processing actually costs you.

When a transformer processes text, it converts each token into a high-dimensional vector and computes attention across all tokens in the context. The computational cost scales quadratically with sequence length — O(n²) for n tokens. A 1,000-word essay is roughly 1,300 tokens. A 10,000-word document is 13,000 tokens. The attention mechanism must compute 169 million pairwise relationships for the longer document, compared to 1.7 million for the shorter one.

But here's the deeper issue: those 13,000 tokens don't represent 13,000 ideas. They represent maybe 200-400 distinct concepts, encoded redundantly across thousands of subword fragments. The model is spending most of its computation processing the syntax and surface form of language rather than its meaning. It's like trying to understand a symphony by analyzing individual sound waves instead of listening to the melody.

I call this the granularity mismatch: the gap between the level at which humans reason about text (sentences, paragraphs, arguments) and the level at which transformers process it (tokens, subwords, characters). Every token-level model pays this tax. The question is whether there's a way around it.

How Large Concept Models Work

Meta's LCM paper, "Large Concept Models: Language Modeling in a Sentence Representation Space" (December 2024), proposes a direct solution: move the entire reasoning process into a semantic embedding space where each point represents a complete sentence.

The architecture has three stages:

┌─────────────────────────────────────────────────────┐
│  1. ENCODE: Text → Sentence Embeddings              │
│     "The market crashed" → vector [0.23, -0.71, ...] │
│     Using SONAR encoder (1,024-dim, 200+ languages)  │
├─────────────────────────────────────────────────────┤
│  2. REASON: Embedding → Embedding                    │
│     Predict next concept from sequence of concepts   │
│     Three approaches: Base / Diffusion / Quantized   │
├─────────────────────────────────────────────────────┤
│  3. DECODE: Embedding → Text                         │
│     vector [0.41, 0.18, ...] → "Investors panicked"  │
│     Using SONAR decoder (any target language)         │
└─────────────────────────────────────────────────────┘

Stage 1 uses SONAR, Meta's multilingual sentence embedding model that maps sentences from over 200 languages into a shared 1,024-dimensional vector space. The key property: sentences with similar meanings land near each other in this space, regardless of the language they were written in. "The sky is blue" in English, Japanese, and Arabic all map to approximately the same point.

Stage 2 is where the LCM does its actual reasoning. Instead of predicting the next token, it predicts the next sentence embedding — a complete concept — given the sequence of previous concepts. This is fundamentally different from autoregressive token generation. The model operates on a sequence of maybe 30-50 concepts instead of 1,300 tokens, with each concept carrying the full semantic weight of an entire sentence.

Stage 3 decodes the predicted embedding back into natural language text using SONAR's decoder. ==Because the embedding space is language-agnostic, the same concept can be decoded into any of the 200+ supported languages — without the model ever being trained on translation explicitly.==

The researchers explored three architectures for Stage 2:

Base LCM — a standard autoregressive transformer operating on continuous embeddings. It predicts the next concept vector by attending to all previous concepts. Simple but effective for short sequences.

Diffusion-based LCM — borrows the iterative refinement process from image generation models. Instead of predicting the next concept in one shot, it starts with noise and gradually denoises it into a coherent concept embedding. This captures richer distributions over possible next concepts, which matters when the continuation is genuinely ambiguous.

Quantized LCM (QLCM) — discretizes the continuous embedding space into a vocabulary of concept tokens using a residual vector quantizer. Each sentence becomes a short sequence of discrete codes (typically 8 codes per sentence), and the model operates on these codes using standard sequence-to-sequence methods. This bridges the LCM approach with conventional language modeling techniques.

The Three Levels of Abstraction

To place LCMs in the broader landscape, it helps to think about language models as operating at different levels of abstraction:

┌─────────────────────────────────────┐
│  Level 3: Concept-Level (LCMs)      │
│  Unit: sentence embedding           │
│  Sequence: ~30-50 concepts          │
│  Captures: meaning, intent, ideas   │
├─────────────────────────────────────┤
│  Level 2: Token-Level (LLMs)        │
│  Unit: subword token                │
│  Sequence: ~1,000-100,000 tokens    │
│  Captures: syntax, grammar, style   │
├─────────────────────────────────────┤
│  Level 1: Character-Level           │
│  Unit: character/byte               │
│  Sequence: ~5,000-500,000 chars     │
│  Captures: spelling, morphology     │
└─────────────────────────────────────┘

Most of the field's energy has gone into Level 2. LCMs propose that for certain tasks — summarization, planning, cross-lingual transfer — Level 3 is simply a better place to work. You lose fine-grained control over word choice and syntax, but you gain the ability to reason about ideas directly.

This isn't a replacement for LLMs. It's a complement. Different tasks live at different abstraction levels. Generating poetry requires Level 2 (word-level craft matters). Summarizing a legal brief requires Level 3 (concept-level reasoning matters). The choice of abstraction level should match the task, not the model's architecture.

What the Results Actually Show

The Meta paper's results are both impressive and humbling. The 7-billion parameter LCM, trained on sentences extracted from a large multilingual corpus, produced these findings:

Summarization is the sweet spot. On the summary expansion benchmark, the diffusion-based LCM (specifically the "Two-Tower" variant) achieved the strongest performance. The model didn't just extract and rearrange text — it genuinely paraphrased and restructured ideas, producing summaries that read like a human wrote them rather than highlighted them. This makes architectural sense: summarization is fundamentally a concept-level operation. You're compressing ideas, not rearranging words.

Multilingual transfer is remarkable. Despite being trained primarily on English data, the LCM outperformed specialized multilingual models on most of the 45 languages tested. This result flows directly from SONAR's architecture: because the embedding space is language-agnostic, reasoning learned in one language transfers automatically to all others. The model doesn't need to "learn" each language separately — it learns to reason about concepts, and SONAR handles the translation.

The numbers are concrete. The 7B model was trained on 2.3 billion documents (2.7 trillion tokens, 142.4 billion sentence concepts) with a context window of 2,048 concepts. On CNN/DailyMail and XSum summarization benchmarks, the Two-Tower diffusion variant was competitive with instruction-tuned models including T5-3B, Gemma-7B, and LLaMA-3.1-8B — while operating on sequences an order of magnitude shorter.

Expansion is the weak spot. When asked to expand a summary into longer text, the model struggled. It repeated ideas, produced circular reasoning, and lost coherence beyond 5-7 sentences. This is the generation horizon problem: concept-level models are good at compressing information but lack the fine-grained control needed for sustained, coherent long-form generation. Token-level models don't have this problem because they control output one word at a time.

The concept space is fragile. Small perturbations in the embedding space can cause large, unpredictable changes in the decoded text. A tiny shift in the vector representing "the company's revenue grew" might decode to "the company's reputation suffered" — semantically distant but geometrically close. This embedding instability is the most fundamental limitation of the approach, and it gets worse for technical or precise content where exact wording matters.

The Planning Problem

Perhaps the most interesting section of Meta's paper is their proposal for the Large Planning Concept Model (LPCM) — a version that generates an explicit plan before producing content.

The idea is compelling: before generating a sequence of concept embeddings, first generate a high-level plan — a sequence of abstract intentions — and then fill in the concepts that satisfy that plan. This is how skilled writers actually work. You don't write an essay one sentence at a time, left to right. You outline the argument, identify the key points, and then flesh out each section.

The LPCM adds a planning layer above the concept layer:

Plan:     [introduce problem] → [present data] → [propose solution] → [acknowledge limits]
           ↓                    ↓                  ↓                    ↓
Concepts: [3-5 sentences]      [3-5 sentences]    [3-5 sentences]     [2-3 sentences]
           ↓                    ↓                  ↓                    ↓
Text:      Paragraph 1          Paragraph 2        Paragraph 3         Paragraph 4

This hierarchical approach addresses the generation horizon problem by constraining the concept sequence to follow a coherent structure. Early results show improved coherence on longer documents, though the planning module itself introduces new challenges around plan faithfulness — the model sometimes generates concepts that satisfy the local context but deviate from the original plan.

Where This Breaks

LCMs have real limitations, and understanding them clarifies when this architecture is and isn't the right choice.

Sentence-level granularity is a ceiling, not just a feature. The model can't reason about individual words. It can't write poetry where word choice matters. It can't generate code where every character is syntactically meaningful. It can't produce dialogue where rhythm and word-level pacing create voice. For these tasks, token-level models are strictly superior.

Embedding instability compounds over sequences. Each concept prediction has some error. Over a sequence of 30 concepts, these errors accumulate. By concept 20, the model may have drifted significantly from the intended meaning. This is the concept-level equivalent of the "hallucination" problem in LLMs, but harder to detect because the output can be semantically plausible while being factually wrong.

SONAR is a bottleneck. The quality of the entire system depends on SONAR's ability to faithfully encode and decode sentences. Any information that SONAR loses during encoding — nuance, emphasis, technical precision — is irrecoverable. The model can't generate what it can't represent. And SONAR's 1,024-dimensional embedding space, while powerful for general semantics, may not capture the distinctions that matter for specialized domains like legal text, medical records, or mathematical proofs.

Training data requirements are different. Token-level LLMs can be trained on raw text. Concept-level LCMs require text that has been segmented into sentences and encoded into SONAR embeddings — an additional preprocessing step that introduces its own errors and biases. Sentence boundary detection is imperfect, especially for languages with non-standard punctuation or for informal text like social media posts.

The diffusion approach is slow. Diffusion-based concept generation requires multiple denoising steps per concept (typically 10-100 steps). For a 30-concept summary, that's 300-3,000 forward passes — significantly slower than autoregressive generation. The QLCM variant is faster but sacrifices the richer distributional properties that make diffusion effective.

What This Means for the Field

==LCMs matter not because they'll replace LLMs — they won't, at least not for general-purpose language generation — but because they demonstrate that the token is not the only viable unit of reasoning for neural language models.==

This insight has implications beyond the specific LCM architecture:

Hierarchical reasoning. The most capable future systems will likely operate at multiple abstraction levels simultaneously — planning at the concept level, generating at the token level, and verifying at both. The LCM paper provides evidence that concept-level reasoning is learnable and useful, even if the current implementation is limited.

Multilingual AI without multilingual training. SONAR's language-agnostic embedding space suggests a path toward AI systems that can reason in any language after being trained in just one. This has enormous implications for languages with limited training data — the "low-resource language" problem that token-level models still struggle with.

Compression as reasoning. LCMs perform a kind of lossy compression: they encode rich, nuanced text into fixed-size vectors, reason over those vectors, and then decode them back. The quality of this compression determines the quality of the reasoning. This frames the problem differently than token-level models, and may lead to architectures that are fundamentally better at tasks that require abstraction — summarization, planning, conceptual reasoning — even if they're worse at tasks that require precision.

The granularity mismatch between human reasoning and token-level processing is real. LCMs are the first serious attempt to close it. The attempt is imperfect — fragile embeddings, limited generation horizons, slow diffusion — but the direction is right. ==We should be building models that reason about ideas, not just words.== The question is how to do it without losing the precision that makes token-level models so useful.

The follow-up work is already proving the direction right. SONAR-LLM (August 2025) eliminates the diffusion sampler entirely, building a decoder-only transformer that "thinks in sentence embeddings and speaks in tokens" — significantly outperforming both diffusion and MSE variants while achieving near-linear computational scaling up to 1M tokens. Dynamic Large Concept Models (December 2025) go further, learning semantic boundaries from latent representations rather than using fixed sentence boundaries, achieving +2.69% across 12 zero-shot benchmarks under matched compute.

==Meta's original LCM doesn't answer the granularity question. But the trajectory from LCM → SONAR-LLM → Dynamic LCM shows the field converging on an answer: the right unit of reasoning isn't fixed — it's learned.==

Sharad Jain builds AI systems in Bengaluru. He previously worked on data infrastructure at Meta and founded autoscreen.ai, a production voice AI platform. He writes about emerging AI architectures at sharadja.in.

Further reading:

Meta AI, "Large Concept Models: Language Modeling in a Sentence Representation Space" — December 2024
Meta AI, SONAR: Multilingual Sentence Embeddings — sentence-level encoder supporting 200+ languages
Anthropic, "Building Effective Agents" — contrasting approach: token-level agents with workflow patterns

The Dance of AI Agents: How Multi-Agent Systems Actually Work

Sharad Jain — Fri, 28 Jun 2024 09:30:00 GMT

The Dance of AI Agents: How Multi-Agent Systems Actually Work

Something interesting happens when you watch a great team. Each person knows their role, understands when to step in, and — more importantly — when to step back. They move with a fluid efficiency that makes complex work look simple.

I've been trying to build AI systems that work the same way. Not a single monolithic model that does everything, but a network of specialized agents that collaborate, delegate, and route around each other's limitations. The results have been surprising — both in what works and in what fails spectacularly.

==The most important lesson: multi-agent AI isn't about making systems smarter. It's about making them smaller.== ==A network of focused agents, each with a narrow responsibility and clear handoff protocols, consistently outperforms a single agent trying to do everything.== Not because the individual agents are more capable, but because the architecture constrains them to do less — and do it well.

The Single-Agent Ceiling

Most AI applications today are single-agent systems: one model, one system prompt, one set of tools. You give it a complex task, it reasons through it, calls some tools, and produces output. This works well for simple tasks. It breaks down predictably for complex ones.

The failure mode is always the same: context pollution. As the task gets more complex, the system prompt grows. More tools are added. The model must hold more state in its context window. And as we saw with the 14K Token Debt, attention degrades as context grows — the model starts losing track of its instructions, its tools interfere with each other, and output quality drops.

Anthropic documented this pattern precisely in their "Building Effective Agents" guide (December 2024). Their central recommendation: start simple, and only add complexity when simpler solutions fall short. But when you do need complexity, the answer isn't a bigger single agent — it's multiple agents working together.

The insight maps to a broader principle from software engineering: the single responsibility principle. A function that does one thing well is easier to test, debug, and maintain than a function that does ten things. The same applies to agents.

Five Workflow Patterns That Actually Work

Anthropic's taxonomy of agent collaboration patterns is the clearest framework I've seen. They distinguish between workflows (LLMs orchestrated through predefined code paths) and agents (LLMs that dynamically direct their own processes). Most production systems are workflows, not agents — and that's a good thing.

Here are the five workflow patterns, ordered by complexity:

1. Prompt Chaining

The simplest pattern: break a task into sequential steps, where each step's output feeds the next step's input. Each step can have its own focused prompt, its own validation gate, and its own error handling.

[Generate outline] → [Gate: outline valid?] → [Write section 1] → [Write section 2] → [Edit]

When to use: Tasks that decompose into fixed, predictable subtasks. Translation pipelines, content generation with review steps, multi-stage data processing.

When not to use: Tasks where the steps can't be known in advance, or where later steps might invalidate earlier ones.

2. Routing

A classifier examines the input and directs it to the appropriate specialized handler. Like a hospital triage nurse — assess, classify, route.

[User message] → [Classifier] → Sales agent
                              → Support agent
                              → Technical agent

When to use: Inputs that fall into distinct categories requiring different handling. Customer service, document processing, multi-domain assistants.

The trap: Over-routing. If your classifier needs to distinguish between 15 categories, it will misclassify frequently. Keep categories to 3-5 for reliable routing.

3. Parallelization

Run multiple agents simultaneously on different aspects of the same input, then combine results. Two variants: sectioning (divide work by subtask) and voting (same task, multiple attempts, pick the best).

                 ┌→ [Grammar check]  ─┐
[Document] ──────┼→ [Fact check]     ─┼→ [Merge results]
                 └→ [Style check]    ─┘

When to use: Tasks where independent subtasks can run concurrently, or where confidence is improved by multiple attempts. Code review, content moderation, multi-criteria evaluation.

Cost awareness: Parallelization multiplies your API costs linearly. Three parallel agents cost 3x. Make sure the quality improvement justifies the expense.

4. Orchestrator-Workers

A central orchestrator agent dynamically breaks down the task and delegates to worker agents. Unlike prompt chaining, the subtasks aren't predetermined — the orchestrator decides what to delegate based on the specific input.

[Orchestrator] → "I need these 3 things done"
     ├→ [Worker A: research]
     ├→ [Worker B: analysis]
     └→ [Worker C: writing]
          ↓
[Orchestrator] → "Now combine the results"

When to use: Complex tasks where the decomposition varies per input. Coding agents that need to read files, plan changes, implement, and test — but which files and what changes depend on the task.

The danger: Orchestrator drift. The orchestrator agent is itself an LLM, subject to the same attention degradation as any other model. If it loses track of the overall plan, workers execute correctly on the wrong tasks. Keep orchestrator context small and plan explicit.

5. Evaluator-Optimizer

One agent generates output; another evaluates it. The evaluation feeds back, and the generator iterates. This loop continues until the evaluator is satisfied or a maximum iteration count is hit.

[Generator] → [Output] → [Evaluator: score 6/10] → [Generator: improve] → [Evaluator: 9/10] → Done

When to use: Tasks with clear, measurable quality criteria. Code that must pass tests, content that must meet a rubric, translations that must match a reference style.

Always set a maximum iteration count. Without it, a generator-evaluator pair can loop indefinitely, each iteration making marginal changes that never satisfy the evaluator. I've seen this consume $50+ in API calls before anyone noticed.

The Handoff Problem

The patterns above describe how agents collaborate. ==But the hardest engineering problem in multi-agent systems isn't collaboration — it's handoffs: the moment when one agent passes control, context, and responsibility to another.==

A bad handoff looks like this: Agent A has been helping a user with a billing issue for 5 turns. Agent A realizes the problem is actually technical and routes to Agent B. Agent B has no context. It asks the user to re-explain the problem. The user is frustrated. This is worse than having no handoff at all.

A good handoff preserves three things:

Context — the full conversation history, not just a summary
State — what has been tried, what has failed, what the user's emotional state is
Intent — why the handoff is happening and what the receiving agent should do

OpenAI's Agents SDK (evolved from the experimental Swarm project) implements two handoff patterns:

The Manager Pattern — a central agent invokes sub-agents as tools, retaining control of the conversation. The sub-agent executes a focused task and returns results to the manager. The user only ever talks to the manager.

The Handoff Pattern — one agent fully transfers control to another. The receiving agent takes over the conversation directly. The original agent is no longer involved.

The manager pattern is safer (the manager maintains continuity) but creates a bottleneck. The handoff pattern is more flexible but requires careful context transfer. In practice, most production systems use the manager pattern for reliability, with handoffs reserved for clear domain boundaries (e.g., sales → support → technical).

Here's what a clean handoff implementation looks like:

from agents import Agent, Runner

triage = Agent(
    name="Triage",
    instructions="Classify the user's need. Route to sales for purchases, "
                 "support for issues. Always transfer full context.",
    handoffs=["sales_agent", "support_agent"],
)

sales_agent = Agent(
    name="Sales",
    instructions="Handle purchase inquiries. If the user has a technical "
                 "issue, transfer back to triage with context.",
    handoffs=["triage"],
)

support_agent = Agent(
    name="Support",
    instructions="Resolve customer issues. Offer refunds only after "
                 "attempting a fix. Escalate to human if unresolved "
                 "after 3 attempts.",
    handoffs=["triage"],
)

The key design choice: bounded handoff depth. An agent should hand off to at most 2-3 specialists, and handoff chains should never exceed 3 hops. Triage → Sales → Triage is fine. Triage → Sales → Support → Technical → Billing → Triage is a system that will produce circular delegation, infinite loops, and confused users.

Where Multi-Agent Systems Fail

Multi-agent architectures introduce failure modes that don't exist in single-agent systems. Understanding these is more important than understanding the successes.

A systematic analysis of multi-agent failures (Cemri et al., 2025) studied 150+ conversation traces across five production frameworks (MetaGPT, ChatDev, AutoGen, and others) and found 41-86.7% failure rates in production. They identified 14 distinct failure modes in three categories: specification errors (42%), inter-agent misalignment (37%), and verification failures (21%). ==The finding that should keep you up at night: unstructured multi-agent networks can amplify errors up to 17.2x compared to single-agent baselines.==

Cascading errors. When Agent A produces subtly wrong output and passes it to Agent B, Agent B has no way to know the input is wrong. It processes it faithfully, amplifying the error. Agent C does the same. By the end of the chain, the output is confidently, thoroughly wrong — and every agent's logs show correct execution. This is the 17.2x amplification in action.

Cost explosion. Each agent in the network makes its own LLM API calls. A 5-agent workflow processing 1,000 requests per day, with an average of 3 turns per agent, generates 15,000 API calls daily. At $0.01 per call (GPT-4o-mini), that's $150/day — $4,500/month. With GPT-4o or Claude Opus, multiply by 10-50x. Multi-agent systems can be dramatically more expensive than single-agent alternatives, and the cost scales with the number of agents, not the quality of output.

Accountability gaps. When five agents collaborate on a decision and the decision is wrong, which agent is responsible? In practice, no one debugs a multi-agent failure by reading one agent's logs — you need to trace the full decision chain across all agents. This requires end-to-end tracing infrastructure, and most teams don't build it until after their first expensive failure.

Coordination overhead. Every handoff, every parallel merge, every evaluator loop adds latency. A prompt chain of 4 agents, each taking 2 seconds, adds 8 seconds of sequential latency — enough to make interactive applications feel sluggish. Parallelization helps but doesn't eliminate the coordination cost.

The "too many agents" trap. Teams that discover multi-agent patterns often over-apply them. A 3-agent customer service system (triage, sales, support) is clean and maintainable. A 15-agent system with specialized agents for "greeting," "empathy," "product recommendation," "upselling," and "farewell" is over-engineered and harder to debug than the monolithic system it replaced. Andrew Ng's practical guidance on agentic design patterns emphasizes starting with the simplest agent architecture that solves the problem.

When to Use What

The decision tree is simpler than most frameworks suggest:

For context on what "works in production" actually looks like: Salesforce's Agentforce, the largest documented agentic deployment, handled 1.5 million+ support requests in its first year with an 84% resolution rate without human intervention and only a 4% handoff rate. But it's a carefully constrained workflow system, not an autonomous agent swarm.

Use a single agent when: the task is straightforward, fits in one context window, requires one domain of expertise, and doesn't need collaboration with other systems. This covers 80% of real-world use cases.

Use prompt chaining when: the task has clear sequential steps with validation gates between them. This is the first multi-agent pattern you should reach for.

Use routing when: inputs fall into 3-5 distinct categories that need genuinely different handling. Not different tones — different tools, different knowledge, different procedures.

Use parallelization when: you need speed (run subtasks concurrently) or confidence (multiple attempts at the same task). Be prepared for the cost multiplier.

Use orchestrator-workers when: the task decomposition varies per input and can't be predetermined. This is the most powerful pattern and the most dangerous — orchestrator drift is real.

Use evaluator-optimizer when: you have clear, measurable quality criteria and the cost of iteration is justified by the cost of a wrong answer.

Use full handoff-based multi-agent systems when: you need agents with genuinely different capabilities (different tools, different models, different system prompts) that must transfer control based on runtime decisions. This is the most complex pattern and should be your last resort, not your first.

Building for Collaboration

If you're building multi-agent systems today, a few hard-won lessons:

Design the handoffs first. Before you build any agent, design the handoff protocol. What context gets transferred? What state is preserved? What triggers a handoff? The handoff design determines whether your system feels seamless or fragmented.

Trace everything. Build end-to-end tracing before you build the agents. Every handoff, every tool call, every decision should be logged with full context. You will need this when things go wrong — and they will go wrong.

Start with workflows, not agents. ==Predefined code paths with LLM steps are more reliable, more debuggable, and cheaper than autonomous agents.== Only use true autonomous agents when the task genuinely can't be decomposed in advance.

Measure the right thing. Don't measure how many agents you have. Measure task completion rate, average cost per task, and time-to-resolution. A 2-agent system that solves 95% of problems for $0.05 each is better than a 10-agent system that solves 98% for $2.50 each.

==The dance of AI agents isn't about choreographing ever-more-complex routines. It's about knowing when a simple two-step is better than a waltz — and having the discipline to keep it simple until simplicity genuinely isn't enough.==

Further reading:

Anthropic, "Building Effective Agents" — the definitive taxonomy of agent workflow patterns
OpenAI, Agents SDK — production agent framework with handoffs and guardrails
Andrew Ng, "Agentic Design Patterns" — four foundational patterns for agent systems
OpenAI, "A Practical Guide to Building Agents" — enterprise agent architecture guide

Building a Salesforce Slack Assistant: Lessons in AI-Powered Enterprise Tools

Sharad Jain — Thu, 27 Jun 2024 09:00:00 GMT

Building a Salesforce Slack Assistant: Lessons in AI-Powered Enterprise Tools

Introduction

When we set out to build a Salesforce assistant for Slack, we knew we were venturing into relatively uncharted territory. The idea seemed simple: allow our sales team to query Salesforce data directly from Slack. But as with most seemingly simple ideas in software, the devil was in the details.

The Genesis

Our sales team was spending too much time switching between Slack and Salesforce. ==This context-switching was more than just an annoyance; it was a real drag on productivity.== We thought, "What if we could bring Salesforce to where our team already spends most of their day?"

Architectural Decisions

The Stack

We settled on a Python backend, leveraging the OpenAI API for natural language processing, and the Salesforce API for data retrieval. Here's why:

Python: Fast development, rich ecosystem for both AI and Salesforce integrations.
OpenAI API: State-of-the-art language models for understanding complex queries.
Salesforce API: Robust and well-documented, essential for reliable data retrieval.

The Flow

The basic flow of our assistant works like this:

User sends a message in Slack.
Our server processes the message.
OpenAI Assistant interprets the query.
If needed, we fetch data from Salesforce.
We generate and send a response back to Slack.

Sounds simple, right? But each of these steps hides layers of complexity.

Challenges and Solutions

1. Context Management

One of the first hurdles we hit was maintaining context across messages. We needed our assistant to remember previous queries within a conversation. Solution: We implemented a thread management system using Python's shelve module. Each Slack thread gets a corresponding entry in our database, allowing us to maintain context efficiently.

2. Query Translation

Turning natural language into SOQL (Salesforce Object Query Language) queries was our next big challenge.

Solution: ==We leveraged OpenAI's function calling capabilities.== We defined custom functions that the AI could "call", effectively translating natural language into structured queries.

3. Data Visualization

Raw data is rarely what users want. Often, they need charts or graphs to make sense of the numbers.

Solution: We integrated a chart generation service. When the AI detects that a visual representation would be beneficial, it triggers this service to create and send images directly in Slack.

The Architecture in Detail

The system follows a modular architecture: Slack receives user messages and forwards them to our Python backend, which orchestrates between the OpenAI API (for natural language understanding) and the Salesforce API (for data retrieval). Responses flow back through the same pipeline, with a caching layer to minimize redundant API calls.

Performance Considerations

As with any system that interfaces with multiple external APIs, performance was a key concern. We implemented several optimizations:

Caching frequently requested data.
Asynchronous processing for long-running queries.
Intelligent query optimization to minimize Salesforce API calls.

We track response latency, cache hit rates, and API call volume in real time — this lets us quickly spot bottlenecks and tune the system before users notice degradation.

User Experience

We paid special attention to the user experience, aiming to make interactions with our assistant as natural as possible. A typical interaction starts with a natural-language question in Slack, which the assistant interprets, queries Salesforce, and returns a formatted response — all within the same thread so context is never lost.

Evolution and Future Plans

Our Salesforce Slack Assistant wasn't built in a day. It evolved over time, and continues to evolve. We started with simple text queries, added chart generation, then context-aware threading, and most recently, proactive notifications. Future plans include enhanced AI capabilities and multi-platform support.

Lessons Learned

==Start simple, but design for complexity: Our initial MVP was basic, but we architected it in a way that allowed us to add features without major rewrites.==
User feedback is gold: Some of our most useful features came directly from user suggestions.
==AI is powerful, but needs guardrails: We learned to carefully define the boundaries of what our AI assistant can do to prevent unexpected behavior.==
==Performance matters more than you think: In a chat interface, even small delays can significantly impact user experience.==

Conclusion

Building this Salesforce Slack Assistant has been a journey of continuous learning and improvement. It's a testament to the power of bringing AI into enterprise tools, but also a reminder of the challenges that come with it.

As we continue to refine and expand our assistant, we're excited about the potential impact on not just our sales team, but on how enterprises interact with their data as a whole.

==Remember, the goal isn't just to build cool tech—it's to solve real problems for real users.== Keep that in mind, and the rest will follow.

Building AI-first Multiplayer Writing Tool for Creative Teams

Sharad Jain — Wed, 15 May 2024 14:00:00 GMT

The Shift : Writing in the age of abundance

In the age of AI, online writing is undergoing a significant shift. ==While AI can generate content faster and more efficiently than humans, it lacks the ability to capture the unique perspectives and experiences that only human writers can provide.== As a result, the Internet is likely to reward writing that prioritizes uniqueness and beauty over mere smartness and speed.

In today's fast-paced digital landscape, collaboration has become a crucial aspect of content creation. Startup founders, VCs, and professionals across various industries are embracing tools that enable real-time collaboration, as they strive to generate ideas, write content, and refine it efficiently. One such platform that has gained significant traction in the design industry is Figma, a collaborative design tool valued at $10 billion. Read this blog post to learn more about Figma's success story.

In this post I want to explore building an AI-native collaborative text editor application, utilizing Jamsocket, Next.js, and OpenAI's Assistant API. Such an interface not only facilitates real-time brainstorming, writing, and content refinement among teams but also unlocks unparalleled creativity, enriched by intelligent suggestions and support from an AI co-writer.

==AI can't replicate your unique voice and perspective.== Your lived experiences, ideas, and writing style are one-of-a-kind, and that's what will make your content stand out in a sea of AI-generated text.
Readers crave high-quality, memorable content. With the abundance of mediocre writing online, people are hungry for exceptional ideas and beautifully crafted prose. By focusing on quality over quantity and authorship over speed, you can capture and hold your audience's attention.
==LLMs are like stochastic parrots, repeating patterns from their training data without true understanding.== They generate text that may seem passable at first glance but lacks the depth, clarity, and intentionality of human writing. The difference is akin to a blurry photograph versus a carefully crafted painting - while the overall picture might be recognizable, the details are distorted and the meaning is lost.
Shareable content is king. When you create content that is both unique and beautiful, it becomes more shareable. People are more likely to engage with and spread the word about writing that resonates with them on a personal level.

To succeed as a writer in the age of AI, embrace your singularity and hone your craft. Collaborate with AI agents to enhance your unique voice and perspective. These AI collaborators can help you refine your ideas, suggest improvements, and even challenge you to think more deeply about your writing.

==Don't try to outsmart or outpace AI; instead, focus on creating content that only you can produce, enhanced by the power of AI assistance.== Use AI agents as a tool to augment your creativity and writing process, helping you craft more compelling and engaging content.

==By leveraging the strengths of both human creativity and AI capabilities, you'll be able to carve out a niche for yourself and build a loyal following of readers who appreciate your distinct voice and high-quality work.== Embrace the collaboration between human ingenuity and artificial intelligence to create truly exceptional content that stands out in the digital landscape.

Building a Real-Time Collaborative Writing Tool

Here's a concise overview of how we can design and implement this AI-powered collaborative writing tool:

Architectural Design

The application leverages a modular and scalable architecture:

Frontend (Next.js + Quill): Next.js powers the user interface, integrating Quill as the rich text editor for seamless collaboration.
AI Integration (OpenAI Assistant API): The OpenAI Assistant API provides contextual suggestions, corrections, and content generation to enhance the writing process.
Real-time Collaboration (Jamsocket): Jamsocket enables real-time synchronization of document changes and user presence across connected clients.
Backend Services (Next.js + Docker): The backend, including session management and AI request handling, is built using Next.js and deployed as containerized services using Docker.

Implementation Steps

Set up Next.js: Create a new Next.js project and configure the necessary routes and API endpoints.
Integrate Quill: Customize the Quill text editor to fit the application's needs and ensure responsiveness.
Connect Jamsocket: Implement real-time collaboration by setting up a Jamsocket service and managing user sessions.
Integrate OpenAI Assistant API: Develop functions to fetch and display AI-powered writing suggestions based on user prompts.
Deploy with Docker: Containerize the backend services using Docker for easy deployment and scalability.

By following this approach and leveraging the mentioned tech stack, you can build a feature-rich, AI-first collaborative writing tool that empowers creative teams to produce high-quality content efficiently.

🛠️ Tech Stack Used:

Technology	Description
Jamsocket	For real-time collaboration and shared state management
Next.js	For building the server-side and client-side components
OpenAI Assistants API	Integrates AI assistants for enhanced writing support
Quill	Powers the text editor and syncs edits
Tailwind CSS	For styling the application
Astro	For the overall project setup and configuration
Docker	For containerizing and deploying the session backend
Jamsocket CLI	For managing the deployment to Jamsocket

Laying the Foundation

Let's start by setting up the project structure. We'll be using a typical Next.js directory layout, with a few key components:

src/app/page.tsx: This is the server-side component responsible for spawning the session backend on Jamsocket.
src/components/Home.tsx: The main client-side component that will handle the bulk of the application's functionality.
src/session-backend/index.ts: The server-side logic for managing the shared state and real-time communication.

We'll also include some helper components like TextCanvas.tsx to encapsulate the text editor and synchronize edits between clients.

Implementing Session Backend

==The session backend is the heart of our collaborative application.== It acts as a stateful layer between the clients and the document storage, maintaining the shared state of the text document and handling real-time updates between connected users.

Here's a simplified version of the session backend implementation in session-backend/index.ts:

// ... imports

const io = new Server(8080);
let documentContent = ""; // Initial document state

io.on("connection", (socket: Socket) => {
  socket.emit("snapshot", documentContent);

  socket.on("update-document", (content: string) => {
    documentContent = content;
    socket.broadcast.emit("update-document", content);
  });

  // ... other event handlers
});

When a client connects, it receives a snapshot of the current document content. When a client updates the document, the session backend updates the shared state and broadcasts the changes to all connected clients.

Architecture

Integrating OpenAI Assistant

To provide AI-powered assistance during the collaborative writing process, we'll integrate the OpenAI Assistant API. First, we'll set up the API key in a .env file:

OPENAI_API_KEY=[YOUR_OPENAI_API_KEY]

Then, in the session-backend/index.ts file, we'll create an OpenAI Assistant instance with instructions tailored for text editing collaboration:

const assistant = await openai.beta.assistants.create({
  instructions: `You are an AI assistant for collaborative writing. 
    Help users with tasks like:
    - Suggesting synonyms and alternative phrasings.
    - Correcting grammar and spelling mistakes.
    - Generating content based on user prompts.
    - Providing feedback on the overall structure and flow of the document.`,
  model: "gpt-4-1106-preview",
  // ... other parameters
});

We'll then modify the handleUserPrompt function to send user prompts to the Assistant and process the responses to update the document state. This might involve parsing the Assistant's response and applying the suggested changes to the shared document content.

Building the Text Editor Component

The heart of our collaborative text editor is the TextCanvas.tsx component. We'll use a library like Quill to handle real-time editing and synchronization, and integrate Jamsocket's useEventListener and useSend hooks to receive updates from and send events to the session backend.

Here's a complete example of the TextCanvas.tsx component:

import React, { useEffect, useRef } from "react";
import Quill from "quill";
import "quill/dist/quill.snow.css";
import { useEventListener, useSend } from "@jamsocket/socketio";
import type { User } from "../types";

interface TextCanvasProps {
  users: User[];
  onCursorMove: (position: { x: number; y: number } | null) => void;
}

export function TextCanvas({ users, onCursorMove }: TextCanvasProps) {
  const editorRef = useRef<HTMLDivElement>(null);
  const sendEvent = useSend();

  useEffect(() => {
    if (!editorRef.current) return;

    const editor = new Quill(editorRef.current, {
      modules: {
        toolbar: [
          // ... toolbar options
        ],
      },
      theme: "snow",
    });

    editor.on("text-change", () => {
      const content = editor.root.innerHTML;
      sendEvent("update-document", content);
    });

    editor.on("selection-change", (range) => {
      onCursorMove(range ? { x: range.index, y: range.index + range.length } : null);
    });

    return () => {
      editor.off("text-change");
      editor.off("selection-change");
    };
  }, [sendEvent, onCursorMove]);

  return <div ref={editorRef} />;
}

The TextCanvas component sets up a Quill editor instance and listens for text changes and cursor position updates. When the text is updated, it sends an update-document event to the session backend. When the cursor position changes, it calls the onCursorMove callback to update the cursor position for other users.

Real-Time Updates and Presence

To handle real-time updates and presence information, such as cursor positions and user avatars, we'll use Jamsocket's useEventListener and useSend hooks in the Home.tsx component:

useEventListener<User>("cursor-position", (user) => {
  // Update the user's cursor position in the local state
  // and display it on the TextCanvas
});

// ...

const handleCursorMove = (position: { x: number; y: number } | null) => {
  sendEvent("cursor-position", { x: position?.x, y: position?.y });
};

These hooks make it easy to synchronize the state between the session backend and the connected clients, creating a truly collaborative experience.

Deploying to Jamsocket

Use Jamsocket to manage shared state and real-time communication through session backends.

// Initialize Jamsocket session
const session = Jamsocket.initSession('your-session-id');
session.on('update', (data) => {
  // Handle real-time updates
});

Integrate OpenAI Assistant API to provide AI-powered suggestions and assistance during collaborative writing.

// Fetch AI suggestions
async function fetchAISuggestions(prompt) {
  const response = await fetch('https://api.openai.com/v1/chat/completions', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Authorization': 'Bearer your-openai-api-key'
    },
    body: JSON.stringify({
      model: 'gpt-4o-mini',
      messages: [{ role: 'user', content: prompt }],
      max_tokens: 100
    })
  });
  const data = await response.json();
  return data.choices[0].text;
}

Build a robust text editor component using libraries like Quill and synchronize edits using Jamsocket hooks.

// Initialize Quill editor and sync with Jamsocket
useEffect(() => {
  const editor = new Quill('#editor', { theme: 'snow' });
  editor.on('text-change', (delta, oldDelta, source) => {
    if (source === 'user') {
      session.send('text-change', delta);
    }
  });
}, []);

Handle presence information, such as cursor positions and user avatars, to enhance the collaborative experience.

// Update cursor position for all users
session.on('cursor-move', (userId, position) => {
  // Update cursor position in the UI
});

Deploy the session backend to Jamsocket for scalability and performance.

# Deploy session backend to Jamsocket
jamsocket deploy --service your-service-name --image your-docker-image

So, there you have it. Stick to the guide, throw in your own twist, and boom - you've got yourself a slick, real-time text editor that's not just smart but also plays well with others. Dive in, mess around, and make something cool.

How Midjourney's Discord Became a Phenomenon: A Lesson in Community-Driven Growth

Sharad Jain — Sat, 20 Apr 2024 10:00:00 GMT

How Midjourney's Discord Became a Phenomenon: A Lesson in Community-Driven Growth

When Midjourney launched its AI-powered image generation tool, few could have predicted that a Discord server would become the epicenter of a new artistic movement. Yet, that's precisely what happened. This unexpected success story offers valuable insights into community-driven growth and the power of shared creativity.

Embracing Imperfection: The Power of Rapid Iteration

==Conventional wisdom suggests that successful tech products require polished interfaces and extensive marketing.== ==Midjourney took a different approach, focusing instead on two critical elements: empowering users and fostering connections between them.==

Discord might seem an unconventional choice for an AI art platform. It's often messy, noisy, and can be overwhelming for new users. However, this apparent weakness became Midjourney's strength. ==The platform's inherent "messiness" allowed for rapid iteration and real-time observation of user behavior.==

Key Insight: When building something innovative, excessive polish can be counterproductive. The time spent refining can delay crucial learning from user interactions.

By leveraging Discord, Midjourney could quickly adapt based on user experiments, challenges, and creative breakthroughs.

Community as a Core Feature

The true magic of Midjourney's Discord lies in its ability to transform users into a vibrant community. ==Unlike traditional products where users operate in isolation, Midjourney's Discord makes sharing the default.== Every generated image is visible to the entire community, creating two powerful effects:

A constant stream of inspiration: Users can see not just the tool's potential but also how others are pushing its boundaries.
A sense of collective exploration: Users feel part of a movement rather than just consumers of a product.

The Impact of Community-Driven Growth

This approach led to explosive growth. Midjourney's Discord quickly became one of the largest on the platform, boasting millions of highly engaged users. These weren't casual visitors; they were dedicated explorers returning daily to push the limits of AI-generated art.

Lessons for Platform Builders

Midjourney's success offers valuable lessons for those building new platforms:

==Prioritize user empowerment and community connection over interface perfection.==
Embrace the "messiness" of rapid iteration and learning.
Make sharing and collaboration core features, not afterthoughts.
Create an environment that fosters collective exploration and creativity.

Navigating the Challenges of Success

Of course, this approach isn't without its challenges. As Midjourney grows, it faces issues like:

Potential overwhelm for new users
Difficulties in maintaining focus amidst constant activity
Increased moderation needs for a large, active community

However, these are ultimately positive problems stemming from success rather than failure.

Embracing Community-Driven Innovation

Midjourney's story isn't just about AI-generated art. It's about creating a space where millions can come together to explore, create, and inspire each other. For anyone building community-driven platforms, the lesson is clear: focus on empowering users and fostering connections. The results may surprise you.

==In the end, Midjourney's phenomenal growth demonstrates that in the age of AI, the most powerful scaling factor isn't just technology—it's human creativity unleashed at scale.==

OpenAI API vs Azure OpenAI: Choosing the Right Platform for Your AI Projects

Sharad Jain — Mon, 15 Apr 2024 10:00:00 GMT

OpenAI vs Azure OpenAI: A Tale of Two AI Platforms

AI is eating the world. And at the forefront of this revolution are two key players: OpenAI and Azure OpenAI. They're both serving up some seriously powerful language models, but they're not quite the same dish. Let's dig in and see what sets them apart.

The Lay of the Land

OpenAI is the OG here. They're the ones who cooked up those mind-bending models like GPT-4, DALL-E, and Whisper. If you're a dev or a business looking to sprinkle some AI magic into your app, their API is your go-to buffet.

Azure OpenAI, though? That's a different beast. ==It's what you get when Microsoft and OpenAI decide to join forces.== Think of it as OpenAI's models, but wrapped in Microsoft's cozy Azure blanket. It's got all those enterprise bells and whistles that big companies drool over.

So, which one's right for you? Well, that depends. Let's break it down.

Key Differences

Let's talk about the real differences between Azure OpenAI and OpenAI. It's not just about features; it's about what matters to you as a user or a business.

1. Security and Compliance

Azure OpenAI is the clear winner here. They've gone all-in on security:

They encrypt everything. Your data is locked down tight within your Azure subscription.
They've got all the fancy compliance certifications. SOC2, ISO, HIPAA, CSA STAR - the works.
They even support private networks. VNETs, private endpoints - you name it.

==OpenAI cares about privacy too, but they're not playing in the same league as Azure when it comes to enterprise-grade security.==

2. Data Handling and Privacy

Here's the deal:

With Azure OpenAI, your data stays put. It doesn't leave the Azure environment, and Microsoft isn't using it to make their AI smarter.
OpenAI has good privacy policies, sure. But you're not getting the same level of control over your data as you would with Azure.

3. Integration and Ecosystem

==If you're already knee-deep in Microsoft's world, Azure OpenAI is a no-brainer.== It plays nice with all the other Azure toys. OpenAI's API is more of a lone wolf. It's flexible, sure, but it's not part of a bigger ecosystem.

4. Model Deployment and Versioning

Azure OpenAI: Requires deploying specific model versions, which may need manual updates when newer versions are released.
OpenAI: Offers model names that always point to the latest version, simplifying updates.

5. Performance

Azure OpenAI is faster. Much faster. In benchmarks:

GPT-3.5-Turbo on Azure responds about 8x quicker than OpenAI.
GPT-4 on Azure is roughly 3x faster.

==This matters. A lot. Especially at scale.==

6. Accessibility and Approval Process

Getting started with these services differs:

Azure OpenAI makes you jump through hoops. You apply, you wait, you get approved (maybe).
OpenAI? Sign up and go. It's that simple.

7. Pricing and Billing

The base costs are similar, but:

Azure might cost more due to infrastructure overhead.
But Azure offers better deals for big enterprises. Volume discounts, flexible agreements. That sort of thing.

8. Features and Customization

Azure OpenAI has some tricks up its sleeve:

You can fine-tune certain models. Tailor them to your needs.
It filters content out of the box. You decide how strict.
You can query local files right through the API. Handy.

These aren't game-changers, but they're nice to have. Especially if you're doing something specific.

Making the Right Choice

So which should you choose? It depends on what you're after.

==Go with Azure OpenAI if you're a big company that cares about security.== If you're already using Azure, it's a no-brainer. It's faster too, which matters if you're doing a lot of AI stuff.

==Pick OpenAI if you just want to get started quickly.== It's better for small projects or if you're just messing around. The API is simpler, and you get the newest models right away.

Conclusion

Both are good, but for different things. Azure OpenAI is for big companies that need serious security and already use Azure. OpenAI is easier to use and works for more people.

In the end, pick based on what you need. How secure does it need to be? What are you already using? How much control do you want?

Whichever you choose, you're getting some pretty amazing AI tech. It's kind of incredible when you think about it.

Citations: [1] https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/switching-endpoints [2] https://www.uscloud.com/blog/the-differences-between-openai-and-microsoft-azure-openai/ [3] https://www.private-ai.com/en/2024/01/09/openai-vs-azure-openai/ [4] https://learn.microsoft.com/en-us/answers/questions/1190853/azure-openai-service-vs-openai [5] https://drlee.io/openai-vs-azure-openai-a-deep-dive-into-their-differences-26f1107677e?gi=994361760e27 [6] https://blogs.visigo.com/chriscoulson/azure-openai-service-vs-openai-api/ [7] https://learn.microsoft.com/en-us/azure/ai-services/openai/overview?WT.mc_id=academic-105485-koreyst [8] https://www.linkedin.com/pulse/exploring-differences-between-openai-azure-which-one-right-rok-rogelj [9] https://www.advancinganalytics.co.uk/blog/2023/4/24/azure-openai-vs-openai-whats-the-difference

Universal Power of Computation

Sharad Jain — Fri, 22 Mar 2024 09:30:00 GMT

In his seminal work "The Beginning of Infinity," philosopher David Deutsch posits that the universality of computation underlies the open-ended nature of human progress. As we stand on the precipice of a new era, dominated by the rapid advancement of large language models (LLMs), this concept takes on a newfound significance for tech entrepreneurs.

LLMs, with their ability to process and generate human-like text, are quickly evolving into universal tools for computation. ==These AI-powered systems are not merely language models; they are becoming all-purpose problem-solving engines, capable of tackling an ever-expanding range of challenges across industries.==

For entrepreneurs, this presents an unprecedented opportunity to leverage the power of LLMs to create value in novel ways. By harnessing the universality of computation embodied in these advanced models, startups can develop innovative solutions that transcend traditional boundaries and disrupt established markets.

One of the key advantages of LLMs is their adaptability. ==Unlike narrow AI systems designed for specific tasks, LLMs can be fine-tuned and applied to a wide variety of domains, from content creation and customer support to research and development.== This versatility allows entrepreneurs to quickly prototype and iterate on ideas, testing hypotheses and refining solutions at an unprecedented pace.

Moreover, LLMs have the potential to democratize access to cutting-edge computational capabilities. Historically, advanced AI systems have been the domain of large corporations and well-funded research institutions. ==However, as LLMs become more accessible through open-source initiatives and commercial offerings, even small startups and solo entrepreneurs can harness their power to drive innovation.==

==The universality of computation also implies that the potential applications of LLMs are virtually limitless.== As these models continue to improve in terms of accuracy, reasoning, and domain-specific knowledge, they will enable entrepreneurs to tackle increasingly complex problems and create value in ways we have yet to imagine.

However, to fully capitalize on the potential of LLMs, entrepreneurs must adopt a mindset of continuous learning and adaptation. The AI landscape is evolving at a breakneck pace, and staying ahead of the curve requires a commitment to staying informed about the latest developments, experimenting with new techniques, and collaborating with experts across disciplines.

==In conclusion, the universality of computation, as embodied in advanced LLMs, is set to revolutionize the landscape of entrepreneurship and innovation.== By leveraging these universal tools for computation, tech entrepreneurs can develop novel solutions, disrupt markets, and drive the progress of knowledge. As we stand at the beginning of this infinite journey, the possibilities are truly unbounded.

Embracing the Art of Writing: Lessons from Kevin Kelly, Founder of Wired

Sharad Jain — Thu, 21 Dec 2023 10:00:00 GMT

The Architecture of Words: Kevin Kelly's Writing Philosophy

Picture a towering skyscraper. Its majesty and elegance are not merely a product of the materials it's built from, but the invisible structure that binds it all together. This is the lens through which Kevin Kelly, the visionary founder of Wired Magazine, views the art of writing.

Imagine writing as a towering skyscraper. Its majesty and elegance are not merely a product of the words it's built from, but the invisible structure that binds it all together. This is how Kevin Kelly, the visionary founder of Wired Magazine, crafts his writing. ==The structure is so seamless that it recedes into the background, allowing the reader to immerse themselves in the ideas being conveyed.== This approach transcends the mere arrangement of words, focusing instead on refining and presenting ideas in their most captivating form.

Just as an architect designs a building with a specific purpose and audience in mind, Kelly crafts his essays. A clear, guiding structure navigates the reader through a journey of idea exploration. This approach underscores the critical role of thoughtful organization and presentation in the craft of writing.

Provoking Thought by Challenging Conventions

Kelly's writing doesn't just convey information; it sparks curiosity, provokes thought, and stimulates intellectual exploration. A piece of writing that causes a bit of disturbance or discomfort in the reader's mind is not necessarily a negative thing. Instead, it can be a sign of reaching new depths in thought, of pushing the reader out of their comfort zone and into a space of deeper understanding and reflection.

This approach is not about being controversial for the sake of it. ==It's about challenging the status quo, questioning accepted norms, and encouraging readers to think critically.== It's about using writing as a tool to advance conversations, to push boundaries, and to explore new perspectives. This commitment to challenging conventional thinking is a key aspect of Kelly's writing philosophy and is reflected in his work.

The Mirror of the Self: Writing as a Tool for Self-Discovery

Kelly's philosophy of writing extends beyond the mere arrangement of words. He views writing as a mirror, a tool for self-discovery and introspection. The act of writing, especially when done consistently, can reveal the deepest layers of the self.

Imagine staring at a blank white page, pushing yourself to fill it with words, not just once or twice, but hundreds of times. The first few attempts might be filled with pretense, a carefully crafted facade. But as you continue, as you reach your 200th piece, the truth begins to seep through. The act of writing, in its arduous and relentless nature, peels away the layers of pretense.

==The blank white page becomes a mirror, reflecting who you truly are.== It answers questions like: What do you actually want? Who do you actually want to be? It's a process of self-discovery, a journey towards authenticity.

==Kelly believes that every person who has been writing for a few years is closer to their true self, having shed the fake and manufactured mask they used to wear.== This transformation is not just about becoming a better writer, but about becoming a more authentic individual.

In essence, Kelly's philosophy of writing is a call to writers to be true to themselves. It's a call to embrace the challenge of originality, to dare to be different, and to have the courage to express one's unique perspective and voice. This commitment to honesty and originality is a key aspect of Kelly's writing philosophy and is reflected in his own work, which is renowned for its insightful, thought-provoking, and often unconventional perspectives.

Writing as a Path to Self-Discovery and Personal Growth

Despite his initial reluctance as a writer, Kelly has come to recognize and appreciate writing as a powerful tool for self-discovery and personal growth. He describes the act of writing as a deeply introspective process, a journey of exploration where he uncovers and articulates his deepest thoughts, beliefs, and convictions. It's through this process that he discovers what he truly thinks about a subject, often surprising himself with the depth and complexity of his own understanding.

But it's not just about understanding oneself. Kelly believes that writing also allows him to communicate his insights and perspectives to others in a way that is both meaningful and impactful. It's a way for him to share his unique worldview, to contribute to the collective wisdom of humanity, and to engage in the larger conversation of ideas. This transformative journey, from the initial spark of an idea to the final crafted piece of writing, is what elevates writing from a mere task to a profound, enriching experience.

The Art and Importance of Editing

Kelly places a significant emphasis on the editing process. He doesn't see it as a mere final touch or a cursory review before publishing. Instead, he views it as an integral part of the creative process, a vital component that is as important as the initial act of writing itself.

In his perspective, editing is a continuous dance between writing and revising. It's about meticulously going through each sentence, each word, and ensuring that they contribute meaningfully to the narrative. It's about refining the raw material of thoughts and ideas, shaping them into a coherent and compelling piece of writing.

This approach to editing reflects Kelly's belief in the power of refinement in the creative process. ==It's not just about getting the words down on paper, but about polishing them, honing them until they shine with clarity and precision.== It's about making sure that every word, every sentence, every paragraph serves a purpose and adds value to the overall narrative.

In essence, for Kelly, editing is not just a task to be completed, but an art to be mastered. It's a crucial part of the writing process that requires as much thought, effort, and creativity as the initial act of writing itself.

The Value of Practical and Helpful Writing

Kelly notes that his writing is more successful when it veers towards being practical and helpful. This perspective shows his commitment to creating content that is not only intellectually stimulating but also of tangible value to the reader.

The Role of Technology and Digital Platforms

Discussing his work with Wired Magazine, Kelly reflects on the impact of technology on writing and culture. He observes how platforms like YouTube have become hotbeds of creativity and innovation, where ideas are continuously exchanged and evolved. This recognition of digital platforms' role underlines the importance of adapting to new media in creative work.

The Concept of a Thousand True Fans

Kelly talks about the importance of cultivating a dedicated fan base, citing his concept of a "Thousand True Fans." This idea has influenced how creators and artists approach their work and audience, emphasizing the value of a loyal community over a vast but passive audience. A popular example of this concept in action is the YouTube personality Mr Beast. His dedicated fan base, though numbering in the millions, exhibits the kind of loyalty and engagement that Kelly's "Thousand True Fans" concept describes. Mr Beast's success is a testament to the power of this approach in the digital age.

Balancing Creativity and Marketing

Kelly acknowledges the essential role of marketing in the creative process. ==He argues that in today's landscape, creating great work is only half the battle; the other half is effectively marketing it.== This insight is crucial for modern writers and creators who must navigate both the creative and commercial aspects of their work.

Conclusion

Kevin Kelly’s insights offer a unique perspective on writing and creativity, blending honesty, originality, and practicality. His reflections provide valuable guidance for anyone looking to deepen their understanding of the creative process and the evolving role of technology and media in shaping our cultural and intellectual landscapes.

Business Success through Dreamers and Doers: A TED Talk Analysis

Sharad Jain — Thu, 21 Dec 2023 10:00:00 GMT

Hello, everyone!

Welcome, fellow explorers! Today, we embark on a journey into the heart of organizations, where we'll unravel the intricate dance between 'dreamers' and 'doers.' Our quest is not merely academic; we seek actionable wisdom to apply in our own realms.

The Dreamer-Doer Dynamic

In the heart of every successful organization, there lies a vibrant diversity. This diversity is not confined to the traditional parameters of demographics, but it extends to the very essence of how individuals perceive and interact with the world around them. It is a diversity of thought, of approach, and of work styles. This rich tapestry of diversity is beautifully epitomized in the existence of two distinct types of individuals. ==On one end of the spectrum, we have the 'zero-to-one' humans, the dreamers.== These are the visionaries, the pioneers who dare to imagine what has not yet been conceived. ==On the other end, we find their 'one-to-n' counterparts, the doers.== These are the pragmatists, the builders who take the dreamers' visions and transform them into tangible realities.

Who are the Dreamers?

Let's take a moment to talk about the dreamers. Who are they, you ask? Well, imagine someone who can look at the world and see not just what is, but what could be. These are the visionaries, the ones who dare to dream of things that don't yet exist. They're the founders, the inventors, the ones who thrive in the face of ambiguity and aren't afraid to seek out-of-the-box solutions. They're the ones who look at a problem and see not a roadblock, but a challenge to be overcome. But it's not always smooth sailing for these dreamers. Their unique approach to reality, while exciting, can sometimes create tension in environments that are more traditionally structured. But don't worry, we'll get to that later.

And the Doers?

Doers, on the other hand, are the builders. They transform visions into tangible successes. They are the glue of the organization, maintaining its stability and ensuring long-term success. While they might not always grasp the dreamers' vision initially, their role in realizing these dreams is indispensable.

Bridging the Gap

The real magic happens when these two groups come together. But how do we bridgethis gap effectively?

Buy-in Through Incentivization

A key strategy is aligning interests through incentives. For instance, in the case of Lola, a feminine health reproductive company, the founders realized the need to expand beyond a single product. To achieve this, they had to integrate doers with retail experience into their dreamer-heavy team. The solution? Tying their compensation to the company's overall performance, ensuring a vested interest in each other's success.

Creating a Corporate Mosh Pit

It's also about creating spaces where dreamers and doers can coexist and collaborate effectively. This 'corporate mosh pit' acknowledges and utilizes the strengths of both. At Marriott, for example, the dreamer team working on future experiences collaborates closely with doers, like franchisees and frontline staff, to prototype and refine new concepts.

Building Speed Bumps, Not Roadblocks

It's crucial to implement processes that guide, not hinder. While dreamers might chafe at too many restrictions, a complete lack of structure can alienate doers. ==The key is to create 'speed bumps' - processes that provide enough structure to maintain alignment without stifling innovation.==

Celebrating All Wins

Finally, it's essential to celebrate the achievements of both dreamers and doers. ==Every success, whether from a dreamer's innovation or a doer's execution, is a win for the organization and should be recognized as such.==

Conclusion

==The synergy between dreamers and doers is not just beneficial but necessary for the growth and innovation of any organization.== ==By understanding and valuing the contributions of both, creating environments for effective collaboration, and aligning incentives, we can harness the full potential of our teams.==

Thank you for taking the time to read this post. I hope it inspires you to look at your team dynamics differently and find new ways to innovate and succeed together!

The Future of Large Language Models: A Deep Dive

Sharad Jain — Wed, 10 May 2023 11:39:36 GMT

Part 1

Think of an app that's like a parrot. But this isn't any ordinary parrot. It doesn't just copy what it hears. Instead, it can come up with a wide range of clear and well-structured responses. And instead of eating bird food, it's been 'fed' with all the different types of information on the internet. This is what Large Language Models (LLMs) like ChatGPT or GPT-4 are about.

These LLMs work using two main parts: a parameters file and a run file. Think of the parameters file as the parrot's memory of phrases it's learned. It holds the 'weights' of the neural network, which are like the strength of connections in the parrot's brain. ==These weights are learned during training on a vast corpus of internet data and are the core of what the model knows - they're like the digital food that our parrot eats, sourced from the entire internet.==

The run file is akin to the parrot's voice box. It takes the 'weights' or learned phrases from the parameters file and uses them to form sentences. This file sets up the structure of the neural network, which is like the parrot's brain. It then performs a forward pass of the network, which is a process of turning inputs (like prompts or questions) into outputs (the parrot's responses). In essence, it's the part of the system that allows our digital parrot to 'speak'.

Training these digital parrots is no small feat. It's akin to teaching a real parrot to understand and mimic human language, but on a much larger scale. This process requires a high-performance computer, known as a GPU cluster, capable of processing and learning from the vast expanse of the internet. It's a time-consuming and costly process, but the result is a model capable of generating articulate and coherent responses to a wide array of prompts.

==The magic of these LLMs lies in their ability to predict the next word in a sequence.== ==This seemingly simple task is actually a powerful learning mechanism.== ==It forces the model to understand the context of the sentence, the subject of the conversation, the tone of the writing, and much more.== It's not just predicting a word; it's understanding language and the world it describes.

In essence, LLMs are like digital parrots that have read the entire internet. ==They don't understand the world in the way humans do, but they can mimic human-like text based on the patterns they've learned.== This makes them powerful tools for a wide range of tasks, from writing articles to answering questions, and much more.

Mastering Marketing Attribution in the age of AI

Sharad Jain — Thu, 29 Dec 2022 11:39:36 GMT

==Crafting a robust marketing strategy across diverse channels and platforms, each with their unique and intricate user journeys, is a complex task.== Yet, assessing the effectiveness of your marketing expenditure and calculating its ROI should be straightforward.

Grasping how your marketing expenditure contributes to conversions and sales is vital. It empowers you to continually adapt your strategy to evolving customer preferences and market conditions. While numerous commercial solutions exist to help you understand your marketing campaigns' performance, they often suffer from two main drawbacks:

==They are siloed.== For instance, Facebook provides an overview of your performance on their platform, but comparing that to the performance of other channels can be challenging.
==They are one-size-fits-all black boxes that remove flexibility and control.== For example, Google Analytics attribution can only attribute one channel to each session.

==This is where GPT (Generative Pretrained Transformer) and other Language Models (LLMs) come into play.== ==They can help early-stage marketers leverage scientific methods to measure marketing effectiveness.==

Modeling Sessions

The Snowplow JavaScript tracker also captures a session ID with each event, the domain_sessionid, as well as a session index. The session ID is used to model sessions in Snowplow’s web data model. This SQL model aggregates out-of-the-box page views and page pings into a set of derived tables: page_views, sessions, and users. These tables have one row per page view ID, session ID (i.e., the domain_sessionid), or user ID (i.e., the domain_userid).

The Snowplow JavaScript tracker also captures a session ID with each event, the domain_sessionid, as well as a session index. The session cookie is set against the same domain as the domain_userid cookie (a first-party cookie set against the domain the tracking is on). By default, it expires after 30 minutes of inactivity, but a different interval can be picked in the tracker initialization (i.e. sessionCookieTimeout: 3600 ).

The session ID is used to model sessions in Snowplow’s web data model. This SQL model aggregates out-of-the-box page views and page pings into a set of derived tables: page_views, sessions and users. These tables have one row per page view ID (as captured in the web page context), session ID (i.e. the domain_sessionid) or user ID (i.e. the domain_userid).

Adding Marketing Costs (Google Ads Example)

If marketing costs are pulled into the data warehouse (using an ETL tool such as Stitch), they can be added to the sessions table based on the marketing parameters. For example, if the Google click and keyword performance reports are available, the average cost per click can be added to sessions that originated from a paid Google search using the marketing click ID:

` CREATE TABLE .ad_kw_click_perf DISTKEY(gclickid) SORTKEY(gclickid) AS ( WITH click_perf AS ( SELECT cpr.googleclickid AS gclickid, cpr.day::date as date, cpr.adgroupid AS adgid, cpr.keywordid AS kwid FROM .click_performance_report AS cpr WHERE cpr.googleclickid IS NOT NULL GROUP BY 1,2,3,4 ) SELECT cp.gclickid, kpr.keywordid as kw_id, kpr.keyword as kw, kpr.adgroup as ad_group, kpr.adgroupid as ad_gid, kpr.adgroupstate as ad_g_state, kpr.campaign as campaign, kpr.campaignid as camp_id, kpr.campaignstate as camp_state, kpr.customerid as cust_id, kpr.clicks as clicks, kpr.impressions as impressions, cast((kpr.cost::float/1000000::float) as numeric(38,6)) as total_cost, cast((kpr.avgcpc::float/1000000::float) as numeric(38,6)) as avg_cpc, kpr.day::date as date FROM .keywords_performance_report AS kpr INNER JOIN click_perf AS cp ON kpr.keywordid = cp.kwid AND kpr.day::date = cp.date AND kpr.adgroupid = cp.adgid GROUP BY 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15 );

Next Steps

Once you have developed an understanding of what channels drive customers to your digital products, you can proceed with defining what activities you want to attribute - whether its newsletter signups, pdf downloads, product purchases, subscriptions, etc. This information can also be added to sessions as additional metrics, or flags. The resulting table can then be used as the basis for your various attribution models.

Marketing Attribution with GPT and Other LLMs

Data is driving more high-stakes decisions across companies and industries, and marketing strategies are no exception. As your channel mix and user journeys grow more complex, it becomes less likely that siloed or one-size-fits-all commercial tools will deliver what you need to attribute and optimize your marketing spend accurately.

Attributing credit to different events in the journey provides evidence of what is and isn't working. ==But without being able to take charge of your data to choose the attribution logic that reflects your customers' journeys (and their touchpoints), you cannot truly understand the real return on your investment.== ==With GPT and other LLMs, you have that flexibility and control.==