Against summary + one session_search round-trip (78.9% @ 55K) Jev's default arm (75.5% @ 115K) loses on recall and retains 2.1x the tokens; the earlier closed-book comparison scored our summary with its recovery pointer unused.
11 KiB
fast-jev-compaction vs Hermes compaction — 3-transcript scorecard (2026-09-19)
Verdict
Do not adopt Jev, and do not adopt its retention rule either. Against what we actually ship (summary + one session_search round-trip) Jev loses: 75.5% @ 115K vs 78.9% @ 55K. Its closed-book recall gain over the bare summary is real but it is bought with 2.1× the retained context re-billed on every turn and with compactions that arrive ever more often (each one a prompt-cache break); our summary frees ~90% per event and then stays cache-warm for a long stretch. Programmatic tool-result removal as the primary compaction multiplies cache breaks; it is the wrong trade for Hermes.
- The +32 pt recall win is entirely "keep user/assistant text verbatim, delete old tool output". Jev itself dropped 100% of 851 candidates at its default threshold and, at a matched token budget, ranked no better than plain recency (77.8 vs 77.8).
- Jev-only compaction is a one-way ratchet: the text floor grows every cycle and is never reduced, so freed-per-compaction decays (89% → 55%, 76% → 20%, 63% → 8% over 32–40 cycles) and a text-heavy session on a 200K-window host hit a hard stop after 0.4M tokens of work (floor ≥ threshold, 0% freed). A summary path is still required; Jev could only be a pre-pass.
- The 32K request window forces the whole history into 25K tokens: in every measured cycle
the ceiling was binding and Jev decided on tool name + 60-char input + result size with
message texts replaced by
[… N chars omitted …]. >~500 tool calls between compactions cannot fit at all (1 of 4 transcripts never got a first compaction). - What it does buy: 1.4 s and < 1¢ per compaction vs 37 s and 6¢, and verbatim retention of everything not a tool result.
What the data does point at: the facts our summary loses (delegation ids, root causes, config keys, exact error strings) sat in assistant text. That is a summariser-retention target (anchor index / identifier capture), not a reason to change compaction cadence.
Question asked: does https://github.com/tamaratran/fast-jev-compaction ("replace the compaction summary with Jev decisions: score every tool call/result, drop or truncate stale ones, keep everything else verbatim") beat our compressor on remaining tokens, compaction cost and recall accuracy?
Harness: evals/compaction/runner.py with the new engine: jev arm
(evals/compaction/jev_arm.py, a Python port of the plugin over OpenRouter's Decisions
API, ~typesafe/jev-latest → served as typesafe/jev-1.13-20260917). Three real 500K-token
lineage prefixes from state.db (PR review campaign, system-prompt token analysis, SIGSEGV
fix), 15-question recall exam each, same bank for every arm, answered and judged by the
configured auxiliary.compression route (gemini-3.8-flash via Nous). A fourth transcript
(541 tool calls in 500K) could not be fitted into Jev's 25K-token state ceiling even at the
last fitting stage — the plugin throws there and Claude Code falls back to its built-in
summary; recorded as jev_fallback, not scored.
Results (recall % @ retained tokens; compaction cost and wall time per event)
| policy | prreview | sysprompt | sigsegv | AVG | compaction $ | compaction s |
|---|---|---|---|---|---|---|
| current (main), closed-book | 36.7 @ 40K | 50.0 @ 64K | 43.3 @ 61K | 43.3 @ 55K | $0.061 | 36.9 |
| current + session_search recovery (what we ship) | 76.7 @ 40K | 90.0 @ 63K | 70.0 @ 62K | 78.9 @ 55K | $0.061 | 36.9 |
| lean | 33.3 @ 40K | 53.3 @ 65K | 36.7 @ 61K | 41.1 @ 55K | $0.062 | 33.4 |
| jev (plugin defaults) | 70.0 @ 50K | 63.3 @ 112K | 93.3 @ 181K | 75.5 @ 115K | $0.007 | 1.4 |
| jev_tail40 (40 pinned rows) | 73.3 @ 71K | 63.3 @ 179K | 93.3 @ 213K | 76.6 @ 154K | $0.006 | 1.4 |
| jev_t15 (threshold 0.15) | 93.3 @ 372K | 76.7 @ 332K | 100.0 @ 484K | 90.0 @ 396K | $0.007 | 1.4 |
| jev_top60k (Jev-ranked, 60K tool budget) | 70.0 @ 113K | 70.0 @ 173K | 93.3 @ 243K | 77.8 @ 176K | $0.007 | 1.5 |
| recent_top60k (recency-ranked, same budget) | 76.7 @ 114K | 63.3 @ 174K | 93.3 @ 245K | 77.8 @ 177K | $0 | 0.0 |
current and lean are the same code path on today's main (lean tail is the default), so
their 2–7 pt spread on identical context is the exam noise floor (15 questions ≈ ±3.3 pts).
Per-question paired comparison, jev vs current across 45 questions: 17 wins, 1 loss, 27 ties.
Findings
-
Against the shipping mechanism (
current+recovery, 78.9% @ 55K) Jev's default arm loses on recall AND retains 2.1× the tokens. The closed-bookcurrentrow below is the summary with its recovery pointer unused; findings 1–3 compare against that weaker arm. -
Jev's default arm is +32 pts closed-book recall (75.5 vs 43.3) at 2.1× the retained tokens (115K vs 55K), for 1/9 the compaction cost ($0.007 vs $0.061) in 1/25 the time (1.4 s vs 37 s). On the one transcript where the sizes are comparable (prreview, 50K vs 40K) it still wins 70.0 vs 36.7.
-
The recall gain is verbatim text, not Jev's judgment. At the plugin's 0.5 threshold Jev's
keep_resultnever exceeded 0.20 (median 0.15) andkeep_calltopped out at 0.50, so it dropped 100% of the 851 unpinned candidates across all three transcripts (kept=0, result-truncated=0). The defaultjevarm is therefore behaviourally identical to "delete every old tool call + result, keep every user/assistant row verbatim". The facts the summary loses (delegation ids, root causes, config keys, exact error strings) sat in assistant text the whole time. -
At a matched budget Jev's ranking ties plain recency: 77.8 vs 77.8. Keeping 60K tokens of tool pairs ranked by
keep_result(jev_top60k) vs ranked by position (recent_top60k) gives the same average; per transcript it is +6.7 / −6.7 / 0, inside the noise floor. Lowering the threshold to 0.15 (jev_t15) reaches 90% but retains 396K of 500K — that is not compaction. -
The state ceiling does not fit Hermes scale. Jev's 32K window forces the whole history into 25K tokens; at 500K every transcript needed the harshest fitting stages ("old calls compacted/merged", "old messages collapsed") and one of four could not fit at all. The plugin is designed for Claude Code's ~200K compaction point; a 1M-window Hermes session compacting at 500K+ will fall back to the summary regularly, and once the tool results are gone a second compaction has nothing left to remove.
-
Cost shape. Jev: 5–7 requests per compaction, ~125–200K input tokens total at $0.042/M ≈ $0.005–0.008, ~1.5 s wall. Our summary: one gemini-3.8-flash call over 52–60K input tokens ≈ $0.06, 25–49 s. Both are noise against the per-turn cost of the retained context that follows (115K vs 55K tokens on every subsequent turn).
What this suggests for the compressor
Nothing structural. Keeping text verbatim wins the closed-book exam but at 2.1× retained tokens per turn and a compaction cadence that tightens every cycle (prompt cache broken far more often); the summary's one-time 90% reduction is the better trade for a long-lived cached conversation. The actionable residue is summariser quality: the misses were exact identifiers in assistant text, which the anchor index is meant to capture — check its coverage on these three banks before touching anything else.
Repeated compaction: how many cycles does Jev-only compaction survive?
scripts/jev_cycles.py feeds a lineage chronologically and compacts with Jev (plugin
defaults) every time the estimate crosses the threshold; 500K ≈ our 1M-window posture, 160K
≈ a 200K-window host. Raw JSON per run in results/jev-cycles-2026-09-19/. Jev spend for all
six runs: $0.60.
| run | cycles | raw session consumed | freed per cycle | text floor | end state |
|---|---|---|---|---|---|
| prreview @500K | 32 | 11.4M (whole lineage) | 89% → 55% | 43K → 139K | still working |
| sysprompt @500K | 40 | 8.6M of 23.3M | 76% → 20% | 91K → 240K | degrading |
| sigsegv @500K | 40 | 5.2M of 8.6M | 63% → 8% | 175K → 359K | 40K freed/cycle; wall ≈ 8M |
| prreview @160K | 60 | 3.9M of 11.4M | 87% → 22% | 12K → 87K | compacting every ~14 rows |
| sigsegv @160K | 11 | 0.42M of 8.6M | 52% → 0% | 85K → 151K | STUCK (floor ≥ threshold) |
| afff57 @500K | 0 | — | — | — | fallback on cycle 1 (541 calls) |
Reading: every cycle has candidates (new tool calls arrive between compactions and Jev
drops ~all of them; 200+ cycles, zero orphaned call/result pairs), so "nothing left to
remove" never happens. What runs out is headroom: each cycle frees at most
threshold − floor, and the floor (all user/assistant rows) only grows. Well before the
hard stop the session runs permanently near the window: sigsegv @500K at cycle 40 compacts
every ~40K tokens of new work with every turn billed at ~460K input.
The 32K window in practice
state_tokwas 24,8xx–25,000 in every one of ~180 measured cycles: the ceiling always binds. Fitting stages reached at 500K: "old messages collapsed" (texts →[… N chars omitted …]), "old calls compacted" (one line per call, input cut to 60 chars), "old messages left out" (text-only old rows removed from the state), "old calls merged". Jev never sees tool result contents (by design) and, at these stages, barely sees message text either — it decides on tool name, input stub, result size and the last 3 user prompts.- Hard limit: ~500+ unpinned tool calls between compactions cannot fit (afff57, 541 calls); the plugin throws and the host falls back to its summary. The text floor does not affect fit (old text rows are left out of the state); the call count does.
- Requests: full state resent per batch of ~40–80 calls → 5–8 concurrent requests per cycle at 500K, 1–2 s, $0.005–0.008.
- Untouchable content: anything in assistant text (pasted logs, long analyses) is never compacted, which is exactly what makes the floor grow.
Method notes
- Transcripts reconstructed with
scripts/reconstruct_lineage.pyfrom a state.db copy; not committed. Question banks generated from the region current compaction summarises (the most conservative boundary) and cached per transcript+cap so every arm answers the identical exam. - The
jevarm counts rows (Hermes has onerole: toolrow per result), sopreserve_recent_messages: 6pins fewer turns than in Claude Code;jev_tail40widens it to roughly lean's 25K tail and changes nothing (+1 pt). - Eval spend for the whole run (question generation, 634 answer/judge calls): 43.5M input tokens ≈ $33.6 at gemini-3.8-flash list, through Nous inference. Jev spend across all arms: $0.08 via OpenRouter.