hermes-agent/evals/compaction/results/SCORECARD-2026-09-19-jev.md
teknium1 dba031b5cb evals(compaction): add the shipping current+recovery arm to the Jev scorecard
Against summary + one session_search round-trip (78.9% @ 55K) Jev's default
arm (75.5% @ 115K) loses on recall and retains 2.1x the tokens; the earlier
closed-book comparison scored our summary with its recovery pointer unused.
2026-09-19 11:58:37 -07:00

11 KiB
Raw Permalink Blame History

fast-jev-compaction vs Hermes compaction — 3-transcript scorecard (2026-09-19)

Verdict

Do not adopt Jev, and do not adopt its retention rule either. Against what we actually ship (summary + one session_search round-trip) Jev loses: 75.5% @ 115K vs 78.9% @ 55K. Its closed-book recall gain over the bare summary is real but it is bought with 2.1× the retained context re-billed on every turn and with compactions that arrive ever more often (each one a prompt-cache break); our summary frees ~90% per event and then stays cache-warm for a long stretch. Programmatic tool-result removal as the primary compaction multiplies cache breaks; it is the wrong trade for Hermes.

  • The +32 pt recall win is entirely "keep user/assistant text verbatim, delete old tool output". Jev itself dropped 100% of 851 candidates at its default threshold and, at a matched token budget, ranked no better than plain recency (77.8 vs 77.8).
  • Jev-only compaction is a one-way ratchet: the text floor grows every cycle and is never reduced, so freed-per-compaction decays (89% → 55%, 76% → 20%, 63% → 8% over 32–40 cycles) and a text-heavy session on a 200K-window host hit a hard stop after 0.4M tokens of work (floor ≥ threshold, 0% freed). A summary path is still required; Jev could only be a pre-pass.
  • The 32K request window forces the whole history into 25K tokens: in every measured cycle the ceiling was binding and Jev decided on tool name + 60-char input + result size with message texts replaced by [… N chars omitted …]. >~500 tool calls between compactions cannot fit at all (1 of 4 transcripts never got a first compaction).
  • What it does buy: 1.4 s and < 1¢ per compaction vs 37 s and 6¢, and verbatim retention of everything not a tool result.

What the data does point at: the facts our summary loses (delegation ids, root causes, config keys, exact error strings) sat in assistant text. That is a summariser-retention target (anchor index / identifier capture), not a reason to change compaction cadence.

Question asked: does https://github.com/tamaratran/fast-jev-compaction ("replace the compaction summary with Jev decisions: score every tool call/result, drop or truncate stale ones, keep everything else verbatim") beat our compressor on remaining tokens, compaction cost and recall accuracy?

Harness: evals/compaction/runner.py with the new engine: jev arm (evals/compaction/jev_arm.py, a Python port of the plugin over OpenRouter's Decisions API, ~typesafe/jev-latest → served as typesafe/jev-1.13-20260917). Three real 500K-token lineage prefixes from state.db (PR review campaign, system-prompt token analysis, SIGSEGV fix), 15-question recall exam each, same bank for every arm, answered and judged by the configured auxiliary.compression route (gemini-3.8-flash via Nous). A fourth transcript (541 tool calls in 500K) could not be fitted into Jev's 25K-token state ceiling even at the last fitting stage — the plugin throws there and Claude Code falls back to its built-in summary; recorded as jev_fallback, not scored.

Results (recall % @ retained tokens; compaction cost and wall time per event)

policy prreview sysprompt sigsegv AVG compaction $ compaction s
current (main), closed-book 36.7 @ 40K 50.0 @ 64K 43.3 @ 61K 43.3 @ 55K $0.061 36.9
current + session_search recovery (what we ship) 76.7 @ 40K 90.0 @ 63K 70.0 @ 62K 78.9 @ 55K $0.061 36.9
lean 33.3 @ 40K 53.3 @ 65K 36.7 @ 61K 41.1 @ 55K $0.062 33.4
jev (plugin defaults) 70.0 @ 50K 63.3 @ 112K 93.3 @ 181K 75.5 @ 115K $0.007 1.4
jev_tail40 (40 pinned rows) 73.3 @ 71K 63.3 @ 179K 93.3 @ 213K 76.6 @ 154K $0.006 1.4
jev_t15 (threshold 0.15) 93.3 @ 372K 76.7 @ 332K 100.0 @ 484K 90.0 @ 396K $0.007 1.4
jev_top60k (Jev-ranked, 60K tool budget) 70.0 @ 113K 70.0 @ 173K 93.3 @ 243K 77.8 @ 176K $0.007 1.5
recent_top60k (recency-ranked, same budget) 76.7 @ 114K 63.3 @ 174K 93.3 @ 245K 77.8 @ 177K $0 0.0

current and lean are the same code path on today's main (lean tail is the default), so their 2–7 pt spread on identical context is the exam noise floor (15 questions ≈ ±3.3 pts). Per-question paired comparison, jev vs current across 45 questions: 17 wins, 1 loss, 27 ties.

Findings

  1. Against the shipping mechanism (current+recovery, 78.9% @ 55K) Jev's default arm loses on recall AND retains 2.1× the tokens. The closed-book current row below is the summary with its recovery pointer unused; findings 1–3 compare against that weaker arm.

  2. Jev's default arm is +32 pts closed-book recall (75.5 vs 43.3) at 2.1× the retained tokens (115K vs 55K), for 1/9 the compaction cost ($0.007 vs $0.061) in 1/25 the time (1.4 s vs 37 s). On the one transcript where the sizes are comparable (prreview, 50K vs 40K) it still wins 70.0 vs 36.7.

  3. The recall gain is verbatim text, not Jev's judgment. At the plugin's 0.5 threshold Jev's keep_result never exceeded 0.20 (median 0.15) and keep_call topped out at 0.50, so it dropped 100% of the 851 unpinned candidates across all three transcripts (kept=0, result-truncated=0). The default jev arm is therefore behaviourally identical to "delete every old tool call + result, keep every user/assistant row verbatim". The facts the summary loses (delegation ids, root causes, config keys, exact error strings) sat in assistant text the whole time.

  4. At a matched budget Jev's ranking ties plain recency: 77.8 vs 77.8. Keeping 60K tokens of tool pairs ranked by keep_result (jev_top60k) vs ranked by position (recent_top60k) gives the same average; per transcript it is +6.7 / −6.7 / 0, inside the noise floor. Lowering the threshold to 0.15 (jev_t15) reaches 90% but retains 396K of 500K — that is not compaction.

  5. The state ceiling does not fit Hermes scale. Jev's 32K window forces the whole history into 25K tokens; at 500K every transcript needed the harshest fitting stages ("old calls compacted/merged", "old messages collapsed") and one of four could not fit at all. The plugin is designed for Claude Code's ~200K compaction point; a 1M-window Hermes session compacting at 500K+ will fall back to the summary regularly, and once the tool results are gone a second compaction has nothing left to remove.

  6. Cost shape. Jev: 5–7 requests per compaction, ~125–200K input tokens total at $0.042/M ≈ $0.005–0.008, ~1.5 s wall. Our summary: one gemini-3.8-flash call over 52–60K input tokens ≈ $0.06, 25–49 s. Both are noise against the per-turn cost of the retained context that follows (115K vs 55K tokens on every subsequent turn).

What this suggests for the compressor

Nothing structural. Keeping text verbatim wins the closed-book exam but at 2.1× retained tokens per turn and a compaction cadence that tightens every cycle (prompt cache broken far more often); the summary's one-time 90% reduction is the better trade for a long-lived cached conversation. The actionable residue is summariser quality: the misses were exact identifiers in assistant text, which the anchor index is meant to capture — check its coverage on these three banks before touching anything else.

Repeated compaction: how many cycles does Jev-only compaction survive?

scripts/jev_cycles.py feeds a lineage chronologically and compacts with Jev (plugin defaults) every time the estimate crosses the threshold; 500K ≈ our 1M-window posture, 160K ≈ a 200K-window host. Raw JSON per run in results/jev-cycles-2026-09-19/. Jev spend for all six runs: $0.60.

run cycles raw session consumed freed per cycle text floor end state
prreview @500K 32 11.4M (whole lineage) 89% → 55% 43K → 139K still working
sysprompt @500K 40 8.6M of 23.3M 76% → 20% 91K → 240K degrading
sigsegv @500K 40 5.2M of 8.6M 63% → 8% 175K → 359K 40K freed/cycle; wall ≈ 8M
prreview @160K 60 3.9M of 11.4M 87% → 22% 12K → 87K compacting every ~14 rows
sigsegv @160K 11 0.42M of 8.6M 52% → 0% 85K → 151K STUCK (floor ≥ threshold)
afff57 @500K 0 — — — fallback on cycle 1 (541 calls)

Reading: every cycle has candidates (new tool calls arrive between compactions and Jev drops ~all of them; 200+ cycles, zero orphaned call/result pairs), so "nothing left to remove" never happens. What runs out is headroom: each cycle frees at most threshold − floor, and the floor (all user/assistant rows) only grows. Well before the hard stop the session runs permanently near the window: sigsegv @500K at cycle 40 compacts every ~40K tokens of new work with every turn billed at ~460K input.

The 32K window in practice

  • state_tok was 24,8xx–25,000 in every one of ~180 measured cycles: the ceiling always binds. Fitting stages reached at 500K: "old messages collapsed" (texts → [… N chars omitted …]), "old calls compacted" (one line per call, input cut to 60 chars), "old messages left out" (text-only old rows removed from the state), "old calls merged". Jev never sees tool result contents (by design) and, at these stages, barely sees message text either — it decides on tool name, input stub, result size and the last 3 user prompts.
  • Hard limit: ~500+ unpinned tool calls between compactions cannot fit (afff57, 541 calls); the plugin throws and the host falls back to its summary. The text floor does not affect fit (old text rows are left out of the state); the call count does.
  • Requests: full state resent per batch of ~40–80 calls → 5–8 concurrent requests per cycle at 500K, 1–2 s, $0.005–0.008.
  • Untouchable content: anything in assistant text (pasted logs, long analyses) is never compacted, which is exactly what makes the floor grow.

Method notes

  • Transcripts reconstructed with scripts/reconstruct_lineage.py from a state.db copy; not committed. Question banks generated from the region current compaction summarises (the most conservative boundary) and cached per transcript+cap so every arm answers the identical exam.
  • The jev arm counts rows (Hermes has one role: tool row per result), so preserve_recent_messages: 6 pins fewer turns than in Claude Code; jev_tail40 widens it to roughly lean's 25K tail and changes nothing (+1 pt).
  • Eval spend for the whole run (question generation, 634 answer/judge calls): 43.5M input tokens ≈ $33.6 at gemini-3.8-flash list, through Nous inference. Jev spend across all arms: $0.08 via OpenRouter.