hermes-agent/evals/compaction/results/SCORECARD-2026-09-19-jev.md
teknium1 dba031b5cb evals(compaction): add the shipping current+recovery arm to the Jev scorecard
Against summary + one session_search round-trip (78.9% @ 55K) Jev's default
arm (75.5% @ 115K) loses on recall and retains 2.1x the tokens; the earlier
closed-book comparison scored our summary with its recovery pointer unused.
2026-09-19 11:58:37 -07:00

160 lines
11 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# fast-jev-compaction vs Hermes compaction — 3-transcript scorecard (2026-09-19)
## Verdict
**Do not adopt Jev, and do not adopt its retention rule either.** Against what we actually ship
(summary + one session_search round-trip) Jev loses: 75.5% @ 115K vs 78.9% @ 55K. Its
closed-book recall gain over the bare summary is real but
it is bought with 2.1× the retained context re-billed on every turn and with compactions that
arrive ever more often (each one a prompt-cache break); our summary frees ~90% per event and
then stays cache-warm for a long stretch. Programmatic tool-result removal as the primary
compaction multiplies cache breaks; it is the wrong trade for Hermes.
- The +32 pt recall win is entirely "keep user/assistant text verbatim, delete old tool
output". Jev itself dropped 100% of 851 candidates at its default threshold and, at a
matched token budget, ranked no better than plain recency (77.8 vs 77.8).
- Jev-only compaction is a one-way ratchet: the text floor grows every cycle and is never
reduced, so freed-per-compaction decays (89% → 55%, 76% → 20%, 63% → 8% over 32–40
cycles) and a text-heavy session on a 200K-window host hit a hard stop after 0.4M tokens
of work (floor ≥ threshold, 0% freed). A summary path is still required; Jev could only be
a pre-pass.
- The 32K request window forces the whole history into 25K tokens: in every measured cycle
the ceiling was binding and Jev decided on tool name + 60-char input + result size with
message texts replaced by `[… N chars omitted …]`. >~500 tool calls between compactions
cannot fit at all (1 of 4 transcripts never got a first compaction).
- What it does buy: 1.4 s and < 1¢ per compaction vs 37 s and 6¢, and verbatim retention
of everything not a tool result.
What the data does point at: the facts our summary loses (delegation ids, root causes,
config keys, exact error strings) sat in assistant text. That is a summariser-retention
target (anchor index / identifier capture), not a reason to change compaction cadence.
Question asked: does https://github.com/tamaratran/fast-jev-compaction ("replace the
compaction summary with Jev decisions: score every tool call/result, drop or truncate
stale ones, keep everything else verbatim") beat our compressor on remaining tokens,
compaction cost and recall accuracy?
Harness: `evals/compaction/runner.py` with the new `engine: jev` arm
(`evals/compaction/jev_arm.py`, a Python port of the plugin over OpenRouter's Decisions
API, `~typesafe/jev-latest` → served as `typesafe/jev-1.13-20260917`). Three real 500K-token
lineage prefixes from state.db (PR review campaign, system-prompt token analysis, SIGSEGV
fix), 15-question recall exam each, same bank for every arm, answered and judged by the
configured `auxiliary.compression` route (gemini-3.8-flash via Nous). A fourth transcript
(541 tool calls in 500K) could not be fitted into Jev's 25K-token state ceiling even at the
last fitting stage — the plugin throws there and Claude Code falls back to its built-in
summary; recorded as `jev_fallback`, not scored.
## Results (recall % @ retained tokens; compaction cost and wall time per event)
| policy | prreview | sysprompt | sigsegv | AVG | compaction $ | compaction s |
|---|---|---|---|---|---|---|
| current (main), closed-book | 36.7 @ 40K | 50.0 @ 64K | 43.3 @ 61K | 43.3 @ 55K | $0.061 | 36.9 |
| **current + session_search recovery** (what we ship) | 76.7 @ 40K | 90.0 @ 63K | 70.0 @ 62K | **78.9 @ 55K** | $0.061 | 36.9 |
| lean | 33.3 @ 40K | 53.3 @ 65K | 36.7 @ 61K | 41.1 @ 55K | $0.062 | 33.4 |
| jev (plugin defaults) | 70.0 @ 50K | 63.3 @ 112K | 93.3 @ 181K | **75.5 @ 115K** | $0.007 | 1.4 |
| jev_tail40 (40 pinned rows) | 73.3 @ 71K | 63.3 @ 179K | 93.3 @ 213K | 76.6 @ 154K | $0.006 | 1.4 |
| jev_t15 (threshold 0.15) | 93.3 @ 372K | 76.7 @ 332K | 100.0 @ 484K | 90.0 @ 396K | $0.007 | 1.4 |
| jev_top60k (Jev-ranked, 60K tool budget) | 70.0 @ 113K | 70.0 @ 173K | 93.3 @ 243K | 77.8 @ 176K | $0.007 | 1.5 |
| recent_top60k (recency-ranked, same budget) | 76.7 @ 114K | 63.3 @ 174K | 93.3 @ 245K | 77.8 @ 177K | $0 | 0.0 |
`current` and `lean` are the same code path on today's main (lean tail is the default), so
their 2–7 pt spread on identical context is the exam noise floor (15 questions ≈ ±3.3 pts).
Per-question paired comparison, jev vs current across 45 questions: 17 wins, 1 loss, 27 ties.
## Findings
0. **Against the shipping mechanism (`current+recovery`, 78.9% @ 55K) Jev's default arm
loses on recall AND retains 2.1× the tokens.** The closed-book `current` row below is the
summary with its recovery pointer unused; findings 1–3 compare against that weaker arm.
1. **Jev's default arm is +32 pts closed-book recall (75.5 vs 43.3) at 2.1× the retained tokens
(115K vs 55K), for 1/9 the compaction cost ($0.007 vs $0.061) in 1/25 the time
(1.4 s vs 37 s).** On the one transcript where the sizes are comparable (prreview,
50K vs 40K) it still wins 70.0 vs 36.7.
2. **The recall gain is verbatim text, not Jev's judgment.** At the plugin's 0.5 threshold
Jev's `keep_result` never exceeded 0.20 (median 0.15) and `keep_call` topped out at 0.50,
so it dropped 100% of the 851 unpinned candidates across all three transcripts
(kept=0, result-truncated=0). The default `jev` arm is therefore behaviourally identical
to "delete every old tool call + result, keep every user/assistant row verbatim". The
facts the summary loses (delegation ids, root causes, config keys, exact error strings)
sat in assistant text the whole time.
3. **At a matched budget Jev's ranking ties plain recency: 77.8 vs 77.8.** Keeping 60K
tokens of tool pairs ranked by `keep_result` (jev_top60k) vs ranked by position
(recent_top60k) gives the same average; per transcript it is +6.7 / −6.7 / 0, inside the
noise floor. Lowering the threshold to 0.15 (jev_t15) reaches 90% but retains 396K of
500K — that is not compaction.
4. **The state ceiling does not fit Hermes scale.** Jev's 32K window forces the whole
history into 25K tokens; at 500K every transcript needed the harshest fitting stages
("old calls compacted/merged", "old messages collapsed") and one of four could not fit at
all. The plugin is designed for Claude Code's ~200K compaction point; a 1M-window Hermes
session compacting at 500K+ will fall back to the summary regularly, and once the tool
results are gone a second compaction has nothing left to remove.
5. **Cost shape.** Jev: 5–7 requests per compaction, ~125–200K input tokens total at
$0.042/M ≈ $0.005–0.008, ~1.5 s wall. Our summary: one gemini-3.8-flash call over
52–60K input tokens ≈ $0.06, 25–49 s. Both are noise against the per-turn cost of the
retained context that follows (115K vs 55K tokens on every subsequent turn).
## What this suggests for the compressor
Nothing structural. Keeping text verbatim wins the closed-book exam but at 2.1× retained
tokens per turn and a compaction cadence that tightens every cycle (prompt cache broken far
more often); the summary's one-time 90% reduction is the better trade for a long-lived
cached conversation. The actionable residue is summariser quality: the misses were exact
identifiers in assistant text, which the anchor index is meant to capture — check its
coverage on these three banks before touching anything else.
## Repeated compaction: how many cycles does Jev-only compaction survive?
`scripts/jev_cycles.py` feeds a lineage chronologically and compacts with Jev (plugin
defaults) every time the estimate crosses the threshold; 500K ≈ our 1M-window posture, 160K
≈ a 200K-window host. Raw JSON per run in `results/jev-cycles-2026-09-19/`. Jev spend for all
six runs: $0.60.
| run | cycles | raw session consumed | freed per cycle | text floor | end state |
|---|---|---|---|---|---|
| prreview @500K | 32 | 11.4M (whole lineage) | 89% → 55% | 43K → 139K | still working |
| sysprompt @500K | 40 | 8.6M of 23.3M | 76% → 20% | 91K → 240K | degrading |
| sigsegv @500K | 40 | 5.2M of 8.6M | 63% → 8% | 175K → 359K | 40K freed/cycle; wall ≈ 8M |
| prreview @160K | 60 | 3.9M of 11.4M | 87% → 22% | 12K → 87K | compacting every ~14 rows |
| sigsegv @160K | 11 | 0.42M of 8.6M | 52% → 0% | 85K → 151K | **STUCK** (floor ≥ threshold) |
| afff57 @500K | 0 | — | — | — | fallback on cycle 1 (541 calls) |
Reading: every cycle has candidates (new tool calls arrive between compactions and Jev
drops ~all of them; 200+ cycles, zero orphaned call/result pairs), so "nothing left to
remove" never happens. What runs out is headroom: each cycle frees at most
`threshold − floor`, and the floor (all user/assistant rows) only grows. Well before the
hard stop the session runs permanently near the window: sigsegv @500K at cycle 40 compacts
every ~40K tokens of new work with every turn billed at ~460K input.
## The 32K window in practice
- `state_tok` was 24,8xx–25,000 in every one of ~180 measured cycles: the ceiling always
binds. Fitting stages reached at 500K: "old messages collapsed" (texts → `[… N chars
omitted …]`), "old calls compacted" (one line per call, input cut to 60 chars), "old
messages left out" (text-only old rows removed from the state), "old calls merged". Jev
never sees tool result contents (by design) and, at these stages, barely sees message
text either — it decides on tool name, input stub, result size and the last 3 user prompts.
- Hard limit: ~500+ unpinned tool calls between compactions cannot fit (afff57, 541 calls);
the plugin throws and the host falls back to its summary. The text floor does not affect
fit (old text rows are left out of the state); the call count does.
- Requests: full state resent per batch of ~40–80 calls → 5–8 concurrent requests per
cycle at 500K, 1–2 s, $0.005–0.008.
- Untouchable content: anything in assistant text (pasted logs, long analyses) is never
compacted, which is exactly what makes the floor grow.
## Method notes
- Transcripts reconstructed with `scripts/reconstruct_lineage.py` from a state.db copy;
not committed. Question banks generated from the region current compaction summarises
(the most conservative boundary) and cached per transcript+cap so every arm answers the
identical exam.
- The `jev` arm counts rows (Hermes has one `role: tool` row per result), so
`preserve_recent_messages: 6` pins fewer turns than in Claude Code; `jev_tail40` widens
it to roughly lean's 25K tail and changes nothing (+1 pt).
- Eval spend for the whole run (question generation, 634 answer/judge calls): 43.5M input
tokens ≈ $33.6 at gemini-3.8-flash list, through Nous inference. Jev spend across all
arms: $0.08 via OpenRouter.