mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-11-17 04:11:35 +00:00
Against summary + one session_search round-trip (78.9% @ 55K) Jev's default arm (75.5% @ 115K) loses on recall and retains 2.1x the tokens; the earlier closed-book comparison scored our summary with its recovery pointer unused.
160 lines
11 KiB
Markdown
160 lines
11 KiB
Markdown
# fast-jev-compaction vs Hermes compaction — 3-transcript scorecard (2026-09-19)
|
||
|
||
## Verdict
|
||
|
||
**Do not adopt Jev, and do not adopt its retention rule either.** Against what we actually ship
|
||
(summary + one session_search round-trip) Jev loses: 75.5% @ 115K vs 78.9% @ 55K. Its
|
||
closed-book recall gain over the bare summary is real but
|
||
it is bought with 2.1× the retained context re-billed on every turn and with compactions that
|
||
arrive ever more often (each one a prompt-cache break); our summary frees ~90% per event and
|
||
then stays cache-warm for a long stretch. Programmatic tool-result removal as the primary
|
||
compaction multiplies cache breaks; it is the wrong trade for Hermes.
|
||
|
||
- The +32 pt recall win is entirely "keep user/assistant text verbatim, delete old tool
|
||
output". Jev itself dropped 100% of 851 candidates at its default threshold and, at a
|
||
matched token budget, ranked no better than plain recency (77.8 vs 77.8).
|
||
- Jev-only compaction is a one-way ratchet: the text floor grows every cycle and is never
|
||
reduced, so freed-per-compaction decays (89% → 55%, 76% → 20%, 63% → 8% over 32–40
|
||
cycles) and a text-heavy session on a 200K-window host hit a hard stop after 0.4M tokens
|
||
of work (floor ≥ threshold, 0% freed). A summary path is still required; Jev could only be
|
||
a pre-pass.
|
||
- The 32K request window forces the whole history into 25K tokens: in every measured cycle
|
||
the ceiling was binding and Jev decided on tool name + 60-char input + result size with
|
||
message texts replaced by `[… N chars omitted …]`. >~500 tool calls between compactions
|
||
cannot fit at all (1 of 4 transcripts never got a first compaction).
|
||
- What it does buy: 1.4 s and < 1¢ per compaction vs 37 s and 6¢, and verbatim retention
|
||
of everything not a tool result.
|
||
|
||
What the data does point at: the facts our summary loses (delegation ids, root causes,
|
||
config keys, exact error strings) sat in assistant text. That is a summariser-retention
|
||
target (anchor index / identifier capture), not a reason to change compaction cadence.
|
||
|
||
Question asked: does https://github.com/tamaratran/fast-jev-compaction ("replace the
|
||
compaction summary with Jev decisions: score every tool call/result, drop or truncate
|
||
stale ones, keep everything else verbatim") beat our compressor on remaining tokens,
|
||
compaction cost and recall accuracy?
|
||
|
||
Harness: `evals/compaction/runner.py` with the new `engine: jev` arm
|
||
(`evals/compaction/jev_arm.py`, a Python port of the plugin over OpenRouter's Decisions
|
||
API, `~typesafe/jev-latest` → served as `typesafe/jev-1.13-20260917`). Three real 500K-token
|
||
lineage prefixes from state.db (PR review campaign, system-prompt token analysis, SIGSEGV
|
||
fix), 15-question recall exam each, same bank for every arm, answered and judged by the
|
||
configured `auxiliary.compression` route (gemini-3.8-flash via Nous). A fourth transcript
|
||
(541 tool calls in 500K) could not be fitted into Jev's 25K-token state ceiling even at the
|
||
last fitting stage — the plugin throws there and Claude Code falls back to its built-in
|
||
summary; recorded as `jev_fallback`, not scored.
|
||
|
||
## Results (recall % @ retained tokens; compaction cost and wall time per event)
|
||
|
||
| policy | prreview | sysprompt | sigsegv | AVG | compaction $ | compaction s |
|
||
|---|---|---|---|---|---|---|
|
||
| current (main), closed-book | 36.7 @ 40K | 50.0 @ 64K | 43.3 @ 61K | 43.3 @ 55K | $0.061 | 36.9 |
|
||
| **current + session_search recovery** (what we ship) | 76.7 @ 40K | 90.0 @ 63K | 70.0 @ 62K | **78.9 @ 55K** | $0.061 | 36.9 |
|
||
| lean | 33.3 @ 40K | 53.3 @ 65K | 36.7 @ 61K | 41.1 @ 55K | $0.062 | 33.4 |
|
||
| jev (plugin defaults) | 70.0 @ 50K | 63.3 @ 112K | 93.3 @ 181K | **75.5 @ 115K** | $0.007 | 1.4 |
|
||
| jev_tail40 (40 pinned rows) | 73.3 @ 71K | 63.3 @ 179K | 93.3 @ 213K | 76.6 @ 154K | $0.006 | 1.4 |
|
||
| jev_t15 (threshold 0.15) | 93.3 @ 372K | 76.7 @ 332K | 100.0 @ 484K | 90.0 @ 396K | $0.007 | 1.4 |
|
||
| jev_top60k (Jev-ranked, 60K tool budget) | 70.0 @ 113K | 70.0 @ 173K | 93.3 @ 243K | 77.8 @ 176K | $0.007 | 1.5 |
|
||
| recent_top60k (recency-ranked, same budget) | 76.7 @ 114K | 63.3 @ 174K | 93.3 @ 245K | 77.8 @ 177K | $0 | 0.0 |
|
||
|
||
`current` and `lean` are the same code path on today's main (lean tail is the default), so
|
||
their 2–7 pt spread on identical context is the exam noise floor (15 questions ≈ ±3.3 pts).
|
||
Per-question paired comparison, jev vs current across 45 questions: 17 wins, 1 loss, 27 ties.
|
||
|
||
## Findings
|
||
|
||
0. **Against the shipping mechanism (`current+recovery`, 78.9% @ 55K) Jev's default arm
|
||
loses on recall AND retains 2.1× the tokens.** The closed-book `current` row below is the
|
||
summary with its recovery pointer unused; findings 1–3 compare against that weaker arm.
|
||
|
||
1. **Jev's default arm is +32 pts closed-book recall (75.5 vs 43.3) at 2.1× the retained tokens
|
||
(115K vs 55K), for 1/9 the compaction cost ($0.007 vs $0.061) in 1/25 the time
|
||
(1.4 s vs 37 s).** On the one transcript where the sizes are comparable (prreview,
|
||
50K vs 40K) it still wins 70.0 vs 36.7.
|
||
|
||
2. **The recall gain is verbatim text, not Jev's judgment.** At the plugin's 0.5 threshold
|
||
Jev's `keep_result` never exceeded 0.20 (median 0.15) and `keep_call` topped out at 0.50,
|
||
so it dropped 100% of the 851 unpinned candidates across all three transcripts
|
||
(kept=0, result-truncated=0). The default `jev` arm is therefore behaviourally identical
|
||
to "delete every old tool call + result, keep every user/assistant row verbatim". The
|
||
facts the summary loses (delegation ids, root causes, config keys, exact error strings)
|
||
sat in assistant text the whole time.
|
||
|
||
3. **At a matched budget Jev's ranking ties plain recency: 77.8 vs 77.8.** Keeping 60K
|
||
tokens of tool pairs ranked by `keep_result` (jev_top60k) vs ranked by position
|
||
(recent_top60k) gives the same average; per transcript it is +6.7 / −6.7 / 0, inside the
|
||
noise floor. Lowering the threshold to 0.15 (jev_t15) reaches 90% but retains 396K of
|
||
500K — that is not compaction.
|
||
|
||
4. **The state ceiling does not fit Hermes scale.** Jev's 32K window forces the whole
|
||
history into 25K tokens; at 500K every transcript needed the harshest fitting stages
|
||
("old calls compacted/merged", "old messages collapsed") and one of four could not fit at
|
||
all. The plugin is designed for Claude Code's ~200K compaction point; a 1M-window Hermes
|
||
session compacting at 500K+ will fall back to the summary regularly, and once the tool
|
||
results are gone a second compaction has nothing left to remove.
|
||
|
||
5. **Cost shape.** Jev: 5–7 requests per compaction, ~125–200K input tokens total at
|
||
$0.042/M ≈ $0.005–0.008, ~1.5 s wall. Our summary: one gemini-3.8-flash call over
|
||
52–60K input tokens ≈ $0.06, 25–49 s. Both are noise against the per-turn cost of the
|
||
retained context that follows (115K vs 55K tokens on every subsequent turn).
|
||
|
||
## What this suggests for the compressor
|
||
|
||
Nothing structural. Keeping text verbatim wins the closed-book exam but at 2.1× retained
|
||
tokens per turn and a compaction cadence that tightens every cycle (prompt cache broken far
|
||
more often); the summary's one-time 90% reduction is the better trade for a long-lived
|
||
cached conversation. The actionable residue is summariser quality: the misses were exact
|
||
identifiers in assistant text, which the anchor index is meant to capture — check its
|
||
coverage on these three banks before touching anything else.
|
||
|
||
## Repeated compaction: how many cycles does Jev-only compaction survive?
|
||
|
||
`scripts/jev_cycles.py` feeds a lineage chronologically and compacts with Jev (plugin
|
||
defaults) every time the estimate crosses the threshold; 500K ≈ our 1M-window posture, 160K
|
||
≈ a 200K-window host. Raw JSON per run in `results/jev-cycles-2026-09-19/`. Jev spend for all
|
||
six runs: $0.60.
|
||
|
||
| run | cycles | raw session consumed | freed per cycle | text floor | end state |
|
||
|---|---|---|---|---|---|
|
||
| prreview @500K | 32 | 11.4M (whole lineage) | 89% → 55% | 43K → 139K | still working |
|
||
| sysprompt @500K | 40 | 8.6M of 23.3M | 76% → 20% | 91K → 240K | degrading |
|
||
| sigsegv @500K | 40 | 5.2M of 8.6M | 63% → 8% | 175K → 359K | 40K freed/cycle; wall ≈ 8M |
|
||
| prreview @160K | 60 | 3.9M of 11.4M | 87% → 22% | 12K → 87K | compacting every ~14 rows |
|
||
| sigsegv @160K | 11 | 0.42M of 8.6M | 52% → 0% | 85K → 151K | **STUCK** (floor ≥ threshold) |
|
||
| afff57 @500K | 0 | — | — | — | fallback on cycle 1 (541 calls) |
|
||
|
||
Reading: every cycle has candidates (new tool calls arrive between compactions and Jev
|
||
drops ~all of them; 200+ cycles, zero orphaned call/result pairs), so "nothing left to
|
||
remove" never happens. What runs out is headroom: each cycle frees at most
|
||
`threshold − floor`, and the floor (all user/assistant rows) only grows. Well before the
|
||
hard stop the session runs permanently near the window: sigsegv @500K at cycle 40 compacts
|
||
every ~40K tokens of new work with every turn billed at ~460K input.
|
||
|
||
## The 32K window in practice
|
||
|
||
- `state_tok` was 24,8xx–25,000 in every one of ~180 measured cycles: the ceiling always
|
||
binds. Fitting stages reached at 500K: "old messages collapsed" (texts → `[… N chars
|
||
omitted …]`), "old calls compacted" (one line per call, input cut to 60 chars), "old
|
||
messages left out" (text-only old rows removed from the state), "old calls merged". Jev
|
||
never sees tool result contents (by design) and, at these stages, barely sees message
|
||
text either — it decides on tool name, input stub, result size and the last 3 user prompts.
|
||
- Hard limit: ~500+ unpinned tool calls between compactions cannot fit (afff57, 541 calls);
|
||
the plugin throws and the host falls back to its summary. The text floor does not affect
|
||
fit (old text rows are left out of the state); the call count does.
|
||
- Requests: full state resent per batch of ~40–80 calls → 5–8 concurrent requests per
|
||
cycle at 500K, 1–2 s, $0.005–0.008.
|
||
- Untouchable content: anything in assistant text (pasted logs, long analyses) is never
|
||
compacted, which is exactly what makes the floor grow.
|
||
|
||
## Method notes
|
||
|
||
- Transcripts reconstructed with `scripts/reconstruct_lineage.py` from a state.db copy;
|
||
not committed. Question banks generated from the region current compaction summarises
|
||
(the most conservative boundary) and cached per transcript+cap so every arm answers the
|
||
identical exam.
|
||
- The `jev` arm counts rows (Hermes has one `role: tool` row per result), so
|
||
`preserve_recent_messages: 6` pins fewer turns than in Claude Code; `jev_tail40` widens
|
||
it to roughly lean's 25K tail and changes nothing (+1 pt).
|
||
- Eval spend for the whole run (question generation, 634 answer/judge calls): 43.5M input
|
||
tokens ≈ $33.6 at gemini-3.8-flash list, through Nous inference. Jev spend across all
|
||
arms: $0.08 via OpenRouter.
|