hermes-agent/evals/postmortem/live_ab/goal_judge_wait.py
Teknium a8ca904922 feat(evals): post-mortem harness — forensics lanes + live A/B + review probes for the #102117 run fixes
evals/postmortem/ turns the one-off audit behind tracking issue #103563 into
something anyone with a Hermes state.db copy (and optionally rotated
agent.log*) can run on their own fan-out:

  forensics/   common.py discovers the run tree (root = most descendants,
               compression-rollover children excluded so cost buckets stay
               disjoint), fits pricing from estimated_cost_usd, and five lanes
               recompute the OBSERVED figures: tokens (buckets, depth/duration
               shares, context reconstruction, excess-cache-write proxy, cap
               replay), logcalls (per-call cache behaviour from agent.log with
               coverage printed first; strict and loose plateau definitions
               reported separately), delegation (timeouts, orphaned children,
               polling hours, batch-join withheld child-hours, truncated
               summaries), tools (hardline blocks, foreground refusals,
               whole-file rewrites), goal_loop (nudges, parked barrier), rework
               (public-surface drop at PR open + post-open commit inventory).
               Every figure is labeled OBSERVED or MODELED.
  live_ab/     the per-PR A/Bs (real code paths, fake providers, temp
               HERMES_HOME), paths from argv.
  review_probes/ the independent /review's probes, credited and adapted; each
               reproduced a round-1 defect and the fixed head must pass it.
  run.py       runs the offline probes against one or two checkouts and prints
               PASS/FAIL side by side (--live adds the ones that spend cents).
  tests/       synthetic-DB smoke test for the lanes and runner.

On the run's DB the lanes reproduce the tracking issue's population exactly
(1,394 sessions, 93,284 calls, $19,302.59; cache_write $11,159.76) and on
main vs an integration checkout of the 13 PRs the runner shows every probe
FAIL -> PASS (two guard-only probes pass on both, noted in run.py).

The trajectories are deliberately not shipped: the DB holds 51,956 home
paths, 5,341 e-mails, private IPs, chat ids and real-shaped credentials in
tool output. The lane reports and recomputed JSON are in a secret gist
linked from #103563.
2026-09-05 09:13:10 -07:00

19 lines
1.4 KiB
Python

"""Live judge A/B on the run's real "waiting" response shape. Usage: python judge_ab.py <repo_root> [n]"""
import os, sys
root = sys.argv[1]; n = int(sys.argv[2]) if len(sys.argv) > 2 else 3
sys.path.insert(0, root)
os.environ.setdefault("HERMES_HOME", os.path.expanduser("~/.hermes")) # LIVE: real auxiliary judge calls (cents)
from hermes_cli import goals
goal = ("Simplify the hermes-agent codebase by >=30% LOC with zero behavior change, decomposing every god file, "
"through parallel subagent waves; integrate, run the full test suite, and open one PR.")
response = ("Round 3 status: 4 worker batches are still running (deleg_9f1a2b, deleg_77c0de, deleg_a1b2c3, deleg_0e9f88; "
"8 workers on adapters/gateway god files, median 38 min in). Round-2 integration merged clean; full suite "
"green on the integration branch. Nothing else is dispatchable until these return: their summaries decide "
"which files round 4 takes. Waiting on the batch-complete notifications; no action needed from me now.")
kw = {"active_delegations": 4} if "active_delegations" in goals.judge_goal.__code__.co_varnames else {}
verdicts = []
for _ in range(n):
v, reason, pf, directive, tf = goals.judge_goal(goal, response, **kw)
verdicts.append((v, directive.get("seconds") if directive else None))
print(v, directive, "|", reason[:90])
print("SUMMARY", verdicts)