hermes-agent/evals/botmode-dm-delivery/README.md
teknium1 8f7853188f fix(cron): keep deferred delivery exceptions from aborting ticks
Catch unexpected delivery exceptions after claim, retain diagnostics and continue
sibling admissions without authorizing replay. Preserve indefinite retention.

Reproduced PermissionError at target traversal after discovery. Native Electron
controlled-fault A/B confirms the healthy sibling settles and renders once.
2026-09-14 17:29:32 -07:00

107 lines
6.4 KiB
Markdown

# Native Bot Mode delivery probe
Real Electron, production Python backend and tool execution, disposable HOME/HERMES_HOME,
loopback scripted inference (no paid model). Linux seat fixture; run from repository root:
```sh
npm ci --no-audit --no-fund
cp evals/botmode-dm-delivery/probe-dm-delivery.spec.ts apps/desktop/e2e/
git apply evals/botmode-dm-delivery/mock-trigger.patch
(cd apps/desktop && npm run build)
(cd apps/desktop && DISPLAY=:0 XAUTHORITY=/run/user/1000/xauth_cnpsqU \
XDG_RUNTIME_DIR=/run/user/1000 VIRTUAL_ENV="$VIRTUAL_ENV" \
HERMES_DESKTOP_CDP_PORT=off npx playwright test e2e/probe-dm-delivery.spec.ts --reporter=list)
git apply -R evals/botmode-dm-delivery/mock-trigger.patch
rm apps/desktop/e2e/probe-dm-delivery.spec.ts
```
Use the current seat's actual Xauthority path and an existing runtime venv.
Artifacts default to `/tmp/botmode-dm-review/native` (override with `BOT_DM_EVIDENCE`); sandbox path is printed. The
fixture's generated hermes shim pins every child to this checkout, not an installed launcher.
## Verified results
Base `cf35e7351e770`: nested beta quiet CLI executes real `message_agent` to the named
alpha Desktop owner. Admission, execution and reply succeed, and the quiet sender
receives its completion. A reload then shows the attributed incoming message. The
live pre-reload view showed the reply but omitted the incoming row in this fixture;
that renderer-refresh behavior is not fixed by this cron change.
Base CLI-owner case: separate cron producer returns exact `SESSION_NOT_OWNED` refusal.
Release owner, run scheduler tick, open Beta in Desktop: no cron output, assertion red.
Fixed case: queued receipt; release owner; real synchronous scheduler tick drains;
Beta Desktop renders `CLI_OWNER_CRON_SENTINEL` and its reply once. Final two-case
native run: **2 passed (1.2m)**. Supported-owner nested case also passed independently
on base (**1 passed (40.0s)**).
This does not establish native Windows parity, general retry after unowned CLI
failure, or correction of the issue #105460 `--in ~` premise. CLI title resolution
is profile-DB-based, not workspace selection; the named live-owner route is positive.
The queue is deliberately at-most-once after claim. A crash before spawning but
after claiming remains inspectable as claimed; it is not retried automatically.
## Independent review follow-up
The probe now admits the named Beta destination from the default scheduler, then
changes the ticker's `HOME` to a different existing directory before drain.
Published head `c91dfcbfe810c` fails with `Profile 'beta' does not exist`, leaving
zero sentinel inputs in Beta. The follow-up pins the admitted home and ID; Beta
renders one input and reply. Set `BOT_DM_CORRUPT=1` to add one malformed JSON
record alongside the valid admission: before the follow-up the real tick raises
`JSONDecodeError`; afterward the damaged record stays on disk while Beta delivers.
Fresh built native run with both root change and corruption: **2 passed (1.4m)**,
including the existing nested `message_agent` control. Receipts/screenshots:
`/tmp/botmode-dm-review/{native-red2,corrupt-red,final-native}` and matching `.log`
files. The first follow-up run also exposed a fixture mistake (the changed HOME
was not created, so `--in ~` correctly refused); that failed receipt is retained
in `native-green.log`, and both source legs were rerun with an existing HOME.
Prior `/tmp/botmode-dm-recovery*` evidence remains untouched.
## Per-record delivery exception isolation
Set `BOT_DM_EXCEPTION=1` and select `-g "cron output"` for the controlled native
exception probe. `exception-tick.py` raises `PermissionError` at the actual
post-discovery target `Path.is_dir()` boundary, not at the delivery helper.
The same exception was first reproduced with real directory traversal permission
loss on Python 3.11/Linux; the retained test uses a portable controlled fault.
Before the guard, native tick exits 1, leaving the head claimed and sibling queued.
Afterward, the head is ambiguous, the sibling settles and renders once, and a
second real tick replays neither. Logs: `/tmp/botmode-dm-exception-{red,green}.log`;
receipts and screenshot: `/tmp/botmode-dm-exception/{red,green}/`.
The review's repeated-head starvation claim is not reachable: the claim commits
before delivery and later scans skip every non-queued record. One failed tick is
real; recurring replay of that same head is not. Indefinite queued/payload retention
is intentional, with no TTL or automatic ambiguous retry introduced here.
## Ordinary custom-root fallback (#104066 / #104055)
`probe-cron-root.spec.ts` adds the never-deferred sibling: copy it to
`apps/desktop/e2e/` and run with the same native fixture (no mock-trigger patch
needed for this case). It keeps default's real Desktop Bot Chat lease, submits
ordinary cron output to unowned Alpha from a separate Python producer under a
custom Hermes root, and holds the real quiet CLI child at loopback inference.
The child shim PID must match Alpha's real CLI lease; default's lease is unchanged.
After release, Alpha has exactly one input and Desktop renders the output.
The same case removes unused Beta and verifies delivery neither recreates Beta nor
creates a second `.hermes` root under HOME.
Both `origin/main`'s scheduler and pre-follow-up `c827ae179d67c` fail with
`Profile 'alpha' does not exist` before any recipient turn. Fixed native run:
**1 passed (46.3s)**. Two invariant tests exercise the actual CLI startup resolver
across named/default/own destinations with a changed active profile, and refusal
when the destination is missing initially or disappears during discovery:
**5 failed before, 5 passed after**. The old env-clearing test is replaced by these
behavior checks rather than retaining the broken expectation.
Evidence: `/tmp/botmode-cron-root/{before2,origin-main,after}.log`,
`after/{owners.json,children.log,rows.json,result.json,missing.json,ordinary-recipient.png}`.
The first fixture attempt (`before.log`) used the wrong default row label; the
actual Desktop label is Hermes. No production failure is claimed for that attempt.
Full cron directory: **1346 passed, 1 skipped across 116 files**; sibling mailbox,
DM, gateway consumer and profile tests: **124 passed, 3 skipped across 4 files**.
Credit @fangliquanflq's #104066 for the root-boundary diagnosis and anchoring fix;
this combined branch reuses its already-resolved destination instead of repeating
name resolution. No retry or receipt semantics change.