hermes-agent/evals
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Siddharth Balyan 5eea87882a
Clarify: one question shape and one result shape on every surface (skip, cancel, timeout and undelivered are told apart) (#127760)
* refactor(desktop): split clarify-tool.tsx into a clarify/ folder

Pure moves, no behaviour change. Parsing, the question-card core
(shell, choice rows, question block), the delivery watchdog, and each
pending/settled card get their own file so the core can be reused.

* feat(clarify): one questions[] shape with per-question status and one outcome

The clarify tool now takes only questions=[{question, choices?, multi_select?}].
A wrong shape is a tool error that names the right one, so a model that
sends the old top-level question/choices corrects itself on the next call.

Every result has the same shape on every surface: each response carries
status (answered, skipped, unanswered) with user_response null unless
answered, and the result carries one outcome (submitted, cancelled,
timed_out, undelivered) plus an optional surface-written notice. The
timeout and cancel sentences that used to pose as the user's answer are
gone, and the compressor reads status instead of matching their prefixes.

The callback contract is callback(questions) -> {answers, outcome, notice?}
(None = skipped, missing qid = unanswered). tui_gateway settles a batch the
same way for the last lock, a cancel, an interrupt and the deadline, and
clarify.lock keeps a null answer as a skip. A multi-select answer that is
not a JSON array counts as one typed ("Other") answer. The tool-row preview
reads the first question.

* feat(clarify): every surface asks through the one question card

CLI: the batch panel is the only panel; Enter on an empty field skips a
question, Ctrl+C cancels and keeps the locked answers, the deadline returns
timed_out. -q and -z return undelivered with a no-user notice.

Ink TUI: the single-question prompt is gone; the card supports multi-select
(Space or a digit toggles, Enter locks, typed Other text joins the picks),
an empty submit skips, Esc cancels, and "Batch" names are dropped.

Desktop: the single-question card is deleted and its keyboard handling moved
into the one card (arrows, letters and digits, auto-advance, Enter picks then
confirms). Confirm enables at one answer; blank questions lock null; Skip and
a composer message cancel; the settled card shows No answer for unanswered
questions. The bots room card follows the same contract.

Messaging: one card per question with a "Reply skip to skip" line in every
locale; timeouts and delivery failures map to timed_out and undelivered.

* fix(clarify): cancelled for a released messaging card, labels win over the skip word

A prose reply to a messaging choice card, /new and session end released
the wait with "", which read as timed_out. They now resolve with a
CANCELLED marker and the tool gets outcome cancelled.

A typed reply that matches a choice label ("Skip") now resolves to that
choice; the skip word applies only when no choice matches.

The bots room card sends the picked labels as they are; the tool already
strips the recommendation label. Unused OUTCOMES and a new docstring and
comment are removed.

* fix(tui): re-editing a multi-select answer keeps its picks

The Ink card stored a multi-select answer as display text ("A, B"), so
revisiting the question put the whole string into Other and re-locked it
as one item. The answers map now keeps the raw JSON array, the card
restores the picks on revisit, and only the display lines format it.

Comments that earlier edits reworded are restored to their original
words, minus the phrases the change made wrong. The compute-host clarify
lock accepts None for a skipped question.

* test(clarify): align three checks with the one-answer confirm and the regenerated keys

The desktop Cmd+Enter test now expects a send with one answer (the blank
question locks null), the compressor test expects the single-answer summary
the code gives, and locales/_keys.desktop.json is regenerated for the
removed and added clarify strings.

* test(tui): the one-question card header is singular
2026-09-29 18:33:44 +05:30
..
botmode-dm-delivery chore(evals,desktop): drop literal /tmp from eval harnesses and desktop scripts 2026-09-19 10:44:26 -07:00
botmode-dm-matrix chore(evals,desktop): drop literal /tmp from eval harnesses and desktop scripts 2026-09-19 10:44:26 -07:00
browser_use Merge remote-tracking branch 'origin/main' into ethie/pm-clean 2026-09-19 22:57:07 -04:00
codebase_navigability feat(pm): isolate developer test environment from runtime extras 2026-09-23 18:13:38 -04:00
codex_echo merge: integrate upstream/main into ethie/pm-clean 2026-09-08 19:17:39 -04:00
compaction evals(compaction): document the Jev repeated-compaction recipe and add the table renderer 2026-09-19 11:58:37 -07:00
core_tool_deferral Clarify: one question shape and one result shape on every surface (skip, cancel, timeout and undelivered are told apart) (#127760) 2026-09-29 18:33:44 +05:30
dashboard_auth fix(dashboard-auth): one refresh single-flight for the cookie gate and the native route, off the event loop 2026-09-13 09:40:44 -07:00
delegation_group_schema chore(evals,desktop): drop literal /tmp from eval harnesses and desktop scripts 2026-09-19 10:44:26 -07:00
desktop-pane-fixture test: keep Desktop pane-focus setup outside cold imports 2026-09-08 07:29:26 -07:00
desktop_bug_campaign fix(pm): isolate bootstrap dependencies and unify YAML on ruamel 2026-09-11 12:23:51 -04:00
desktop_mcp_oauth Merge remote-tracking branch 'origin/main' into ethie/pm-clean 2026-09-19 13:45:07 -04:00
gateway fix(pm): isolate bootstrap dependencies and unify YAML on ruamel 2026-09-11 12:23:51 -04:00
gateway_completion fix: wake idle process watches on their owning profile transport 2026-09-07 14:16:57 -07:00
gateway_failure_ownership fix(gateway): carry accepted-input ownership through persistence 2026-09-07 14:11:18 -07:00
gateway_status_render chore(evals): harness stubs follow the _delivery_adapter_for seam 2026-09-19 01:28:23 -07:00
liveness test(tui-gateway): add isolated websocket orphan reconnect probe 2026-09-07 07:05:21 -07:00
memory chore(evals,desktop): drop literal /tmp from eval harnesses and desktop scripts 2026-09-19 10:44:26 -07:00
native_compaction chore(evals,desktop): drop literal /tmp from eval harnesses and desktop scripts 2026-09-19 10:44:26 -07:00
openrouter_pkce_ab test(auth): OpenRouter PKCE invariants, fake-authority A/B harness, docs, contributor map 2026-09-12 22:07:41 -07:00
postmortem Merge remote-tracking branch 'origin/main' into ethie/pm-clean 2026-09-19 22:57:07 -04:00
prompt_footprint fix(desktop): load property card guidance only on demand 2026-09-08 13:39:01 -07:00
provider_fallback fix(pm): isolate bootstrap dependencies and unify YAML on ruamel 2026-09-11 12:23:51 -04:00
provider_wire fix(pm): isolate bootstrap dependencies and unify YAML on ruamel 2026-09-11 12:23:51 -04:00
providers chore(evals,desktop): drop literal /tmp from eval harnesses and desktop scripts 2026-09-19 10:44:26 -07:00
readtool chore(evals,desktop): drop literal /tmp from eval harnesses and desktop scripts 2026-09-19 10:44:26 -07:00
session_search_schema test(delegate): tidy #121572 follow-ups 2026-09-27 18:17:20 +05:30
subagent_process_handoff chore(evals,desktop): drop literal /tmp from eval harnesses and desktop scripts 2026-09-19 10:44:26 -07:00
token_accounting chore(evals,desktop): drop literal /tmp from eval harnesses and desktop scripts 2026-09-19 10:44:26 -07:00
tool_search Merge remote-tracking branch 'origin/main' into ethie/pm-clean 2026-09-19 22:57:07 -04:00
toolperf_abeval Merge remote-tracking branch 'origin/main' into ethie/pm-clean 2026-09-14 15:38:34 -04:00
update_pipeline fix(ci): prepare icon renderer in prebuilt TUI lane 2026-09-24 02:29:34 -04:00
update_streaming fix(update): stream build progress without concealing silent stalls 2026-09-07 05:56:50 -07:00
webhook_auth fix(webhook): only webhook-signature commits a delivery to the Standard Webhooks path 2026-09-06 05:35:48 -07:00
__init__.py docs+evals: AGENTS.md for the facade/siblings layout; codebase-navigability benchmark harness 2026-09-03 23:55:56 -07:00
acp_empty_session_wire.py test(acp): verify existing empty metadata over native session load 2026-09-07 04:46:04 -07:00
anthropic_proxy_thinking_replay.py test(anthropic): trim DeepSeek proxy replay tests to two invariants + local wire probe 2026-09-06 05:37:03 -07:00
api_delegation_http_probe.py test: verify API Runs continuation consumers over loopback HTTP 2026-09-09 10:55:32 -07:00
api_delegation_sync_probe.py fix: persist API delegation units once without waking the model 2026-09-09 10:55:32 -07:00
approval_deny_dispatch.py fix(approvals): honor GNU env split escapes and argv0 operands 2026-09-07 14:30:41 -07:00
auth_pool_controls.py fix: reject independent Nous account refresh without clearing cooldown 2026-09-07 08:06:48 -07:00
auxiliary_resource_exhausted.py fix(auxiliary): ResourceExhausted spellings reach the payment/quota fallback gate 2026-09-06 05:48:39 -07:00
cli_deferred_notice.py Own passive source update checks in Python 2026-09-12 19:03:26 -04:00
cli_fallback_add_picker_error.py test(fallback): keep the two persisted-config invariants; PTY harness for the picker error path 2026-09-06 05:41:46 -07:00
codex_masked_replay_review.py fix(codex): masked "invalid_prompt: Request blocked." replay rejection reaches the replay-strip recovery 2026-09-06 05:40:02 -07:00
completion_backlog_probe.py test: verify completion barriers and document local delivery proof 2026-09-07 08:27:07 -07:00
cron_error_diagnostics.py test(cron): exercise historical error redaction in live probe 2026-09-07 08:18:56 -07:00
cron_timeout_fork_race.py fix(process): stop late forks escaping deadline tree cleanup 2026-09-07 08:19:32 -07:00
delivery_flood_wire.py refactor: repoint 10 MessageEvent/MessageType importers that landed on main after the event.py split 2026-09-07 22:47:33 +05:30
desktop-reap-clock-pressure.cjs test: make desktop orphan-reap deadline checks deterministic 2026-09-07 17:08:51 -07:00
fanout_resource_bench.py bench: explicit utf-8 encoding on text-mode opens (ruff PLW1514) 2026-09-03 02:31:59 -07:00
gemini_type_array_probe.py fix: preserve native Gemini union constraints 2026-09-07 21:10:50 -07:00
goal_command_parity.py test: retain a reproducible CLI goal parity oracle 2026-09-07 00:51:41 -07:00
heartbeat_idle_wire.py chore(evals,desktop): drop literal /tmp from eval harnesses and desktop scripts 2026-09-19 10:44:26 -07:00
kanban_graph_identity.py fix: retain Kanban decomposition identity and inherit parent tenants 2026-09-07 07:09:59 -07:00
kanban_pr_acceptance_live.py test(kanban): exercise acceptance races through real run lifecycle 2026-09-07 04:46:54 -07:00
kanban_scope_probe.py fix(evals): align Kanban probe with managed MCP overrides 2026-09-15 19:05:29 -07:00
key_cmd_picker_live.py fix: keep command-auth model discovery lazy across config and setup 2026-09-07 21:22:49 -07:00
mcp_device_flow.py test: device login through a later issuer-bound authorization server 2026-09-15 19:00:42 -07:00
output_caps_local_capture.py test: exercise output-cap removal across native and child surfaces 2026-09-07 06:15:43 -07:00
output_caps_scope.md docs: record native output-cap save validation 2026-09-07 06:15:43 -07:00
output_caps_surfaces.py test: exercise output-cap removal across native and child surfaces 2026-09-07 06:15:43 -07:00
process_result_receipt_probe.py fix(tools): keep retained terminal results scoped to their owner 2026-09-07 08:25:33 -07:00
session_snapshot_removal.py fix: remove automatic session JSON snapshots 2026-09-07 08:08:41 -07:00
slack_stream_wire_contract.py test(evals): wire-contract A/B harness for Slack chat.*Stream payloads 2026-09-06 05:34:36 -07:00
update_check_ssh_pty.py Own passive source update checks in Python 2026-09-12 19:03:26 -04:00
update_obligation_identity.py fix(update): settle stale receipt warnings from matching live gateways 2026-09-07 08:20:46 -07:00
update_unit_client_budget.py test: pass snapshot listings to the catch-up restart budget probe 2026-09-07 08:20:09 -07:00
vault_fill_live_e2e.py fix(vault): dogfood fixes — offer save-login on every backend, keep the model off passwords, bind to the login tab 2026-09-10 10:35:07 -07:00