hermes-agent/tests/agent/test_credential_pool_unmatched_rotation_bound.py
kshitijk4poor 93a7267b10 test: fold private-predicate and replayed-contract tests into their keepers
Each removed test re-asserts a contract a stronger test already owns at
the real boundary; the keeper is named per group.

- api_server CORS: `_origin_allowed` for an allowlisted origin and for an
  Origin-less client -> the configured-origin preflight test and every
  Origin-less /health request. The Idempotency-Key allow-header assertion
  moves into the preflight test instead of running a second identical
  preflight.
- api_server room grants: the execution-policy predicate's two outcomes ->
  test_api_server_runs' HTTP grant-refresh tests (403 on drift, 200 on the
  authorized digest).
- api_server multimodal: the pure input_image canonicalization -> the
  /v1/responses test that sends the same parts and asserts the same shape.
- credential pool: matched-hint rotation (401) and second-429 rotation ->
  test_credential_pool_routing's real-pool hint test and TestPoolRotationCycle,
  which assert the same steps plus the exact rotate kwargs and swap.
- shared metrics: two runtime cases that replay the send resolver's truth
  table -> TestSendRequiresCollection; the e2e gzip-retry test (both
  attempts compress in the same second, so it cannot see mtime) ->
  TestCompression::test_gzip_is_deterministic_across_time; e2e outbox
  untouched -> TestOutboxIsNotAQueue.
- mcp oauth: provider reuse across reconnects -> the manager restore test;
  the non-tty private predicate -> build_oauth_auth's fail-fast test; the
  manager disk-watch test -> the integration test, which gains its one
  unique assertion (an unchanged file returns False).
- startup watchdog: disarm-without-arm -> test_disarm_is_idempotent, whose
  second disarm runs in the same state and branch.
2026-09-29 15:58:55 +05:30

87 lines
3.4 KiB
Python

"""#70401: the unmatched-identity rotation branch in
``mark_exhausted_and_rotate()`` must be bounded and must not write cooldowns
onto innocent healthy keys.
With OAuth-token auth (provider ``nous``), the upstream 401's ``api_key_hint``
never matches any pool entry's ``runtime_api_key`` — the wrapper's runtime key
rotates. The no-match branch deliberately marks nothing exhausted (marking
would quarantine an innocent healthy key for the full cooldown TTL) and hands
back a fresh selection. But because nothing is ever marked, the pool can never
converge to the "no available entries" state: with the old code the caller
retried the same dead token forever (~6/sec), starving the event loop so chat
``/stop`` interrupts were never processed; only killing the gateway ended it.
The fix keeps the don't-mark-innocent-keys semantics (see the breaker/cooldown
design notes in ``mark_exhausted_and_rotate`` — the pool only trips on
confirmed-empty state, and no cooldown is invented here) but BOUNDS the
branch: after one full lap of the available entries with no recovery, the
rotation returns None so the caller surfaces the error / activates fallback.
Healthy keys carry no cooldown and are immediately available next turn — this
does not reintroduce hammering, it stops it.
"""
import json
import pytest
def _seed_pool(tmp_path, monkeypatch, entries, provider="openrouter"):
hermes_home = tmp_path / "hermes"
hermes_home.mkdir(parents=True, exist_ok=True)
(hermes_home / "auth.json").write_text(
json.dumps({"version": 1, "credential_pool": {provider: entries}})
)
monkeypatch.setenv("HERMES_HOME", str(hermes_home))
from agent.credential_pool import load_pool
return load_pool(provider)
def _entry(idx, key):
return {
"id": f"cred-{idx}",
"label": f"key-{idx}",
"auth_type": "api_key",
"priority": idx,
"source": "manual",
"access_token": key,
}
class TestUnmatchedHintRotationIsBounded:
def test_multi_entry_pool_unmatched_hint_loop_terminates(
self, tmp_path, monkeypatch
):
"""Multi-entry pool: consecutive unmatched-hint rotations must reach
None within one lap of the pool instead of ping-ponging forever."""
pool = _seed_pool(
tmp_path, monkeypatch,
[_entry(0, "key-a"), _entry(1, "key-b"), _entry(2, "key-c")],
)
assert pool.select() is not None
results = []
for _ in range(10): # caller's retry loop
nxt = pool.mark_exhausted_and_rotate(
status_code=401,
error_context={"reason": "unauthorized"},
api_key_hint="oauth-runtime-token-that-matches-nothing",
)
results.append(nxt)
if nxt is None:
break
else:
pytest.fail(
"unbounded 401 retry loop: 10 unmatched-hint rotations never "
"returned None (#70401)"
)
# Bounded within one lap (3 available entries → at most 3 rotations
# before the streak trips).
assert len(results) <= 4
assert results[-1] is None
# The escape must NOT have invented cooldowns for healthy keys.
statuses = {e.id: e.last_status for e in pool._entries}
assert all(status != "exhausted" for status in statuses.values()), (
f"innocent keys were quarantined: {statuses}"
)