hermes-agent/gateway/AGENTS.md
tancou c7c9c18ccf fix(profiles): pin the launch home so a mirrored HERMES_HOME cannot flip routed-profile decisions
Symptom: a host that serves several profiles from one process and mirrors
the active turn's profile into `os.environ["HERMES_HOME"]` for legacy
readers (Hermes WebUI does this on every chat turn, next to the
context-local override) makes every launch-home decision see the served
profile as the launch profile. Two profiles that both configure `atlassian`
with different credentials share whichever MCP connection came first: a
READ_ONLY_MODE=false profile ends up calling a read-only server
(nesquena/hermes-webui#7721). The same misjudgement leaves the launch
residue in the served profile's child env, seeds the launch profile's
bridged allow-all grant into the served profile's secret scope, and lets
the served profile's `terminal.*` config bridge into the shared process env.

Cause: four launch-home checks compare the task's override with
`get_process_hermes_home()`, which reads `HERMES_HOME` live:
`agent.secret_scope.serves_routed_profile` (keys the MCP ledger via
`_mcp_registry_scope`, #108352 / #111481, and the check_fn cache, #111151),
`agent.secret_scope._is_process_home`, `tools.environments.local._is_routed_home`
and `hermes_cli.env_loader._process_hermes_home`. Under the mirror the two
sides are equal for every turn.

Change: `hermes_constants.pin_process_hermes_home(path | None)` lets the
host record the home it serves as its own; `get_routing_process_hermes_home()`
returns the pin when set, else `get_process_hermes_home()`; the four checks
compare against it. The pin is deliberately NOT folded into
`get_process_hermes_home()`: `get_hermes_home()` falls back to it for tasks
carrying no override (MCP loop, spawners), and the host's mirror exists
precisely so those readers see the served profile. Only "is this task
routed / is this the launch home" changes. Unpinned, behaviour is
byte-for-byte the old one; hosts that never mutate `HERMES_HOME` need not
call it. `activate_multi_profile_hosting()` is not the seam for this: it
flips `get_secret` fail-closed process-wide and freezes the launch env,
which an embedding host cannot adopt as a bug fix.

Tests (2 invariants, parametrized over the four checks plus the MCP ledger
key; red on main, green here): pinned + mirrored env -> the served home is
routed and the launch home is not, the MCP key is `(home_key, name)`,
`get_process_hermes_home()` still follows the env var; never pinned or
pinned-then-cleared -> old semantics, including "a mirrored env var IS the
launch home". `tests/conftest.py` resets the pin per test so the
module-global cannot leak between files.

Live repro (WebUI + a stdio FastMCP server named `atlassian` in two
profiles, one gated by READ_ONLY_MODE): base -> one ledger key
`'atlassian'`, the write profile lists only the read-only tools; fixed ->
`(<read_home_key>, 'atlassian')` and `(<write_home_key>, 'atlassian')`,
each profile lists its own tools.

Docs: `gateway/AGENTS.md` § Profile scope (one launch-home identity) and the
isolation table in `website/docs/user-guide/multi-profile-gateways.md`.
Also maps the author e-mail under contributors/emails/ (attribution check).

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-09-23 08:09:47 -07:00

255 lines
20 KiB
Markdown

# gateway/ — messaging gateway, adapters, delivery
Applies on top of the root `AGENTS.md`. Long-form: `website/docs/developer-guide/gateway-internals.md`.
New platform adapter: follow `gateway/platforms/ADDING_A_PLATFORM.md` step by step.
## Shape
`gateway/run.py` is the facade; phases live in `run_*.py` (startup, adapters, inbound, turn, busy,
goals, notifications, shutdown, ...), sessions in `session*.py`, slash handlers in
`slash_commands_*.py` mixins, authorization in `authz_mixin.py`, adapters in `platforms/<name>.py`
over `platforms/base.py`. `builtin_hooks/` is the extension point for always-registered gateway
hooks (none shipped). The gateway reads user YAML **raw** (`run.py` + `config.py`), not through
`DEFAULT_CONFIG` — a key the CLI sees but the gateway doesn't means you're on the wrong loader
(`hermes_cli/AGENTS.md`). Each adapter picks a base toolset (Telegram → `"messaging"`).
Slash commands: handlers are looked up by name through `_command_handler_table`; a command is
listed in `_IDLE_COMMANDS` or `_PLAIN_COMMANDS` (works mid-run) in `run_busy.py`. No
`if canonical == ...` chains. Registry + adding a command: `hermes_cli/AGENTS.md`.
## The gateway has TWO message guards — both must bypass approval/control commands
While an agent is running, an inbound message passes two sequential guards: (1) the **base
adapter** (`platforms/base.py`) queues it in `_pending_messages` when `session_key in
self._active_sessions`; (2) the **runner** (`run.py`/`run_busy.py`) intercepts `/stop`, `/new`,
`/queue`, `/status`, `/approve`, `/deny` before they reach `running_agent.interrupt()`. Any new
command that must reach the runner while the agent is blocked (approval prompts) MUST bypass BOTH
guards and be dispatched inline — never via `_process_message_background()`, which races session
lifecycle.
## Streaming delivery contract (stream-is-the-message adapters)
Adapters with `draft_stream_is_message = True` (relay Slack native streaming) keep ONE cumulative
native stream per turn; the stream IS the final message. Four invariants, each from a live
duplicate-final incident (NS-658 canary ledger, hermes#85796 / gateway-gateway#210); violating any
re-creates a duplicate or frozen stream:
1. **Draft frames are prefix-stable.** Frame N must be a string prefix of frame N+1. Never mutate
drafts per tick — no fence-closing (`ensure_closed_code_fences`), cursor suffix, segment-state
resets at tool boundaries, or mrkdwn conversion. A non-prefix frame forces a whole-snapshot
re-append ("stacked copies"). The finalize path may still transform the real final.
2. **The consumer declares the final; the adapter never guesses.** `finish(final_text)` carries the
completed `final_response` (verifier footer, completion explainer included). New post-stream
augmentation MUST ride this payload — mutating `final_response` after the seal re-opens the
`delivered_final_matches` mismatch → corrective duplicate send.
3. **Interim sends carry `metadata["_interim_send"] = True`.** Any consumer-side `adapter.send()`
that is not the turn-final (commentary, segment-tail flushes) must set it or seal-interception
seals the live stream with interim text. Seal-interception exists at BOTH egress doors (`send()`
and `send_for_platform()`); a new egress door needs the same two checks.
4. **Reconcile by edit, never by plain send.** A lane delivering a final beside a sealed stream
(queued follow-ups, media-accompanied finals) first tries `edit_message` on the consumer's
`message_id`; plain `send()` only when no editable message exists. A sealed native stream is a
regular message — `chat.update` works (live-verified).
Contract tests: `tests/gateway/test_stream_final_contract.py` (mutation-checked). Slack ground
truth: `chat.*Stream` speaks STANDARD markdown, not mrkdwn; `stopStream.markdown_text` APPENDS;
`startStream`/`stopStream` are Tier 2 (~20/min). Check `draft_stream_is_message is True` —
MagicMock adapters in older tests auto-create truthy attributes.
## Background process notifications
`terminal(background=true, notify_on_complete=true)` starts a gateway watcher that detects
completion and triggers a new agent turn. Verbosity: `display.background_process_notifications`
(or `HERMES_BACKGROUND_NOTIFICATIONS`): `concise` (default; one line, failures append an output
tail), `all` (running updates + final raw output), `result` (final raw output only), `error`
(final raw output only on non-zero exit), `off`.
The watcher is armed on the gateway loop at registration time (`terminal_tool_background.py::
_register_completion_watcher` → `run_notifications.py::arm_process_watcher`); `pending_watchers`
is only the fallback for processes registered before the gateway serves (checkpoint recovery) or
while it stops, drained at startup and post-turn. Agent-notify watchers send no user-facing
receipt (the agent's next turn is the report) unless the launching turn is still running at exit
— then the injection only queues a follow-up, so the concise receipt goes out immediately.
The idle completion watcher also drains `watch_match` / `watch_disabled`; no user follow-up is
required. Notify-off drains these without waking. Transport failures are retried; unavailable
durable completion owners/transports do not spend delivery attempts. Profile-namespaced process
events keep their owning adapter, including raw progress/final notices. Adapter acceptance is
at-least-once admission, not proof that a model turn or final outbound reply completed. API-server
async completions remain durable delivery rows, never autonomous new model turns. Require the
`admit_internal_event` receipt for completion/watch injection: a handler returning None is not
acceptance. Refused admission refunds every claimed batch sibling without spending an attempt;
actual delivery errors keep their bounded retry policy. Recognized raw API routes resolve after
persisted messaging origins and defer quietly when unavailable; malformed routes still warn.
Cron execution has its own session. Eligible continuable deliveries may mirror or seed the
reply-facing conversation: origin, origin-less home fallback, user-written bare-platform home,
or opted-in explicit targets. `all` expansions do not gain home mirror eligibility. Mirrored
briefs are labelled user turns appended at a turn boundary, preserving role alternation
(`cron/AGENTS.md`).
## `/login` (off-turn, paired DM only)
`/login` is registered in `hermes_cli/commands.py` with `busy_policy="dispatch"` and
`desktop="settings"`, listed in `run_busy.py::_PLAIN_COMMANDS`, and handled by
`GatewayLoginCommandsMixin` (`gateway/slash_commands_login.py`). It refuses outside a paired DM:
`chat_type in {"dm","private"}`, a truthy `chat_id`, and a platform whose `"dm"` really is a paired
conversation — ntfy, raft and a2a all report `chat_type="dm"` for a broadcast topic, a channel and
an agent peer, so posting a consent link there would publish it.
**It binds the whole install.** The sign-in writes the singleton `providers.nous`, so whoever
approves the code owns this gateway's inference and connectors for every chat it serves. Slash
gating is opt-in (`gateway/slash_access.py`): with no `allow_admin_from` set, every user allowed to
DM the bot can run it. Operators of shared gateways must set it.
The handler returns its ack at once and drains `anon_auth.run_sign_in` on a **private single-worker
executor** (`_login_executor`), never the shared 10-thread gateway pool — a promotion wait can last
the code's full expiry, and a live worker in the shared pool makes shutdown skip the SessionDB
close/checkpoint. One attempt per process, stamped with the identity that started it: the same
identity's second `/login` supersedes (the loser ends with the superseded copy pushed into its own
chat); a different identity is refused. `task.cancel()` cannot interrupt a blocking poll inside a
worker thread, so shutdown and supersede both work through `attempt.cancelled`, which
`wait_for_promotion` now polls on a ≤1 s tick. A shutdown-cancelled attempt pushes nothing — the
task is dying and its adapters may already be gone — but if the server had already completed the
transfer it still persists, because that transfer is irreversible.
On completion the handler **evicts** every cached agent still on `nous/welcome` and clears any
session model override pinned to it. It does not switch agents in place and does not write a model
override: `settle_after_upgrade` already moved `model.default`/`model.base_url` in the config, every
turn re-resolves the config and the credentials, and `_agent_config_signature` already forces a
rebuild when the route changes — so an in-place swap buys no cache warmth and an override would pin
an expiring access token that nothing refreshes.
## Gateway lifecycle vs. the Desktop app
`hermes serve` (control plane, desktop-spawned child) dies with the app — by design. The messaging
gateway (`gateway run`) SURVIVES the app: the serve backend's `/api/gateway/*` endpoints spawn it
detached (`_spawn_hermes_action` — `start_new_session` / `DETACHED_PROCESS`), so `before-quit`'s
SIGTERM never reaches it and bots keep running. The known breach is the Windows shim-unlock
teardown (`taskkill /T /F` on venv-shim holders, #85265), which exists to let updates proceed and is
replaced by #92091's `pause-for-update`. Do NOT "fix" gateway-dies-with-app by re-parenting the
gateway under the backend, and do NOT "fix" update locks by widening the tree-kill. Gateways stamp
`code_sha`/`code_version` into `gateway_state.json` (`status.py`) so the updater can verify a fleet.
## Profile scope (adapters, turns, and everything between turns)
- **One identity per inbound event, canonicalized FIRST.** `gateway/session_identity.py::resolve_identity`
answers "which bot received it / who may admit it / where does it run" ONCE per event and pins a
frozen `RoutingIdentity` on the source (wire-invisible, like `_transport_adapter_ref`). Every
ingress path calls the canonicalize seam before it derives a key: the adapter side
(`platforms/base.py::_canonicalize` — `handle_message`, `_enqueue_text_event`, Telegram photo /
album routing, `_handle_message_while_active`, every `_source_session_key`) and the runner side
(`run_adapters.py::_canonicalize` — the per-profile / default message, busy and platform-event
handlers, the adapter auth-check callback, `run_inbound.py::_hm_admit_event`). No key derivation
before it; an unresolved identity under multiplexing (route to an unserved profile) is dropped
with one WARNING at the first seam it reaches, never keyed into `agent:main`. `_transport_owner`,
`_authorization_home_for_source`, `_resolve_profile_home_for_source`, `_session_key_profile` and
`_resolve_profile_for_key` read the identity when present and fall back to their old chain only
for sources nothing resolved (restored rows, hand-built sources). Never derive a second answer
next to the identity; extend the object. `transport_profile` ≠ `runtime_profile` is normal
(shared bot → routed satellite). A source copy goes through `session_identity.replace_source`
so the identity travels with it (`_apply_topic_recovery` does).
- **Intake vs delivery adapter.** `authz_mixin.py::_intake_adapter_for(source)` is the bot that
RECEIVED the event (live transport ref → relay socket → identity's `transport_profile`); it gates
intake policy (ignored channels, relay fronting, re-dispatch of a still-live event) and fails
closed to `None` for a source with no live provenance. `_delivery_adapter_for(source)` is the bot
that ANSWERS — the receiving bot whenever known, else the unique owner of `(platform,
runtime_profile)` via `_adapters_for_profile`. Never read `self.adapters[platform]` for a source;
never add a third resolver. The matrix (`tests/gateway/test_multiplex_transport_matrix.py`):
| Topology | runtime | intake | delivery |
|---|---|---|---|
| per-credential bot, no route | owner | owner adapter | owner adapter |
| shared credential → satellite (`profile_routes`) | routed | receiving adapter | receiving adapter (`_is_shared_bot_satellite` after restart) |
| shared bot → profile that owns its own bot | routed | receiving adapter | receiving adapter (conversation continuity; #70625's "routed bot" reading was not adopted) |
| secondary-owned bot → `default` (`bot_profile`) | default (`agent:main`) | receiving adapter | receiving adapter |
| restored / hand-built, no live provenance | stored `source.profile` | **`None`** | unique owner of `(platform, runtime)`; a disconnected secondary → `None`, never the default bot |
- **Identity survives the process.** `SessionEntry.transport_profile` (routing index +
`sessions.transport_profile`, nullable, reconciled by `SCHEMA_SQL`) persists the receiving bot
next to the key namespace; the namespace says where a lane RUNS, the column says which bot may
DELIVER to it. Anything reviving a session from durable state (auto-resume, heartbeat restore,
plugin injection, background-process events) reads `entry.origin` through
`authz_mixin.py::_restored_source`, which re-pins a `RoutingIdentity(transport=None)` via
`session_identity.restore_identity`; `_delivery_adapter_for` then delivers through that bot's
adapter or nothing (never the default bot by heuristic). Rows without the column (pre-PR-5)
keep the `_is_shared_bot_satellite` fallback. Deferred callbacks capture the identity/home at
command time (`/model` picker); `_run_in_executor_with_context` carries the scope over thread
hops. Relay: `_with_scope` echoes the routed `profile` on every outbound frame and `follow_up`
derives it from the key namespace so the connector stamps it on the next `passthrough_forward`.
Ambient `get_active_profile_name()` reads in `gateway/` are boot-only and marked
`# launch profile, pre-identity`; a path with an identity reads `identity.runtime_profile`.
- **Token locks.** An adapter that connects with a unique credential (bot token, API key) calls
`acquire_scoped_lock()` from `gateway.status` in `connect()`/`start()` and `release_scoped_lock()`
in `disconnect()`/`stop()`, so two profiles cannot share one credential. Canonical:
`plugins/platforms/irc/adapter.py`.
- **Multiplex profile-scoped env reads MUST fail closed — never borrow from `os.environ`**
(`agent/secret_scope.py`; #72348, #86905). Under `gateway.multiplex_profiles`, `os.environ` holds
the DEFAULT profile's values; a secondary profile's `.env` exists only in its secret scope,
bound per profile *activity* by `_profile_runtime_scope` (turn, callback, eviction, tick — never
only "per turn"). All profile-level env config — credentials
(`app_secret`, tokens) AND authorization (`FEISHU_ALLOWED_USERS`, `{PLATFORM}_ALLOW_ALL_USERS`,
`GATEWAY_ALLOW_ALL_USERS`, `group_policy`, `allow_bots`) — is read scope-aware: adapters via
`gateway.platforms._shared.get_scoped_secret` (the ONE implementation; adapters import it as
`_get_scoped_secret`; `extra_or_secret` / `seed_extra_from_env` / `env_is_connected` build on it),
gateway authz via `_shared.platform_gate_env` (imported by `authz_mixin.py` as `_auth_env`). Scope installed +
multiplex active → a scoped miss returns the **default**, NEVER `os.environ` (a leaked allowlist
skips the allow-all check and silently rejects every secondary-profile sender, #86905). The
unscoped default-profile path (`UnscopedSecretError`) and single-profile deployments keep the
`os.environ` read — there it IS the profile's own value. Never re-implement the reader in an
adapter (the `try get_secret / except UnscopedSecretError: os.getenv` shape drifts into a
fallback-after-miss leak); import the shared one. `tests/gateway/test_shared_platform_boilerplate.py`
asserts every plugin's `_env_enablement` reads only through it.
- **Scope is bound per profile activity, not per turn.** `run.py::_profile_runtime_scope(home)`
wraps a routed turn (`run_turn.py::_profile_scope_for_source`); the same binding wraps every path
that touches a session's home, secrets or terminal scope with no turn on the stack: release and
eviction (`run_agent_cache.py::_run_release_in_profile_scope` — TTL, LRU and memory-pressure
eviction all route through it so `on_session_end`/memory flush hit the OWNING profile's provider),
shutdown (`run_shutdown.py::_finalize_session`), post-turn media delivery
(`platforms/base.py::_media_delivery_scope`), deferred callbacks (pickers, reactions — capture the
routed home at command time, re-enter the scope in the callback), notifiers and outbound webhooks.
Resolve the owning home from the session record (`profile_home`, `agent:<profile>:` key), never
from `os.environ`, which holds the launch profile. Why: eviction that flushed under the launch scope
wrote a secondary profile's memories into the default profile's store, silently.
- **`api_server` rebuilds the agent per request but not the memory provider.** The adapter bypasses
`TurnRunner` and the agent cache (per-request callbacks, model route, ephemeral prompt), so
`platforms/api_server_memory_sessions.py` parks each session's initialised `MemoryManager` between
requests (exclusive check-out in `_create_agent`, check-in in the turn's `finally`, keyed by profile
home + `agent.session_id`; idle/LRU eviction shuts down under the owning home) and
`AIAgent(memory_manager=...)` adopts it without a second provider init. Without it the previous
turn's queued recall never reaches the next request (#120116).
- **`multiplex_profiles: false` is not "no scope ever".** A native hosted room serving a second
profile flips the process-wide guard (`tui_gateway/launch_profile_policy.py::
activate_multi_profile_hosting`) inside the gateway process, after the adapters were wired; every
standalone entry point (`run_turn.py::_profile_scope_for_source`, the primary adapter's message /
busy / platform-event handlers via `run_adapters.py::_standalone_scoped`) then binds the launch
profile's OWN scope through `run_turn.py::_standalone_launch_scope` — `.env` over the env frozen at
activation, never a `.env`-only rebuild (systemd / `op run` keys have no file) and never live
`os.environ`. Gate a new standalone path on that helper, not on the config flag (#112878).
- **Hooks and observers register per served profile.** `builtin_hooks/`, `agent/shell_hooks.py::
register_from_config` and lifecycle observers are prepared under each profile's scope at startup
and on profile add/remove; idempotence keys include the profile, and `hooks/` paths resolve at call
time (a module constant freezes to the first importer).
- **Adapter YAML never reaches `os.environ` under multiplex.** `platforms/_shared.py::
apply_yaml_bridge` seeds `PlatformConfig.extra` and skips the environ write under a secondary
profile's scope; gate/allowlist reads go through `platform_gate_env`. A `if not os.getenv(X):
os.environ[X] = …` bridge is first-profile-wins across the process — test two profiles with
conflicting flags before touching precedence.
- **Unserved is reported, never silent.** Shared-ingress platforms (WhatsApp bridge, Relay) run on
the default profile only; a secondary enabling one is logged once with the remedy and stamped
into runtime status (`run_adapters.py::_note_unserved_secondary_platform`). `needs_attention` is
set and cleared at the single writer (`_update_platform_runtime_status`) on the connect path.
- **One launch-home identity.** "Does this task serve a routed profile?" compares the override
with `hermes_constants.get_routing_process_hermes_home()` (`agent/secret_scope.py::
serves_routed_profile` and `_is_process_home`, `tools/environments/local.py::_is_routed_home`,
`hermes_cli/env_loader.py::_process_hermes_home`), never with `os.environ["HERMES_HOME"]` read
live: an embedding host that mirrors the served profile into the env var per turn (Hermes
WebUI) pins its own home with `pin_process_hermes_home()`, and without a pin the resolver is
`get_process_hermes_home()` unchanged. Do not add another routing decision that compares
against `get_process_hermes_home()` directly; that resolver is for process-level assets.
## Tests
`tests/gateway/`. Adapter tests exercise the real delivery path with a fake transport; never
assert on hardcoded platform lists or command counts (root: no change-detectors). Session-key and
guard behaviour are invariants worth a test; platform API quirks belong in connector comments +
tests, not in prose.