mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-11-17 04:11:35 +00:00
Symptom: a host that serves several profiles from one process and mirrors the active turn's profile into `os.environ["HERMES_HOME"]` for legacy readers (Hermes WebUI does this on every chat turn, next to the context-local override) makes every launch-home decision see the served profile as the launch profile. Two profiles that both configure `atlassian` with different credentials share whichever MCP connection came first: a READ_ONLY_MODE=false profile ends up calling a read-only server (nesquena/hermes-webui#7721). The same misjudgement leaves the launch residue in the served profile's child env, seeds the launch profile's bridged allow-all grant into the served profile's secret scope, and lets the served profile's `terminal.*` config bridge into the shared process env. Cause: four launch-home checks compare the task's override with `get_process_hermes_home()`, which reads `HERMES_HOME` live: `agent.secret_scope.serves_routed_profile` (keys the MCP ledger via `_mcp_registry_scope`, #108352 / #111481, and the check_fn cache, #111151), `agent.secret_scope._is_process_home`, `tools.environments.local._is_routed_home` and `hermes_cli.env_loader._process_hermes_home`. Under the mirror the two sides are equal for every turn. Change: `hermes_constants.pin_process_hermes_home(path | None)` lets the host record the home it serves as its own; `get_routing_process_hermes_home()` returns the pin when set, else `get_process_hermes_home()`; the four checks compare against it. The pin is deliberately NOT folded into `get_process_hermes_home()`: `get_hermes_home()` falls back to it for tasks carrying no override (MCP loop, spawners), and the host's mirror exists precisely so those readers see the served profile. Only "is this task routed / is this the launch home" changes. Unpinned, behaviour is byte-for-byte the old one; hosts that never mutate `HERMES_HOME` need not call it. `activate_multi_profile_hosting()` is not the seam for this: it flips `get_secret` fail-closed process-wide and freezes the launch env, which an embedding host cannot adopt as a bug fix. Tests (2 invariants, parametrized over the four checks plus the MCP ledger key; red on main, green here): pinned + mirrored env -> the served home is routed and the launch home is not, the MCP key is `(home_key, name)`, `get_process_hermes_home()` still follows the env var; never pinned or pinned-then-cleared -> old semantics, including "a mirrored env var IS the launch home". `tests/conftest.py` resets the pin per test so the module-global cannot leak between files. Live repro (WebUI + a stdio FastMCP server named `atlassian` in two profiles, one gated by READ_ONLY_MODE): base -> one ledger key `'atlassian'`, the write profile lists only the read-only tools; fixed -> `(<read_home_key>, 'atlassian')` and `(<write_home_key>, 'atlassian')`, each profile lists its own tools. Docs: `gateway/AGENTS.md` § Profile scope (one launch-home identity) and the isolation table in `website/docs/user-guide/multi-profile-gateways.md`. Also maps the author e-mail under contributors/emails/ (attribution check). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Co-Authored-By: Claude Fable 5.1 <[email protected]>
255 lines
20 KiB
Markdown
255 lines
20 KiB
Markdown
# gateway/ — messaging gateway, adapters, delivery
|
|
|
|
Applies on top of the root `AGENTS.md`. Long-form: `website/docs/developer-guide/gateway-internals.md`.
|
|
New platform adapter: follow `gateway/platforms/ADDING_A_PLATFORM.md` step by step.
|
|
|
|
## Shape
|
|
|
|
`gateway/run.py` is the facade; phases live in `run_*.py` (startup, adapters, inbound, turn, busy,
|
|
goals, notifications, shutdown, ...), sessions in `session*.py`, slash handlers in
|
|
`slash_commands_*.py` mixins, authorization in `authz_mixin.py`, adapters in `platforms/<name>.py`
|
|
over `platforms/base.py`. `builtin_hooks/` is the extension point for always-registered gateway
|
|
hooks (none shipped). The gateway reads user YAML **raw** (`run.py` + `config.py`), not through
|
|
`DEFAULT_CONFIG` — a key the CLI sees but the gateway doesn't means you're on the wrong loader
|
|
(`hermes_cli/AGENTS.md`). Each adapter picks a base toolset (Telegram → `"messaging"`).
|
|
|
|
Slash commands: handlers are looked up by name through `_command_handler_table`; a command is
|
|
listed in `_IDLE_COMMANDS` or `_PLAIN_COMMANDS` (works mid-run) in `run_busy.py`. No
|
|
`if canonical == ...` chains. Registry + adding a command: `hermes_cli/AGENTS.md`.
|
|
|
|
## The gateway has TWO message guards — both must bypass approval/control commands
|
|
|
|
While an agent is running, an inbound message passes two sequential guards: (1) the **base
|
|
adapter** (`platforms/base.py`) queues it in `_pending_messages` when `session_key in
|
|
self._active_sessions`; (2) the **runner** (`run.py`/`run_busy.py`) intercepts `/stop`, `/new`,
|
|
`/queue`, `/status`, `/approve`, `/deny` before they reach `running_agent.interrupt()`. Any new
|
|
command that must reach the runner while the agent is blocked (approval prompts) MUST bypass BOTH
|
|
guards and be dispatched inline — never via `_process_message_background()`, which races session
|
|
lifecycle.
|
|
|
|
## Streaming delivery contract (stream-is-the-message adapters)
|
|
|
|
Adapters with `draft_stream_is_message = True` (relay Slack native streaming) keep ONE cumulative
|
|
native stream per turn; the stream IS the final message. Four invariants, each from a live
|
|
duplicate-final incident (NS-658 canary ledger, hermes#85796 / gateway-gateway#210); violating any
|
|
re-creates a duplicate or frozen stream:
|
|
|
|
1. **Draft frames are prefix-stable.** Frame N must be a string prefix of frame N+1. Never mutate
|
|
drafts per tick — no fence-closing (`ensure_closed_code_fences`), cursor suffix, segment-state
|
|
resets at tool boundaries, or mrkdwn conversion. A non-prefix frame forces a whole-snapshot
|
|
re-append ("stacked copies"). The finalize path may still transform the real final.
|
|
2. **The consumer declares the final; the adapter never guesses.** `finish(final_text)` carries the
|
|
completed `final_response` (verifier footer, completion explainer included). New post-stream
|
|
augmentation MUST ride this payload — mutating `final_response` after the seal re-opens the
|
|
`delivered_final_matches` mismatch → corrective duplicate send.
|
|
3. **Interim sends carry `metadata["_interim_send"] = True`.** Any consumer-side `adapter.send()`
|
|
that is not the turn-final (commentary, segment-tail flushes) must set it or seal-interception
|
|
seals the live stream with interim text. Seal-interception exists at BOTH egress doors (`send()`
|
|
and `send_for_platform()`); a new egress door needs the same two checks.
|
|
4. **Reconcile by edit, never by plain send.** A lane delivering a final beside a sealed stream
|
|
(queued follow-ups, media-accompanied finals) first tries `edit_message` on the consumer's
|
|
`message_id`; plain `send()` only when no editable message exists. A sealed native stream is a
|
|
regular message — `chat.update` works (live-verified).
|
|
|
|
Contract tests: `tests/gateway/test_stream_final_contract.py` (mutation-checked). Slack ground
|
|
truth: `chat.*Stream` speaks STANDARD markdown, not mrkdwn; `stopStream.markdown_text` APPENDS;
|
|
`startStream`/`stopStream` are Tier 2 (~20/min). Check `draft_stream_is_message is True` —
|
|
MagicMock adapters in older tests auto-create truthy attributes.
|
|
|
|
## Background process notifications
|
|
|
|
`terminal(background=true, notify_on_complete=true)` starts a gateway watcher that detects
|
|
completion and triggers a new agent turn. Verbosity: `display.background_process_notifications`
|
|
(or `HERMES_BACKGROUND_NOTIFICATIONS`): `concise` (default; one line, failures append an output
|
|
tail), `all` (running updates + final raw output), `result` (final raw output only), `error`
|
|
(final raw output only on non-zero exit), `off`.
|
|
|
|
The watcher is armed on the gateway loop at registration time (`terminal_tool_background.py::
|
|
_register_completion_watcher` → `run_notifications.py::arm_process_watcher`); `pending_watchers`
|
|
is only the fallback for processes registered before the gateway serves (checkpoint recovery) or
|
|
while it stops, drained at startup and post-turn. Agent-notify watchers send no user-facing
|
|
receipt (the agent's next turn is the report) unless the launching turn is still running at exit
|
|
— then the injection only queues a follow-up, so the concise receipt goes out immediately.
|
|
|
|
The idle completion watcher also drains `watch_match` / `watch_disabled`; no user follow-up is
|
|
required. Notify-off drains these without waking. Transport failures are retried; unavailable
|
|
durable completion owners/transports do not spend delivery attempts. Profile-namespaced process
|
|
events keep their owning adapter, including raw progress/final notices. Adapter acceptance is
|
|
at-least-once admission, not proof that a model turn or final outbound reply completed. API-server
|
|
async completions remain durable delivery rows, never autonomous new model turns. Require the
|
|
`admit_internal_event` receipt for completion/watch injection: a handler returning None is not
|
|
acceptance. Refused admission refunds every claimed batch sibling without spending an attempt;
|
|
actual delivery errors keep their bounded retry policy. Recognized raw API routes resolve after
|
|
persisted messaging origins and defer quietly when unavailable; malformed routes still warn.
|
|
|
|
Cron execution has its own session. Eligible continuable deliveries may mirror or seed the
|
|
reply-facing conversation: origin, origin-less home fallback, user-written bare-platform home,
|
|
or opted-in explicit targets. `all` expansions do not gain home mirror eligibility. Mirrored
|
|
briefs are labelled user turns appended at a turn boundary, preserving role alternation
|
|
(`cron/AGENTS.md`).
|
|
|
|
## `/login` (off-turn, paired DM only)
|
|
|
|
`/login` is registered in `hermes_cli/commands.py` with `busy_policy="dispatch"` and
|
|
`desktop="settings"`, listed in `run_busy.py::_PLAIN_COMMANDS`, and handled by
|
|
`GatewayLoginCommandsMixin` (`gateway/slash_commands_login.py`). It refuses outside a paired DM:
|
|
`chat_type in {"dm","private"}`, a truthy `chat_id`, and a platform whose `"dm"` really is a paired
|
|
conversation — ntfy, raft and a2a all report `chat_type="dm"` for a broadcast topic, a channel and
|
|
an agent peer, so posting a consent link there would publish it.
|
|
|
|
**It binds the whole install.** The sign-in writes the singleton `providers.nous`, so whoever
|
|
approves the code owns this gateway's inference and connectors for every chat it serves. Slash
|
|
gating is opt-in (`gateway/slash_access.py`): with no `allow_admin_from` set, every user allowed to
|
|
DM the bot can run it. Operators of shared gateways must set it.
|
|
|
|
The handler returns its ack at once and drains `anon_auth.run_sign_in` on a **private single-worker
|
|
executor** (`_login_executor`), never the shared 10-thread gateway pool — a promotion wait can last
|
|
the code's full expiry, and a live worker in the shared pool makes shutdown skip the SessionDB
|
|
close/checkpoint. One attempt per process, stamped with the identity that started it: the same
|
|
identity's second `/login` supersedes (the loser ends with the superseded copy pushed into its own
|
|
chat); a different identity is refused. `task.cancel()` cannot interrupt a blocking poll inside a
|
|
worker thread, so shutdown and supersede both work through `attempt.cancelled`, which
|
|
`wait_for_promotion` now polls on a ≤1 s tick. A shutdown-cancelled attempt pushes nothing — the
|
|
task is dying and its adapters may already be gone — but if the server had already completed the
|
|
transfer it still persists, because that transfer is irreversible.
|
|
|
|
On completion the handler **evicts** every cached agent still on `nous/welcome` and clears any
|
|
session model override pinned to it. It does not switch agents in place and does not write a model
|
|
override: `settle_after_upgrade` already moved `model.default`/`model.base_url` in the config, every
|
|
turn re-resolves the config and the credentials, and `_agent_config_signature` already forces a
|
|
rebuild when the route changes — so an in-place swap buys no cache warmth and an override would pin
|
|
an expiring access token that nothing refreshes.
|
|
|
|
## Gateway lifecycle vs. the Desktop app
|
|
|
|
`hermes serve` (control plane, desktop-spawned child) dies with the app — by design. The messaging
|
|
gateway (`gateway run`) SURVIVES the app: the serve backend's `/api/gateway/*` endpoints spawn it
|
|
detached (`_spawn_hermes_action` — `start_new_session` / `DETACHED_PROCESS`), so `before-quit`'s
|
|
SIGTERM never reaches it and bots keep running. The known breach is the Windows shim-unlock
|
|
teardown (`taskkill /T /F` on venv-shim holders, #85265), which exists to let updates proceed and is
|
|
replaced by #92091's `pause-for-update`. Do NOT "fix" gateway-dies-with-app by re-parenting the
|
|
gateway under the backend, and do NOT "fix" update locks by widening the tree-kill. Gateways stamp
|
|
`code_sha`/`code_version` into `gateway_state.json` (`status.py`) so the updater can verify a fleet.
|
|
|
|
## Profile scope (adapters, turns, and everything between turns)
|
|
|
|
- **One identity per inbound event, canonicalized FIRST.** `gateway/session_identity.py::resolve_identity`
|
|
answers "which bot received it / who may admit it / where does it run" ONCE per event and pins a
|
|
frozen `RoutingIdentity` on the source (wire-invisible, like `_transport_adapter_ref`). Every
|
|
ingress path calls the canonicalize seam before it derives a key: the adapter side
|
|
(`platforms/base.py::_canonicalize` — `handle_message`, `_enqueue_text_event`, Telegram photo /
|
|
album routing, `_handle_message_while_active`, every `_source_session_key`) and the runner side
|
|
(`run_adapters.py::_canonicalize` — the per-profile / default message, busy and platform-event
|
|
handlers, the adapter auth-check callback, `run_inbound.py::_hm_admit_event`). No key derivation
|
|
before it; an unresolved identity under multiplexing (route to an unserved profile) is dropped
|
|
with one WARNING at the first seam it reaches, never keyed into `agent:main`. `_transport_owner`,
|
|
`_authorization_home_for_source`, `_resolve_profile_home_for_source`, `_session_key_profile` and
|
|
`_resolve_profile_for_key` read the identity when present and fall back to their old chain only
|
|
for sources nothing resolved (restored rows, hand-built sources). Never derive a second answer
|
|
next to the identity; extend the object. `transport_profile` ≠ `runtime_profile` is normal
|
|
(shared bot → routed satellite). A source copy goes through `session_identity.replace_source`
|
|
so the identity travels with it (`_apply_topic_recovery` does).
|
|
- **Intake vs delivery adapter.** `authz_mixin.py::_intake_adapter_for(source)` is the bot that
|
|
RECEIVED the event (live transport ref → relay socket → identity's `transport_profile`); it gates
|
|
intake policy (ignored channels, relay fronting, re-dispatch of a still-live event) and fails
|
|
closed to `None` for a source with no live provenance. `_delivery_adapter_for(source)` is the bot
|
|
that ANSWERS — the receiving bot whenever known, else the unique owner of `(platform,
|
|
runtime_profile)` via `_adapters_for_profile`. Never read `self.adapters[platform]` for a source;
|
|
never add a third resolver. The matrix (`tests/gateway/test_multiplex_transport_matrix.py`):
|
|
|
|
| Topology | runtime | intake | delivery |
|
|
|---|---|---|---|
|
|
| per-credential bot, no route | owner | owner adapter | owner adapter |
|
|
| shared credential → satellite (`profile_routes`) | routed | receiving adapter | receiving adapter (`_is_shared_bot_satellite` after restart) |
|
|
| shared bot → profile that owns its own bot | routed | receiving adapter | receiving adapter (conversation continuity; #70625's "routed bot" reading was not adopted) |
|
|
| secondary-owned bot → `default` (`bot_profile`) | default (`agent:main`) | receiving adapter | receiving adapter |
|
|
| restored / hand-built, no live provenance | stored `source.profile` | **`None`** | unique owner of `(platform, runtime)`; a disconnected secondary → `None`, never the default bot |
|
|
- **Identity survives the process.** `SessionEntry.transport_profile` (routing index +
|
|
`sessions.transport_profile`, nullable, reconciled by `SCHEMA_SQL`) persists the receiving bot
|
|
next to the key namespace; the namespace says where a lane RUNS, the column says which bot may
|
|
DELIVER to it. Anything reviving a session from durable state (auto-resume, heartbeat restore,
|
|
plugin injection, background-process events) reads `entry.origin` through
|
|
`authz_mixin.py::_restored_source`, which re-pins a `RoutingIdentity(transport=None)` via
|
|
`session_identity.restore_identity`; `_delivery_adapter_for` then delivers through that bot's
|
|
adapter or nothing (never the default bot by heuristic). Rows without the column (pre-PR-5)
|
|
keep the `_is_shared_bot_satellite` fallback. Deferred callbacks capture the identity/home at
|
|
command time (`/model` picker); `_run_in_executor_with_context` carries the scope over thread
|
|
hops. Relay: `_with_scope` echoes the routed `profile` on every outbound frame and `follow_up`
|
|
derives it from the key namespace so the connector stamps it on the next `passthrough_forward`.
|
|
Ambient `get_active_profile_name()` reads in `gateway/` are boot-only and marked
|
|
`# launch profile, pre-identity`; a path with an identity reads `identity.runtime_profile`.
|
|
- **Token locks.** An adapter that connects with a unique credential (bot token, API key) calls
|
|
`acquire_scoped_lock()` from `gateway.status` in `connect()`/`start()` and `release_scoped_lock()`
|
|
in `disconnect()`/`stop()`, so two profiles cannot share one credential. Canonical:
|
|
`plugins/platforms/irc/adapter.py`.
|
|
- **Multiplex profile-scoped env reads MUST fail closed — never borrow from `os.environ`**
|
|
(`agent/secret_scope.py`; #72348, #86905). Under `gateway.multiplex_profiles`, `os.environ` holds
|
|
the DEFAULT profile's values; a secondary profile's `.env` exists only in its secret scope,
|
|
bound per profile *activity* by `_profile_runtime_scope` (turn, callback, eviction, tick — never
|
|
only "per turn"). All profile-level env config — credentials
|
|
(`app_secret`, tokens) AND authorization (`FEISHU_ALLOWED_USERS`, `{PLATFORM}_ALLOW_ALL_USERS`,
|
|
`GATEWAY_ALLOW_ALL_USERS`, `group_policy`, `allow_bots`) — is read scope-aware: adapters via
|
|
`gateway.platforms._shared.get_scoped_secret` (the ONE implementation; adapters import it as
|
|
`_get_scoped_secret`; `extra_or_secret` / `seed_extra_from_env` / `env_is_connected` build on it),
|
|
gateway authz via `_shared.platform_gate_env` (imported by `authz_mixin.py` as `_auth_env`). Scope installed +
|
|
multiplex active → a scoped miss returns the **default**, NEVER `os.environ` (a leaked allowlist
|
|
skips the allow-all check and silently rejects every secondary-profile sender, #86905). The
|
|
unscoped default-profile path (`UnscopedSecretError`) and single-profile deployments keep the
|
|
`os.environ` read — there it IS the profile's own value. Never re-implement the reader in an
|
|
adapter (the `try get_secret / except UnscopedSecretError: os.getenv` shape drifts into a
|
|
fallback-after-miss leak); import the shared one. `tests/gateway/test_shared_platform_boilerplate.py`
|
|
asserts every plugin's `_env_enablement` reads only through it.
|
|
- **Scope is bound per profile activity, not per turn.** `run.py::_profile_runtime_scope(home)`
|
|
wraps a routed turn (`run_turn.py::_profile_scope_for_source`); the same binding wraps every path
|
|
that touches a session's home, secrets or terminal scope with no turn on the stack: release and
|
|
eviction (`run_agent_cache.py::_run_release_in_profile_scope` — TTL, LRU and memory-pressure
|
|
eviction all route through it so `on_session_end`/memory flush hit the OWNING profile's provider),
|
|
shutdown (`run_shutdown.py::_finalize_session`), post-turn media delivery
|
|
(`platforms/base.py::_media_delivery_scope`), deferred callbacks (pickers, reactions — capture the
|
|
routed home at command time, re-enter the scope in the callback), notifiers and outbound webhooks.
|
|
Resolve the owning home from the session record (`profile_home`, `agent:<profile>:` key), never
|
|
from `os.environ`, which holds the launch profile. Why: eviction that flushed under the launch scope
|
|
wrote a secondary profile's memories into the default profile's store, silently.
|
|
- **`api_server` rebuilds the agent per request but not the memory provider.** The adapter bypasses
|
|
`TurnRunner` and the agent cache (per-request callbacks, model route, ephemeral prompt), so
|
|
`platforms/api_server_memory_sessions.py` parks each session's initialised `MemoryManager` between
|
|
requests (exclusive check-out in `_create_agent`, check-in in the turn's `finally`, keyed by profile
|
|
home + `agent.session_id`; idle/LRU eviction shuts down under the owning home) and
|
|
`AIAgent(memory_manager=...)` adopts it without a second provider init. Without it the previous
|
|
turn's queued recall never reaches the next request (#120116).
|
|
- **`multiplex_profiles: false` is not "no scope ever".** A native hosted room serving a second
|
|
profile flips the process-wide guard (`tui_gateway/launch_profile_policy.py::
|
|
activate_multi_profile_hosting`) inside the gateway process, after the adapters were wired; every
|
|
standalone entry point (`run_turn.py::_profile_scope_for_source`, the primary adapter's message /
|
|
busy / platform-event handlers via `run_adapters.py::_standalone_scoped`) then binds the launch
|
|
profile's OWN scope through `run_turn.py::_standalone_launch_scope` — `.env` over the env frozen at
|
|
activation, never a `.env`-only rebuild (systemd / `op run` keys have no file) and never live
|
|
`os.environ`. Gate a new standalone path on that helper, not on the config flag (#112878).
|
|
- **Hooks and observers register per served profile.** `builtin_hooks/`, `agent/shell_hooks.py::
|
|
register_from_config` and lifecycle observers are prepared under each profile's scope at startup
|
|
and on profile add/remove; idempotence keys include the profile, and `hooks/` paths resolve at call
|
|
time (a module constant freezes to the first importer).
|
|
- **Adapter YAML never reaches `os.environ` under multiplex.** `platforms/_shared.py::
|
|
apply_yaml_bridge` seeds `PlatformConfig.extra` and skips the environ write under a secondary
|
|
profile's scope; gate/allowlist reads go through `platform_gate_env`. A `if not os.getenv(X):
|
|
os.environ[X] = …` bridge is first-profile-wins across the process — test two profiles with
|
|
conflicting flags before touching precedence.
|
|
- **Unserved is reported, never silent.** Shared-ingress platforms (WhatsApp bridge, Relay) run on
|
|
the default profile only; a secondary enabling one is logged once with the remedy and stamped
|
|
into runtime status (`run_adapters.py::_note_unserved_secondary_platform`). `needs_attention` is
|
|
set and cleared at the single writer (`_update_platform_runtime_status`) on the connect path.
|
|
- **One launch-home identity.** "Does this task serve a routed profile?" compares the override
|
|
with `hermes_constants.get_routing_process_hermes_home()` (`agent/secret_scope.py::
|
|
serves_routed_profile` and `_is_process_home`, `tools/environments/local.py::_is_routed_home`,
|
|
`hermes_cli/env_loader.py::_process_hermes_home`), never with `os.environ["HERMES_HOME"]` read
|
|
live: an embedding host that mirrors the served profile into the env var per turn (Hermes
|
|
WebUI) pins its own home with `pin_process_hermes_home()`, and without a pin the resolver is
|
|
`get_process_hermes_home()` unchanged. Do not add another routing decision that compares
|
|
against `get_process_hermes_home()` directly; that resolver is for process-level assets.
|
|
|
|
## Tests
|
|
|
|
`tests/gateway/`. Adapter tests exercise the real delivery path with a fake transport; never
|
|
assert on hardcoded platform lists or command counts (root: no change-detectors). Session-key and
|
|
guard behaviour are invariants worth a test; platform API quirks belong in connector comments +
|
|
tests, not in prose.
|