mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-11-16 19:36:54 +00:00
* feat(i18n): layered catalogs — plugin packs and user overlay over bundled locales
* feat(tui): i18n layer — en catalog, nanostore runtime, RPC pack loader, _keys.tui.json emitter
ui-tui/src/i18n/: en.ts (facade over topical siblings under en/), types.ts
(Translations + dotted TranslationKey derived from en), runtime.ts ($locale/
$catalog atoms, translateFrom active→en→key, pack merge with string→fn
wrapping for {0}/{1} placeholders), loader.ts (display.language →
i18n.catalog {lang, surface:'tui'}, English when the method is missing),
useT()/useLocale() hooks, t() for non-React code. useConfigSync feeds the
loader from the existing config.get full hydration. `npm run i18n:keys`
writes locales/_keys.tui.json (sorted flat key list) and runs before build.
* feat(plugins): provides_locales manifest field, ctx.register_locale/register_locale_dir, manifest-only language packs
* chore(tui): split en catalog siblings by lane (slash sibling)
* feat(plugins): validate language packs — parse, text-only, key-subset WARN against en / _keys exports
* feat(tui_gateway): i18n.languages / i18n.catalog RPC + regenerated contracts
* feat(config): display.language accepts any supported_languages() id, refuses unknown ids with the list
* docs(i18n): language packs user guide, pluggable display.language, plugin developer section, AGENTS notes
* feat(plugins): report language-pack layers in the mid-run activation summary
* feat(tui): wire status bar, composer placeholders, hotkey help and approval/clarify/confirm prompts through i18n
StatusRule maps compared state values (ready/running…/summoning) to catalog
text at render via displayStatus(); hotkeys()/placeholder() resolve lazily so
a pack that arrives after boot applies. Catalog grows to 81 keys.
* feat(desktop): pluggable app locales — registry, host.i18n.registerAppLocale, backend packs, keys emitter
- Locale widens to string (BundledLocale keeps the union); TRANSLATIONS stays
the bundled record and every consumer resolves through the registry.
- src/i18n/registry.ts: registerAppLocale(id, {endonym, rtl, translations})
layers partial packs (nested or flat dotted) over bundled/en via
mergeTranslations; a string over a function-valued en entry becomes a
positional {0}/{1} formatter; $appLocaleVersion bumps so translators
re-render; per-source disposers + replaceAppLocaleSource for atomic swaps.
- Backend packs: i18n.languages + i18n.catalog {surface:'desktop'} feed the
registry as source 'backend' (method-not-found is silent); re-synced on
socket open, display.language change and profile switch. A saved pack-only
language is promoted once its pack registers.
- SDK: host.i18n.registerAppLocale / languageOptions; ctx.i18n.registerAppLocale
tracked for unload. Docs in the desktop plugin SDK guide + skill reference.
- Language switcher lists bundled ∪ registered ∪ backend, endonym-only; RTL
from the registry (applyDocumentLocale takes rtl).
- npm run i18n:keys emits locales/_keys.desktop.json (wired into build).
* i18n(cli): route /topup + /subscription copy through t() (cli.billing.*, cli.subscription.*)
Module-level copy tables and modal choice tuples in cli_billing_mixin.py froze
English at import, before display.language was known. They are now key tables /
builder functions evaluated at call time; every user-facing line in the /usage
balance block, /subscription and the five /topup screens reads the catalog.
Choice VALUES stay English identifiers. Fragment-assembled status lines
(Plan: … → cancels · $x left · renews …) become full templates.
* i18n(gateway): exec-approval card contract + base/run/run_busy/run_inbound replies through t()
- base_exec_approval: EA_* English constants stay; add ea_header_text()/ea_reason_label_text()/
ea_smart_deny_line_text()/ea_default_reason_text()/ea_action_labels()/approval_timed_out_notice()
accessors; deadline + timed-out notice resolve via gateway.exec_approval.*
- BasePlatformAdapter._EA_HEADER/_EA_REASON_LABEL/_EA_SMART_DENY_LINE/_EA_ACTION_LABELS become
properties (adapters still shadow them with markup class attrs)
- run.py: provider error replies table holds catalog keys; _CONTEXT_OVERFLOW_REPLY -> _context_overflow_reply()
- run_busy/run_inbound: typed approval + slash-confirm matchers accept English ∪ approval.inputs.* (t())
- locales/en.yaml: gateway.exec_approval/busy/errors/... namespaces
* i18n(cli): wire modal, loops, agent-setup mixins through t() (cli.* keys)
* i18n(platforms): route Slack, Matrix and Feishu user-facing text through t()
Exec-approval markup overrides (_EA_HEADER/_EA_REASON_LABEL/_EA_SMART_DENY_LINE/
_EA_ACTION_LABELS) become per-call properties over the shared
gateway.exec_approval.* contract keys, so Slack's 3000-char section budget
measures the resolved template. Slack _APPROVAL_DECISIONS/_CONFIRM_DECISIONS,
Feishu _APPROVAL_LABEL_MAP and Matrix _EA_LEGEND/_EA_TYPED_HINT turn into
key tables resolved at click time; the Matrix typed hints become whole
sentences per offered tier instead of spliced fragments. Slack button labels
are cut to 75 chars and select placeholders to 150 after translation; the
model-facing clarify fallback answer ('choice N') stays English while the
card copy localizes.
locales/en.yaml gains the gateway.exec_approval.* contract keys plus the
platform.shared.* / platform.slack.* / platform.matrix.* / platform.feishu.*
namespaces (and the keys for the other adapters wired in follow-up commits).
* i18n(gateway): run_turn / run_turn_runner / approval-settle copy through t()
- status hints, proxy errors, background task notices, progress heartbeats, session info lines
- tool progress chrome (tool_head/tool_pending/tool_preview/tool_verbose) shared by base.format_tool_event
- run_turn_runner:1406 Chinese clarify placeholder -> gateway.clarify.native_stream_placeholder (zh text kept in zh.yaml)
- _UNEXPECTED_SILENCE_REPLY/_CLARIFY_EXPIRED_NOTICE -> accessor functions
* i18n(platforms): route Google Chat and Teams user-facing text through t()
Google Chat clarify card, typing placeholder, orphan-card labels and the whole
/setup-files reply set (module constants become platform.google_chat.setup_files.*
keys resolved at reply time). The attachment-fallback notice that shipped
hardcoded in Spanish is keyed with an English en value; es.yaml carries the
original Spanish text for those four keys.
Teams approval card header/reason use the gateway.exec_approval.* contract,
_APPROVAL_LABELS becomes a key table resolved at click time, and the meeting
summary writer resolves its section headings/fallbacks per render.
* i18n(platforms): route LINE, WeCom, email, DingTalk, IRC and Home Assistant text through t()
LINE default copy constants become catalog keys resolved in __init__ (the
LINE_*_TEXT / extra.* operator overrides still win); the busy-ack bypass
matcher keys on the leading emoji marker only, so it keeps firing once the
gateway busy heads are localized. WeCom media size/format notices that shipped
hardcoded in Chinese are keyed with English en values and zh.yaml carries the
original Chinese text. DingTalk emotion bubbles resolve per send.
* i18n(cli): route /model switch output and -q status lines through t() (cli.model.*, cli.single_query.*)
Switch-summary labels shared with the gateway reuse gateway.model.* keys
(provider/context/max-output/capabilities/prompt-caching); CLI-only variants
(glyph or no-backtick forms) live under cli.model.*. The hand-padded /model usage
block becomes a (form, description-key) table padded at render time so the
command syntax stays fixed while descriptions translate. -q 'Error:' reuses
gateway.model.error_prefix.
* i18n(cli): route TUI panel/hint/placeholder copy through t() (cli.tui.*)
_APPROVAL_CHOICE_LABELS and _TUI_MODAL_HINTS become key tables resolved at
render time; vault/sudo panel bodies are one catalog value per panel split on
newline; inline plurals use <key>_one/<key>_other. Adds the cli.* namespace
(shared/tui/voice/render/subagents/dock) to locales/en.yaml.
* i18n(cli): voice/wake-word CLI copy through t() (cli.voice.*)
RuntimeError texts raised in _voice_start_recording are human copy (callers
print {e}) and are keyed; the Termux requirement-check match stays English.
Wake state ids stay internal, only their labels localize.
* i18n(cli): live-work dock, subagent monitor and render copy through t()
cli.subagents.* / cli.dock.* / cli.render.*; count fragments pluralize via
_one/_other keys, verdict table holds keys resolved at paint time so width
clipping measures the translated text.
* i18n(gateway): unauthorized/pairing, voice, topics, shutdown, startup, notifications, kanban pings through t()
* i18n(cli): move tips + composer placeholders into the catalog (tips.tNNN / tips.placeholder.pNN)
get_random_tip()/get_random_composer_placeholder() pick a key from the English catalog
(the parity baseline, probed once per process) and resolve it through t() for the active
language, so language packs translate tips like any other string. Also lands the cli.*
en.yaml namespace consumed by the CLI info/help/error-copy wiring in the next commit.
* i18n(cli): wire chat-turn + session mixins through t(); kanban log trimmer matches t() output
* test(cli): assert TUI/dock/voice copy via t(key); prove labels resolve at render time
Pinned-English assertions in the approval-UI, live-work dock and voice tests now
go through the catalog. New test swaps the catalog after import and checks the
approval panel + hint row follow it (the reason _APPROVAL_CHOICE_LABELS and
_TUI_MODAL_HINTS became key tables).
* i18n(cli): route CLI info/help/error copy through t() (cli.* namespace)
cli_info_mixin: /help consumes CommandDef.describe() (added to commands.py: slash.<name>.description
with fallback to .description), section titles/skill/quick-command headers, /tools, /toolsets,
/usage labels, /context, /whoami, /insights, /gateway status, tool-progress labels, bang-shell
denials, MCP config-watch + /reload-mcp confirm/reload lines, /reload-skills, and the session-store
warning all read the catalog at call time (module-level label tables became functions so the
active language is honoured after startup). cli.py: worktree cleanup, tirith warning, show_config
(labels re-padded at print time), quick/plugin/skill slash-command errors, ambiguous-command hint,
stdin error, gateway start, profile warning. cli_chat_error_copy / cli_unknown_command /
cli_output / cli_init_mixin: chat error panel copy, did-you-mean lines, n-more / yes-no prompt
(localized affirmative initial alongside 'y'), unknown-toolsets warning.
* i18n(w1a): wire agent display/explainers/approval + slash registry/help through t()
- hermes_cli/commands.py: CommandDef.describe() resolves slash.<name>.description
at call time; category labels via slash.category.*; help/alias/usage suffixes
via slash.shared.*; gateway_help_lines and commands_platforms/slash_exec use them.
- agent/display.py: display.verb.* resolved at call time via get_tool_verb();
bridge/spinner/thinking-verb/cute-row/failure/preview/diff text via display.*.
- agent/turn_explainers.py: exit-reason / persistence-cause tables become call-time
lookups (explainer.exit.*, explainer.persistence.*, explainer.file_mutation.*).
- agent/background_review.py, session_activity.py, context_breakdown.py,
status_output.py: review summaries, iteration progress, context notices.
- tools/approval.py, approval_context.py: approval.summary.*, approval.noun.*,
approval.window.* pluralized keys.
- gateway/slash_commands*.py: remaining raw strings (busy, whoami, platform,
bundles, memory, skills, approvals, set_home, diff, update, debug, profile,
heartbeat, refine, review, subgoal, loop, retry, compress codex path, save,
sessions, model guard/errors, agents rows, topup, login). HISTORY_UNREADABLE
keeps its English constant; callers use history_unreadable() ->
gateway.shared.history_unreadable.
- locales/en.yaml: new approval/display/explainer/slash blocks + gateway leftovers.
* i18n(telegram): route adapter chat copy through t()
Approval card (header/reason/smart-deny as HTML-escaped properties), inline
button labels, callback toasts (cut at Telegram's 200-char cap), model/choice
pickers, clarify/update/slash-confirm prompts, gmail-triage labels and the
inbound-media failure notice now come from the catalog. _UNAUTHORIZED is a
lazy _unauthorized() so the import no longer binds a language. The command
menu carries a language+payload fingerprint (forum scopes re-register on
change) and BotCommand descriptions are cut at 256.
Adds gateway.exec_approval.* (WAVE2 contract), platform.telegram.*,
platform.discord.* and the slash.*.description keys the Discord table shares
with the CLI registry to locales/en.yaml.
* i18n(gateway/platforms): whatsapp_cloud, yuanbao, weixin, signal, api_server copy through t()
- whatsapp_cloud: clarify list/buttons, approve/deny + slash-confirm labels via platform.whatsapp.* (t()-then-truncate at 20/24/72 caps); _EA_HEADER becomes a property wrapping ea_header_text()
- yuanbao: SLOW_RESPONSE_MESSAGE -> slow_response_message() (platform.yuanbao.slow_response_notice; zh keeps the original text); cron-wrapper markers centralized as module constants for strip_cron_wrapper
- api_server: PROVIDER_AUTH_FAILED_LABEL/PROVIDER_RATE_LIMITED_LABEL stay English for run.py matchers; user_text() renders via t()
- signal/_format_wait, weixin voice caption, openai_routes transformed notice
- run_turn: second _UNEXPECTED_SILENCE_REPLY consumer -> accessor
* i18n(discord): route adapter chat copy through t()
Native slash-command table becomes _NATIVE_SLASH_COMMAND_SPECS holding catalog
keys; _native_slash_commands() resolves descriptions, parameter descriptions
and Choice names for the active language, each cut at Discord's 100-char cap,
and the app-command sync fingerprint now includes get_language() so a
display.language change re-syncs. Exec-approval card (gateway.exec_approval.*
contract), slash-confirm / clarify / update views, model+choice pickers,
thread creation, forum titles, voice acks, the response-truncation notice,
the unauthorized-slash security alert and the media upload-size notices all
read from platform.discord.*. Decorator-declared button labels are relabelled
in __init__ (80-char cap); embed titles cut at 256, select placeholders at
150, option label/description at 100. _UNAUTHORIZED is a lazy _unauthorized().
* i18n(cli): wire status-bar, stream, terminal mixins + terminal_input through t(); rename kwargs that shadow t(key)
* i18n: wire hermes_cli/cli_commands_mixin.py slash-command copy through t()
- 431 new leaves under cli.commands.<cmd>.* in locales/en.yaml; 12 rows reuse
existing gateway.* keys (rollback, diff, resume, branch, btw, model, reasoning)
via a _gt() helper so CLI and gateway replies stay identical.
- Module-level English tables (_BUSY_MODE_*, _REASONING_TOGGLES, _HATCH_PROGRESS,
_DIFF_LABELS, _LOCAL_ENGINE_LINES) become call-time catalog lookups keyed by id.
- Verb tables (Enabling/Disabling, Paused/Resumed/Triggered, planned/done,
Updating/Generating) are one full template per variant; plurals use
<key>_one/<key>_other via _tn(); hand-padded column labels (/snapshot list)
translate the value and re-pad at the call site.
- Multi-line usage blocks are single catalog values split with _lines().
- Model-facing system notes and DB-stored reasons stay English (EXCLUDED).
* tests: assert /handoff, /worktree, /login CLI copy via t(key) instead of pinned English
* test(i18n): pin Telegram/Discord adapter catalog wiring
Lazy unauthorized notice, exec-approval contract keys, HTML escaping before
Telegram <b> wrapping, 200-char toast / 256-char BotCommand caps, Discord
100-char app-command text and 80-char button caps, and language-bearing
command-menu fingerprints on both platforms.
* i18n: reconcile cli.shared on/off vs enabled/disabled after lane merge
* i18n: describe() in TUI-gateway slash listings; localize TUI exit resume hint
* i18n(tr): translate bundled catalog + tui pack
* i18n(ja): translate bundled catalog + tui pack
* i18n(ko): translate bundled catalog + tui pack
* i18n(zh): translate bundled catalog + tui pack
* i18n(fr): translate bundled catalog + tui pack
* i18n(af): translate bundled catalog + tui pack
* i18n(uk): translate bundled catalog + tui pack
* i18n(ar): translate bundled catalog + tui pack
* i18n(pt): translate bundled catalog + tui pack
* i18n(it): translate bundled catalog + tui pack
* i18n(es): translate bundled catalog + tui pack
* i18n(zh-hant): translate bundled catalog + tui pack
* i18n(ru): translate bundled catalog + tui pack
* i18n(hu): translate bundled catalog + tui pack
* i18n(hu): translate pre-existing English-valued leftovers (kanban wake, /context, /status, fast labels)
* i18n(de): translate bundled catalog + tui pack
* i18n(ga): translate bundled catalog + tui pack
* test(i18n): fixture matches _normalize_lang(lang, home) signature
* i18n(tui): scaffold userMessages/slashCmd en siblings
* i18n(tui): wire secure prompts + content tables
* feat(tui): i18n — wire billing, subscription, connection-setup and journey overlays
Adds en siblings billing.ts / subscription.ts / connection.ts (namespaces
billing, subscription, connection, journey) and routes every user-facing
literal in billingOverlay, subscriptionOverlay, connectionSetupOverlay and
journey through useT()/messages(). Module-level label tables became lazy
(scopeStillDeniedResult(), verbOf(T, action)); auto-reload rows dispatch on
stable ids instead of label text. Regenerates locales/_keys.tui.json.
* i18n(tui): wire slash ops/wake replies
* i18n(tui): wire pickers (modelPicker, activeSessionSwitcher, petPicker)
* i18n(tui): wire slash core/debug/setup replies
* i18n(tui): wire hubs (agents overlay/panel/controls, skills, plugins)
* i18n(tui): wire slash session/topup/subscription replies
* i18n(tui): wire chat bits (branding, thinking, messageLine, loaders, todo, queued, banner, entry)
* i18n(tui): register t3 siblings (pickers, hubs, secure, content, chatBits) and regenerate keys
* i18n(tui): wire userMessages copy through the userMessages namespace
* i18n(tui): lazy-copy test for userMessages, regenerate _keys.tui.json
* i18n(tui): wire session/gateway/lib text through the TUI catalog (lane t2)
Adds en siblings session.ts, gatewayMsg.ts, libText.ts (namespaces session,
gatewayMsg, libText) and routes user-facing literals in app/{useMainApp,
useSessionLifecycle,useInputHandlers,turnController,createServerRequestHandler,
setupHandoff,createGatewayEventHandler}.ts, gatewayClient displayed reasons,
lib/*, domain/*, hooks/* through t()/messages(). Status-bar state values that
code compares against, backend-matched strings, log lines, model-bound text,
and machine 'error:' prefixes stay literal. Regenerates locales/_keys.tui.json
(232 keys).
* i18n: serve bundled locales/<lang>.tui.yaml under overlay/packs; TUI pack parity test; regen _keys.tui.json (1250)
* i18n: translate pre-existing English stubs in bundled locales (424 leaves, 14 locales)
* tui: i18n-export-en script (English templates for pack translators)
* docs(i18n): bundled TUI packs are the bottom layer of the tui surface
* i18n(ru): translate TUI pack
* i18n(ar): translate TUI pack
* i18n(es): translate TUI pack
* i18n(pt): translate TUI pack
* i18n(ko): translate TUI pack
* i18n(de): translate TUI pack
* i18n(ja): translate TUI pack
* i18n(fr): translate TUI pack
* i18n(tr): translate TUI pack
* i18n(it): translate TUI pack
* i18n(zh): translate TUI pack
* i18n(zh-hant): translate TUI pack
* i18n(hu): translate TUI pack
* i18n(uk): translate TUI pack
1,169 missing keys translated; 81 pre-existing kept byte-identical. Parity OK missing=0 extra=0 placeholder_mismatch=0 empty=0.
Deliberately identical to en: chatBits.branding.mcpSummary ({0} MCP), chatBits.thinking.agentsHint ((/agents)), session.main.voiceStt (◉ STT), session.main.voiceTtsSuffix ( [tts]), slashCmd.core.help.tuiSection (TUI), slashCmd.core.history.hermesTag (Hermes #{0}), slashCmd.debug.heapdump.heapPath (heapdump: {0}), slashCmd.debug.mem.rss (rss), subscription.stepUp.title (Remote Spending — product feature name, as in core catalog), content.faces.* (glyph-only kaomoji).
* i18n(ga): translate TUI pack
* i18n(af): translate TUI pack
* plugin_guard: locale catalogs in language packs step down the agent-config family
A translated status line such as "Updating AGENTS.md" in locales/<lang>.yaml is UI text the loader
reads as a string leaf; it cannot edit a file. The bundled en.yaml itself tripped agent_config_mod
at critical, making any faithful language pack uninstallable. Injection shapes keep full severity.
* plugin_validate_locales: read key exports with utf-8-sig (Windows footgun lint)
* i18n(relay): route relay adapter prompt copy through t(); drop dead import-bound approval header
Adds platform.relay.* (5 keys) to en and all 16 bundled locales, reusing the sibling platform
translations for the confirm buttons and the Other option.
* ci: fix TUI import order, MDX table pipe, main's overflow-warning wording in all locales; fresh-install fixture carries the i18n kernel
- ui-tui/src/i18n/en.ts: perfectionist/sort-imports (slash before slashCmd)
- docs plugins/index.md: escape the | inside the provides_locales table cell (MDX parsed <id> as JSX)
- display.notice.uncompressed_context_overflow: adopt main's wording (names compression.enabled: false
and /compact) in en + 16 locales; the guardrail test pins that phrase
- tests/scripts/test_fresh_source_install.py: the installer tail now resolves CLI text through
agent.i18n, so the fixture tree carries the i18n kernel + en.yaml (not the agent runtime)
* docs(desktop-plugin-sdk): double-backtick the template-literal example (MDX evaluated ${n})
* test(e2e): display.language is validated against the live language set; exclude it from the arbitrary-string set property
* commands: keep the localized COMMANDS/COMMANDS_BY_CATEGORY module __getattr__ after the compat block removal
* build: never write locales/_keys.*.json from the desktop/TUI builds; regenerate the committed desktop key export
The desktop build regenerated locales/_keys.desktop.json in the checkout, so a
hermes update that rebuilt the app left the tree dirty (Desktop update E2E:
'M locales/_keys.desktop.json'). The key exports are committed artifacts pinned
to en.ts by apps/desktop/scripts/i18n-keys.test.mjs and ui-tui i18n:keys:check;
builds read them, never write them. Regenerated after main's new desktop strings.
* test: unbreak two main-red timing tests the PR merge-ref inherits
- test_local_runtime racing fake publishes the modern state record (legacy pid-only
records are rejected since 65ff3ad353; main has been red on this test since)
- test_run_progress_topics ManyProgressLinesAgent waits for the first bubble instead of
a fixed 0.35s, which a loaded CI runner does not always meet
* chore(i18n): regenerate desktop key catalog for main's new strings (model pricing, copy changelog)
* test(e2e): torture-chamber fd monitor confirms a deleted sidecar is still held before calling it a leak
SQLite's WAL last-close unlinks -shm before closing its descriptor (unixShmUnmap, then
unixShmPurge), so a healthy close shows a (deleted) -shm for microseconds; the 20ms poll
occasionally caught that window on the short-lived opener and failed the episode.
* chore(i18n): regenerate desktop key catalog for main's telemetry/consent strings
* chore(i18n): regenerate desktop key catalog after main sync
---------
Co-authored-by: Teknium <[email protected]>
1336 lines
76 KiB
Python
1336 lines
76 KiB
Python
"""OpenAI-compatible routes for the API server adapter.
|
|
|
|
``OpenAICompatRoutesMixin`` (inherited by ``APIServerAdapter``) carries ``/v1/chat/completions``,
|
|
``/v1/responses`` (+ GET/DELETE), their SSE writers and the Responses-transcript helpers.
|
|
api_server-internal helpers are imported lazily inside each method: the origin imports this
|
|
module (top-level import = cycle), and lazy lookup keeps ``patch("...api_server.X")`` effective.
|
|
"""
|
|
|
|
import asyncio
|
|
import json
|
|
import logging
|
|
import re
|
|
import time
|
|
import uuid
|
|
from contextlib import suppress
|
|
from typing import Any, Dict, List, Optional
|
|
|
|
from agent.i18n import t
|
|
|
|
try:
|
|
from aiohttp import web
|
|
except ImportError: # pragma: no cover - mirrors api_server's optional import
|
|
web = None # type: ignore[assignment]
|
|
|
|
# Logger parity with the origin module (moved log records keep their name).
|
|
logger = logging.getLogger("gateway.platforms.api_server")
|
|
|
|
async def _iter_stream_items(stream_q, agent_task, response):
|
|
"""Yield agent stream items until EOS, writing SSE keepalives while idle.
|
|
|
|
Yields the ``None`` sentinel once so callers can run EOS-only work; when ``agent_task``
|
|
is already done the remaining queue is drained and the sentinel swallowed.
|
|
"""
|
|
from gateway.platforms.api_server import CHAT_COMPLETIONS_SSE_KEEPALIVE_SECONDS
|
|
|
|
last_activity = time.monotonic()
|
|
while True:
|
|
try:
|
|
item = await asyncio.wait_for(stream_q.get(), timeout=0.5)
|
|
except asyncio.TimeoutError:
|
|
if agent_task.done():
|
|
while True:
|
|
try:
|
|
item = stream_q.get_nowait()
|
|
except asyncio.QueueEmpty:
|
|
return
|
|
if item is None:
|
|
return
|
|
yield item
|
|
last_activity = time.monotonic()
|
|
if time.monotonic() - last_activity >= CHAT_COMPLETIONS_SSE_KEEPALIVE_SECONDS:
|
|
await response.write(b": keepalive\n\n")
|
|
last_activity = time.monotonic()
|
|
continue
|
|
if item is None:
|
|
yield None
|
|
return
|
|
yield item
|
|
last_activity = time.monotonic()
|
|
|
|
|
|
def _result_flags(result: Any) -> tuple:
|
|
"""``(completed, partial, failed, error)`` from an agent result dict (defaults if not a dict)."""
|
|
if not isinstance(result, dict):
|
|
return True, False, False, None
|
|
return (bool(result.get("completed", True)), bool(result.get("partial")),
|
|
bool(result.get("failed")), result.get("error"))
|
|
|
|
|
|
def _finish_reason(completed, is_partial, is_failed, err_msg, agent_error=None) -> str:
|
|
"""OpenAI ``finish_reason``: "length" for truncation, "error" for failure, else "stop"."""
|
|
# OpenAI uses "length" for truncation, "stop" for normal completion, and downstream SDKs accept "error"
|
|
# / custom codes. See issue #22496.
|
|
if is_partial and err_msg and "truncat" in err_msg.lower():
|
|
return "length"
|
|
if agent_error is not None or is_failed or (not completed and err_msg):
|
|
return "error"
|
|
return "stop"
|
|
|
|
|
|
def _hermes_extras(completed, is_partial, is_failed, err_msg, finish_reason: str) -> Dict[str, Any]:
|
|
return {
|
|
"completed": completed, "partial": is_partial, "failed": is_failed, "error": err_msg,
|
|
"error_code": "output_truncated" if finish_reason == "length" else "agent_error"}
|
|
|
|
|
|
def _transformed_notice() -> str:
|
|
return t("platform.api_server.transformed_notice")
|
|
|
|
|
|
def _post_stream_transform(result: Any) -> tuple:
|
|
"""``(text, appended)`` a stream still owes after ``transform_llm_output`` rewrote the final
|
|
(the deltas already carried the raw reply): the appended suffix, or the whole rewrite with
|
|
``appended=False`` when it is not a pure append. ``("", False)`` when nothing was transformed."""
|
|
if not isinstance(result, dict) or not result.get("response_transformed"):
|
|
return "", False
|
|
final = result.get("final_response") or ""
|
|
original = result.get("pre_transform_response") or ""
|
|
if original and final.startswith(original):
|
|
return final[len(original):], True
|
|
return final, False
|
|
|
|
|
|
def _message_item(text: Any) -> Dict[str, Any]:
|
|
"""Responses ``message`` output item carrying one ``output_text`` part."""
|
|
return {"type": "message", "role": "assistant",
|
|
"content": [{"type": "output_text", "text": text}]}
|
|
|
|
|
|
def _cap_text(text: str, keep: int) -> str:
|
|
"""Head of ``text`` plus a marker saying how much was cut (the Responses truncation rule)."""
|
|
return text[:keep] + "...[" + str(len(text) - keep) + " more chars]"
|
|
|
|
|
|
def _cap_history_tool_outputs(history: List[Dict[str, Any]], max_chars: int) -> List[Dict[str, Any]]:
|
|
"""Copy of ``history`` with tool outputs and string tool-call arguments longer than
|
|
``max_chars`` cut down. Only tool rows and ``tool_calls`` blobs change; user/assistant text
|
|
is left alone, and the agent's own transcript rows are never mutated (rows are copied).
|
|
Opt-in via gateway.api_server.history_tool_output_max_chars: a single stored snapshot
|
|
embeds the full cumulative history, so a few large tool outputs pushed one
|
|
response_store.db write to ~677 KB (#82513)."""
|
|
if max_chars <= 0:
|
|
return history
|
|
out: List[Dict[str, Any]] = []
|
|
for msg in history:
|
|
if not isinstance(msg, dict):
|
|
out.append(msg)
|
|
continue
|
|
content = msg.get("content")
|
|
if msg.get("role") == "tool" and isinstance(content, str) and len(content) > max_chars:
|
|
msg = {**msg, "content": _cap_text(content, max_chars)}
|
|
tool_calls = msg.get("tool_calls")
|
|
if msg.get("role") == "assistant" and isinstance(tool_calls, list):
|
|
capped_calls = []
|
|
for call in tool_calls:
|
|
fn = call.get("function") if isinstance(call, dict) else None
|
|
raw = fn.get("arguments") if isinstance(fn, dict) else None
|
|
if isinstance(raw, str) and len(raw) > max_chars:
|
|
try:
|
|
args = json.loads(raw)
|
|
except ValueError:
|
|
args = None
|
|
if isinstance(args, dict):
|
|
for k, v in args.items():
|
|
if isinstance(v, str) and len(v) > max_chars:
|
|
args[k] = _cap_text(v, max_chars)
|
|
call = {**call, "function": {**fn, "arguments": json.dumps(args)}}
|
|
capped_calls.append(call)
|
|
msg = {**msg, "tool_calls": capped_calls}
|
|
out.append(msg)
|
|
return out
|
|
|
|
|
|
def _reasoning_item(text: str) -> Dict[str, Any]:
|
|
"""Completed Responses ``reasoning`` output item (same shape the SSE writer closes with)."""
|
|
return {"id": f"rs_{uuid.uuid4().hex[:24]}", "type": "reasoning", "status": "completed",
|
|
"summary": [{"type": "summary_text", "text": text}]}
|
|
|
|
|
|
def _is_reasoning_input_item(item: Any) -> bool:
|
|
"""Echoed-back ``reasoning`` output item: Responses SDK clients replay a prior response's
|
|
``output`` list as the next ``input``. It carries no message content, so it must be
|
|
skipped rather than parsed into an empty ``user`` turn (#99552)."""
|
|
return isinstance(item, dict) and item.get("type") == "reasoning"
|
|
|
|
|
|
def _turn_reasoning_text(
|
|
conversation_history: List[Dict[str, Any]], user_message: Any, result: Dict[str, Any]) -> str:
|
|
"""Reasoning the model produced on this turn, joined for a non-streaming
|
|
``message.reasoning_content``. Read from the assistant messages the agent already
|
|
persisted (``build_assistant_message`` stores the structured reasoning under
|
|
``reasoning``) rather than re-accumulating callback deltas, so it is exactly what the
|
|
stream would have carried and cannot double-count the post-response fallback."""
|
|
messages = result.get("messages") if isinstance(result, dict) else None
|
|
if not isinstance(messages, list):
|
|
return ""
|
|
start = OpenAICompatRoutesMixin._response_messages_turn_start_index(
|
|
conversation_history, user_message, result)
|
|
parts = [m["reasoning"] for m in messages[start:]
|
|
if isinstance(m, dict) and m.get("role") == "assistant"
|
|
and isinstance(m.get("reasoning"), str) and m["reasoning"].strip()]
|
|
return "\n\n".join(parts)
|
|
|
|
|
|
def _trim_tool_items(items: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
|
|
"""Trim large tool payloads in place so response.completed stays under ~100KB (clients
|
|
already received the full details via the incremental events)."""
|
|
for item in items:
|
|
if item.get("type") == "function_call":
|
|
try:
|
|
raw = item.get("arguments")
|
|
args = json.loads(raw) if isinstance(raw, str) else item.get("arguments", {})
|
|
if isinstance(args, dict):
|
|
for k in ("content", "query", "pattern", "old_string", "new_string"):
|
|
if isinstance(args.get(k), str) and len(args[k]) > 500:
|
|
args[k] = f"[{len(args[k])} chars — truncated for response.completed]"
|
|
item["arguments"] = json.dumps(args)
|
|
except Exception:
|
|
pass
|
|
elif item.get("type") == "function_call_output":
|
|
output = item.get("output", [])
|
|
if isinstance(output, list) and output:
|
|
first = output[0]
|
|
if isinstance(first, dict) and first.get("type") == "input_text":
|
|
text = first.get("text", "")
|
|
if len(text) > 1000:
|
|
first["text"] = _cap_text(text, 500)
|
|
item["output"] = [first]
|
|
return items
|
|
|
|
|
|
class _ResponsesStream:
|
|
"""Per-request state and event emitters for the POST /v1/responses SSE writer.
|
|
|
|
Every event carries a monotonic ``sequence_number`` (canonical Responses SSE schema). Text
|
|
deltas are batched (50ms) against Open WebUI re-render storms; tool events flush first.
|
|
"""
|
|
|
|
def __init__(self, adapter, response, *, response_id: str, model: str, created_at: int,
|
|
conversation_history: List[Dict[str, str]], user_message: str,
|
|
instructions: Optional[str], conversation: Optional[str], store: bool, session_id: str):
|
|
from gateway.platforms import api_server as api
|
|
self._api = api
|
|
self.adapter, self.response, self.response_id = adapter, response, response_id
|
|
self.model, self.created_at, self.conversation_history = model, created_at, conversation_history
|
|
self.user_message, self.instructions = user_message, instructions
|
|
self.conversation, self.store, self.session_id = conversation, store, session_id
|
|
# Resolved in the request's profile scope: a snapshot written after it (disconnect) must not follow another.
|
|
self.response_store = adapter._current_response_store()
|
|
self.final_text_parts: List[str] = []
|
|
self.pending_tool_calls: List[Dict[str, Any]] = [] # open function_call items, in order
|
|
self.emitted_items: List[Dict[str, Any]] = [] # output items so far (terminal payload)
|
|
self.output_index = 0
|
|
self.call_counter = 0 # call_id fallback when the agent supplies no tool_call_id
|
|
self.sequence_number = 0
|
|
self.message_item_id = f"msg_{uuid.uuid4().hex[:24]}"
|
|
self.message_output_index: Optional[int] = None
|
|
self.message_opened = False
|
|
self.reasoning_item: Optional[Dict[str, Any]] = None # open ``reasoning`` output item
|
|
self.final_response_text = ""
|
|
self.transformed_final = "" # non-append transform_llm_output rewrite; replaces the deltas
|
|
self.agent_error: Optional[str] = None
|
|
self.usage: Dict[str, int] = {"input_tokens": 0, "output_tokens": 0, "total_tokens": 0}
|
|
self.terminal_snapshot_persisted = False
|
|
self.result: Any = None
|
|
self._batch_buf: List[str] = []
|
|
self._batch_timer: Optional[asyncio.Task] = None
|
|
self._batch_lock = asyncio.Lock()
|
|
|
|
async def write_event(self, event_type: str, data: Dict[str, Any]) -> None:
|
|
if "sequence_number" not in data:
|
|
data["sequence_number"] = self.sequence_number
|
|
self.sequence_number += 1
|
|
await self.response.write(self._api._sse_frame(data, event=event_type))
|
|
|
|
def envelope(self, status: str) -> Dict[str, Any]:
|
|
return {"id": self.response_id, "object": "response", "status": status,
|
|
"created_at": self.created_at, "model": self.model}
|
|
|
|
def terminal_envelope(self, status: str, output: List[Dict[str, Any]], *, error=None) -> dict:
|
|
"""``envelope`` + ``output`` (+ ``error`` when given) + ``usage``, in wire key order."""
|
|
env = self.envelope(status)
|
|
env["output"] = output
|
|
if error is not None:
|
|
env["error"] = {"message": error, "type": "server_error"}
|
|
env["usage"] = self._api._responses_usage_payload(self.usage)
|
|
return env
|
|
|
|
def _history_with_user(self) -> List[Dict[str, Any]]:
|
|
return list(self.conversation_history) + [{"role": "user", "content": self.user_message}]
|
|
|
|
def persist_snapshot(self, response_env: Dict[str, Any], *, history=None, session_id=None):
|
|
if not self.store:
|
|
return
|
|
self.response_store.put(self.response_id, {
|
|
"response": response_env,
|
|
"conversation_history": self._history_with_user() if history is None else history,
|
|
"instructions": self.instructions,
|
|
"session_id": session_id or self.session_id})
|
|
if self.conversation:
|
|
self.response_store.set_conversation(self.conversation, self.response_id)
|
|
|
|
def persist_incomplete_if_needed(self) -> None:
|
|
"""Persist an ``incomplete`` snapshot when no terminal one was written (disconnect /
|
|
cancel paths), so GET /v1/responses/{id} and ``previous_response_id`` chaining survive."""
|
|
if not self.store or self.terminal_snapshot_persisted:
|
|
return
|
|
text = "".join(self.final_text_parts) or self.final_response_text
|
|
items = list(self.emitted_items)
|
|
history = self._history_with_user()
|
|
if text:
|
|
items.append(_message_item(text))
|
|
history.append({"role": "assistant", "content": text})
|
|
self.persist_snapshot(self.terminal_envelope("incomplete", items), history=history)
|
|
|
|
async def emit_created(self) -> None:
|
|
env = self.envelope("in_progress")
|
|
env["output"] = []
|
|
await self.write_event("response.created", {"type": "response.created", "response": env})
|
|
self.persist_snapshot(env)
|
|
|
|
async def _open_message_item(self) -> None:
|
|
"""Emit output_item.added for the assistant message on the first text delta."""
|
|
if self.message_opened:
|
|
return
|
|
self.message_opened = True
|
|
self.message_output_index = self.output_index
|
|
self.output_index += 1
|
|
await self.write_event("response.output_item.added", {
|
|
"type": "response.output_item.added", "output_index": self.message_output_index,
|
|
"item": {"id": self.message_item_id, "type": "message", "status": "in_progress",
|
|
"role": "assistant", "content": []}})
|
|
|
|
async def emit_text_delta(self, delta_text: str) -> None:
|
|
await self.close_reasoning_item()
|
|
await self._open_message_item()
|
|
self.final_text_parts.append(delta_text)
|
|
await self.write_event("response.output_text.delta", {
|
|
"type": "response.output_text.delta", "item_id": self.message_item_id,
|
|
"output_index": self.message_output_index, "content_index": 0, "delta": delta_text,
|
|
"logprobs": []})
|
|
|
|
async def emit_reasoning_delta(self, delta_text: str) -> None:
|
|
"""Responses reasoning-summary family (#99552): one ``reasoning`` output item per
|
|
thinking burst, closed before the next message/tool item opens."""
|
|
if self.reasoning_item is None:
|
|
item = {"id": f"rs_{uuid.uuid4().hex[:24]}", "type": "reasoning", "status": "in_progress",
|
|
"summary": []}
|
|
self.reasoning_item = {"item": item, "output_index": self.output_index, "parts": []}
|
|
self.output_index += 1
|
|
await self.write_event("response.output_item.added", {
|
|
"type": "response.output_item.added",
|
|
"output_index": self.reasoning_item["output_index"], "item": item})
|
|
await self.write_event("response.reasoning_summary_part.added", {
|
|
"type": "response.reasoning_summary_part.added", "item_id": item["id"],
|
|
"output_index": self.reasoning_item["output_index"], "summary_index": 0,
|
|
"part": {"type": "summary_text", "text": ""}})
|
|
rs = self.reasoning_item
|
|
rs["parts"].append(delta_text)
|
|
await self.write_event("response.reasoning_summary_text.delta", {
|
|
"type": "response.reasoning_summary_text.delta", "item_id": rs["item"]["id"],
|
|
"output_index": rs["output_index"], "summary_index": 0, "delta": delta_text})
|
|
|
|
async def close_reasoning_item(self) -> None:
|
|
rs, self.reasoning_item = self.reasoning_item, None
|
|
if rs is None:
|
|
return
|
|
text = "".join(rs["parts"])
|
|
item = dict(rs["item"], status="completed", summary=[{"type": "summary_text", "text": text}])
|
|
base = {"item_id": item["id"], "output_index": rs["output_index"], "summary_index": 0}
|
|
await self.write_event("response.reasoning_summary_text.done", {
|
|
"type": "response.reasoning_summary_text.done", **base, "text": text})
|
|
await self.write_event("response.reasoning_summary_part.done", {
|
|
"type": "response.reasoning_summary_part.done", **base,
|
|
"part": {"type": "summary_text", "text": text}})
|
|
self.emitted_items.append(item)
|
|
await self.write_event("response.output_item.done", {
|
|
"type": "response.output_item.done", "output_index": rs["output_index"], "item": item})
|
|
|
|
async def emit_commentary(self, text: str) -> None:
|
|
"""Mid-turn assistant commentary as its own completed ``message`` item carrying
|
|
``"phase": "commentary"`` — never appended to ``final_text_parts``, so the final answer
|
|
item stays clean (#67580). Closes any open reasoning item first so a reasoning item
|
|
never straddles a message item."""
|
|
await self.close_reasoning_item()
|
|
item = {"id": f"msg_{uuid.uuid4().hex[:24]}", "status": "completed", "phase": "commentary",
|
|
**_message_item(text)}
|
|
idx = self.output_index
|
|
self.output_index += 1
|
|
self.emitted_items.append({"phase": "commentary", **_message_item(text)})
|
|
for event in ("response.output_item.added", "response.output_item.done"):
|
|
await self.write_event(event, {"type": event, "output_index": idx, "item": item})
|
|
|
|
async def emit_tool_started(self, payload: Dict[str, Any]) -> None:
|
|
"""function_call ``output_item.added``; the agent's tool_call_id beats a generated call id."""
|
|
await self.close_reasoning_item()
|
|
self.call_counter += 1
|
|
call_id = payload.get("tool_call_id") or f"call_{self.response_id[5:]}_{self.call_counter}"
|
|
args = payload.get("arguments", {})
|
|
arguments_str = json.dumps(args) if isinstance(args, dict) else str(args)
|
|
name = payload.get("name", "")
|
|
item = {"id": f"fc_{uuid.uuid4().hex[:24]}", "type": "function_call",
|
|
"status": "in_progress", "name": name, "call_id": call_id, "arguments": arguments_str}
|
|
idx = self.output_index
|
|
self.output_index += 1
|
|
self.pending_tool_calls.append({
|
|
"call_id": call_id, "name": name, "arguments": arguments_str, "item_id": item["id"],
|
|
"output_index": idx})
|
|
self.emitted_items.append(
|
|
{"type": "function_call", "name": name, "arguments": arguments_str, "call_id": call_id})
|
|
await self.write_event("response.output_item.added", {
|
|
"type": "response.output_item.added", "output_index": idx, "item": item})
|
|
|
|
async def emit_tool_completed(self, payload: Dict[str, Any]) -> None:
|
|
"""function_call ``output_item.done`` + function_call_output added/done; orphans skipped."""
|
|
call_id = payload.get("tool_call_id")
|
|
pending = next((p for p in self.pending_tool_calls if p["call_id"] == call_id), None)
|
|
if not call_id or pending is None:
|
|
return
|
|
self.pending_tool_calls.remove(pending)
|
|
done_item = {"id": pending["item_id"], "type": "function_call", "status": "completed",
|
|
"name": pending["name"], "call_id": pending["call_id"],
|
|
"arguments": pending["arguments"]}
|
|
await self.write_event("response.output_item.done", {
|
|
"type": "response.output_item.done", "output_index": pending["output_index"],
|
|
"item": done_item})
|
|
result = payload.get("result", "")
|
|
result_str = result if isinstance(result, str) else json.dumps(result)
|
|
output_parts = [{"type": "input_text", "text": result_str}]
|
|
output_item = {"id": f"fco_{uuid.uuid4().hex[:24]}", "type": "function_call_output",
|
|
"call_id": pending["call_id"], "output": output_parts, "status": "completed"}
|
|
idx = self.output_index
|
|
self.output_index += 1
|
|
self.emitted_items.append(
|
|
{"type": "function_call_output", "call_id": pending["call_id"], "output": output_parts})
|
|
for event in ("response.output_item.added", "response.output_item.done"):
|
|
await self.write_event(event, {"type": event, "output_index": idx, "item": output_item})
|
|
|
|
async def emit_status(self, payload: Dict[str, Any]) -> None:
|
|
"""Lifecycle/warning status (provider wait, auto-recovery countdown, fallback switch) as a
|
|
``hermes.status`` custom event; not a Responses output item."""
|
|
await self.response.write(self._api._sse_frame(payload, event="hermes.status"))
|
|
|
|
# queue tag -> (method name, payload adapter)
|
|
_TAG_HANDLERS = {
|
|
"__tool_started__": ("emit_tool_started", lambda p: p),
|
|
"__tool_completed__": ("emit_tool_completed", lambda p: p),
|
|
"__commentary__": ("emit_commentary", lambda p: p["text"]),
|
|
"__reasoning__": ("emit_reasoning_delta", lambda p: p),
|
|
"__status__": ("emit_status", lambda p: p),
|
|
}
|
|
|
|
async def dispatch(self, item: Any) -> None:
|
|
"""Route one queue item: tagged tuples emit immediately, strings are batched, others dropped."""
|
|
if isinstance(item, tuple) and len(item) == 2 and isinstance(item[0], str):
|
|
tag, payload = item
|
|
await self.flush_batch()
|
|
handler = self._TAG_HANDLERS.get(tag)
|
|
if handler is not None:
|
|
method, adapt = handler
|
|
await getattr(self, method)(adapt(payload))
|
|
elif isinstance(item, str):
|
|
self._batch_buf.append(item)
|
|
if self._batch_timer is None:
|
|
self._batch_timer = asyncio.create_task(self._batch_flush_after(0.05))
|
|
|
|
async def _batch_flush_after(self, delay: float) -> None:
|
|
try:
|
|
await asyncio.sleep(delay)
|
|
except asyncio.CancelledError:
|
|
return
|
|
# Clear the timer BEFORE flushing so new deltas can start a fresh timer while we emit.
|
|
self._batch_timer = None
|
|
await self.flush_batch()
|
|
|
|
def cancel_batch_timer(self) -> None:
|
|
if self._batch_timer and not self._batch_timer.done():
|
|
self._batch_timer.cancel()
|
|
self._batch_timer = None
|
|
|
|
async def flush_batch(self) -> None:
|
|
"""Emit a single delta for all buffered text."""
|
|
if not self._batch_buf:
|
|
return
|
|
async with self._batch_lock:
|
|
if self._batch_buf:
|
|
combined = "".join(self._batch_buf)
|
|
self._batch_buf = []
|
|
await self.emit_text_delta(combined)
|
|
|
|
async def collect_result(self, agent_task) -> None:
|
|
"""Await the agent; when it produced a final_response but streamed no deltas
|
|
(some providers only emit the full text at the end), emit one fallback delta."""
|
|
try:
|
|
result, agent_usage = await agent_task
|
|
self.result = result
|
|
self.usage = agent_usage or self.usage
|
|
agent_final = result.get("final_response", "") if isinstance(result, dict) else ""
|
|
tail, appended = _post_stream_transform(result)
|
|
if tail and self.final_text_parts:
|
|
if appended:
|
|
await self.emit_text_delta(tail)
|
|
else:
|
|
self.transformed_final = agent_final
|
|
if agent_final and not self.final_text_parts:
|
|
await self.emit_text_delta(agent_final)
|
|
if agent_final and not self.final_response_text:
|
|
self.final_response_text = agent_final
|
|
if isinstance(result, dict) and result.get("error") and not self.final_response_text:
|
|
self.agent_error = self._api._redact_api_error_text(result["error"])
|
|
except Exception as e: # noqa: BLE001
|
|
logger.error("Error running agent for streaming responses: %s", e, exc_info=True)
|
|
self.agent_error = self._api._redact_api_error_text(e)
|
|
|
|
async def close_message_item(self) -> None:
|
|
await self.close_reasoning_item()
|
|
self.final_response_text = (
|
|
self.transformed_final or "".join(self.final_text_parts) or self.final_response_text)
|
|
if not self.message_opened:
|
|
return
|
|
await self.write_event("response.output_text.done", {
|
|
"type": "response.output_text.done", "item_id": self.message_item_id,
|
|
"output_index": self.message_output_index, "content_index": 0,
|
|
"text": self.final_response_text, "logprobs": []})
|
|
await self.write_event("response.output_item.done", {
|
|
"type": "response.output_item.done", "output_index": self.message_output_index,
|
|
"item": {"id": self.message_item_id, "type": "message", "status": "completed",
|
|
"role": "assistant",
|
|
"content": [{"type": "output_text", "text": self.final_response_text}]}})
|
|
|
|
def _final_items(self) -> List[Dict[str, Any]]:
|
|
"""Emitted items (trimmed) plus a final message item, so clients that only parse
|
|
the terminal payload still see the assistant text (mirrors _extract_output_items)."""
|
|
items = _trim_tool_items(list(self.emitted_items))
|
|
redact = self._api._redact_api_error_text
|
|
text = self.final_response_text or (redact(self.agent_error) if self.agent_error else "")
|
|
items.append(_message_item(text))
|
|
return items
|
|
|
|
async def emit_failed(self) -> None:
|
|
redact = self._api._redact_api_error_text
|
|
env = self.terminal_envelope("failed", self._final_items(), error=redact(self.agent_error))
|
|
history = self._history_with_user()
|
|
history.append(
|
|
{"role": "assistant", "content": self.final_response_text or redact(self.agent_error)})
|
|
self.persist_snapshot(env, history=history)
|
|
self.terminal_snapshot_persisted = True
|
|
await self.write_event("response.failed", {"type": "response.failed", "response": env})
|
|
|
|
async def emit_completed(self) -> None:
|
|
env = self.terminal_envelope("completed", self._final_items())
|
|
result = self.result
|
|
full_history = self.adapter._build_response_conversation_history(
|
|
self.conversation_history, self.user_message, result, self.final_response_text,
|
|
tool_output_max_chars=self.adapter._history_tool_output_max_chars)
|
|
# Transcript substitution for result["_compressed"] happens in the history builder; only
|
|
# a compression-rotated session_id is propagated so chaining resumes the child session.
|
|
sid = result.get("session_id") if isinstance(result, dict) else None
|
|
self.persist_snapshot(
|
|
env, history=full_history, session_id=sid if isinstance(sid, str) and sid else None)
|
|
self.terminal_snapshot_persisted = True
|
|
await self.write_event(
|
|
"response.completed", {"type": "response.completed", "response": env})
|
|
|
|
async def emit_crash(self, exc: BaseException) -> None:
|
|
error = self._api._redact_api_error_text(exc, limit=500)
|
|
env = self.terminal_envelope("failed", list(self.emitted_items), error=error)
|
|
await self.write_event("response.failed", {"type": "response.failed", "response": env})
|
|
|
|
|
|
class OpenAICompatRoutesMixin:
|
|
"""/v1/chat/completions and /v1/responses handlers + SSE writers."""
|
|
|
|
def _select_request_route(
|
|
self, body: Dict[str, Any], *, session_id, gateway_session_key, model_alias) -> tuple:
|
|
"""Resolve the model_routes alias + per-request overrides ->
|
|
``(route, agent_overrides, error_response_or_None)``."""
|
|
from gateway.platforms.api_server import _error_response, _request_agent_overrides
|
|
route = self._resolve_route(model_alias)
|
|
overrides = _request_agent_overrides(
|
|
body, virtual_model=self._model_name, allow_bare_model=self._direct_model_requests)
|
|
err = self._request_route_conflict_error(
|
|
session_id=session_id, gateway_session_key=gateway_session_key,
|
|
requested_model=overrides.get("requested_model"),
|
|
requested_provider=overrides.get("requested_provider"), route=route)
|
|
return route, overrides, (_error_response(err, 400) if err else None)
|
|
|
|
def _register_stream_approval(self, request, completion_id, stream_q, session_id) -> tuple:
|
|
"""Expose a streaming completion as a run for ``POST /v1/runs/{completion_id}/approval``
|
|
(#51871) -> ``(notify, on_done)``. Owner + status are stamped like the session stream, so
|
|
``_load_owned_run`` finds it; keyed by the completion id (never the shared session key) so
|
|
concurrent turns can't cross-resolve. ``on_done`` retires it when the agent task ends."""
|
|
from gateway.platforms.api_server import _approval_request_event
|
|
self._run_owners[completion_id] = self._run_idempotency_scope(request)
|
|
self._set_run_status(completion_id, "running", session_id=session_id or "")
|
|
self._run_approval_sessions[completion_id] = completion_id
|
|
|
|
def _approval_notify(approval_data):
|
|
event = _approval_request_event(completion_id, approval_data, session_id=session_id or "")
|
|
self._set_run_status(completion_id, "waiting_for_approval", last_event="approval.request",
|
|
approval=event)
|
|
stream_q.put_threadsafe(("__approval__", event))
|
|
|
|
def _on_done(fut):
|
|
from gateway.platforms.api_server_runs import terminal_run_status
|
|
self._run_approval_sessions.pop(completion_id, None)
|
|
if fut.cancelled():
|
|
status = "cancelled"
|
|
elif fut.exception() is not None:
|
|
status = "failed"
|
|
else:
|
|
result = (fut.result() or (None,))[0]
|
|
status = terminal_run_status(result)[0] if isinstance(result, dict) else "completed"
|
|
self._set_run_status(completion_id, status)
|
|
self._release_run_owner_if_forgotten(completion_id)
|
|
return _approval_notify, _on_done
|
|
|
|
def _spawn_stream_agent(self, stream_q, *, on_done=None, **run_kwargs) -> tuple:
|
|
"""Start ``_run_agent`` for an SSE writer -> ``(agent_task, agent_ref)``. ``agent_ref[0]``
|
|
lets the writer interrupt on disconnect; the EOS sentinel is enqueued from the task's done
|
|
callback so drain loops never race a polled ``agent_task.done()``. ``on_done(task)`` runs
|
|
in that same callback, before EOS."""
|
|
def _on_delta(delta):
|
|
# None from the agent is a CLI box-close signal, not EOS — forwarding it would end
|
|
# the stream early. Called from the run_conversation worker thread: put_threadsafe.
|
|
if delta is not None:
|
|
stream_q.put_threadsafe(delta)
|
|
def _on_reasoning(text):
|
|
# Structured reasoning deltas (#99552): the agent's reasoning_callback, not the
|
|
# lossy 500-char ``reasoning.available`` progress preview. Tagged so the writers
|
|
# keep them distinct from answer text.
|
|
if text:
|
|
stream_q.put_threadsafe(("__reasoning__", text))
|
|
def _on_status(kind, message=None):
|
|
# Lifecycle/warning status (provider wait, auto-recovery countdown, fallback switch) as a
|
|
# ``hermes.status`` event, so a client sees why the stream is silent instead of a dead socket.
|
|
from gateway.platforms.api_server import _redact_api_error_text
|
|
text = _redact_api_error_text(message if message is not None else kind or "").strip()
|
|
if text:
|
|
stream_q.put_threadsafe(("__status__", {"kind": str(kind), "text": text}))
|
|
agent_ref = [None]
|
|
agent_task = asyncio.ensure_future(self._run_agent(
|
|
stream_delta_callback=_on_delta, reasoning_callback=_on_reasoning, status_callback=_on_status,
|
|
agent_ref=agent_ref, **run_kwargs))
|
|
def _done(fut):
|
|
try:
|
|
if on_done is not None:
|
|
on_done(fut)
|
|
except Exception:
|
|
logger.debug("[api_server] stream on_done hook failed", exc_info=True)
|
|
finally:
|
|
stream_q.put_nowait(None)
|
|
agent_task.add_done_callback(_done)
|
|
return agent_task, agent_ref
|
|
|
|
async def _handle_chat_completions(self, request: "web.Request") -> "web.Response":
|
|
"""POST /v1/chat/completions — OpenAI Chat Completions format."""
|
|
from gateway.platforms.api_server import (
|
|
ThreadSafeAsyncQueue, _api_request_profile, _chat_usage_payload, _coerce_request_bool,
|
|
_content_has_visible_payload, _derive_chat_session_id, _error_response, _invalid_request,
|
|
_multimodal_validation_error, _normalize_chat_content, _normalize_multimodal_content,
|
|
_openai_error, _redact_api_error_text, _resolve_media_to_data_urls)
|
|
# Bound total in-flight agent runs (configurable; #7483).
|
|
limited = self._concurrency_limited_response()
|
|
if limited is not None:
|
|
return limited
|
|
try:
|
|
body = await request.json()
|
|
except Exception:
|
|
return _error_response("Invalid JSON in request body", 400)
|
|
from gateway.platforms.api_server import _request_relay_metadata
|
|
relay_metadata = _request_relay_metadata(body)
|
|
messages = body.get("messages")
|
|
if not messages or not isinstance(messages, list):
|
|
return _invalid_request("Missing or invalid 'messages' field")
|
|
stream = _coerce_request_bool(body.get("stream"), default=False)
|
|
|
|
# System messages -> ephemeral system prompt layered ON TOP of core, flattened to text
|
|
# (Anthropic rejects images there, OpenAI text models ignore them).
|
|
system_prompt = None
|
|
conversation_messages: List[Dict[str, str]] = []
|
|
for idx, msg in enumerate(messages):
|
|
role = msg.get("role", "")
|
|
raw_content = msg.get("content", "")
|
|
if role == "system":
|
|
content = _normalize_chat_content(raw_content)
|
|
system_prompt = content if system_prompt is None else system_prompt + "\n" + content
|
|
elif role in {"user", "assistant"}:
|
|
try:
|
|
content = _normalize_multimodal_content(raw_content)
|
|
except ValueError as exc:
|
|
return _multimodal_validation_error(exc, param=f"messages[{idx}].content")
|
|
conversation_messages.append({"role": role, "content": content})
|
|
user_message: Any = (conversation_messages[-1].get("content", "") if conversation_messages else "")
|
|
history = conversation_messages[:-1]
|
|
if not _content_has_visible_payload(user_message):
|
|
return _invalid_request("No user message found in messages")
|
|
|
|
# X-Hermes-Session-Key scopes long-term memory per channel; independent of
|
|
# X-Hermes-Session-Id (the key persists across transcripts, the id rotates on /new).
|
|
gateway_session_key, key_err = self._parse_session_key_header(request)
|
|
if key_err is not None:
|
|
return key_err
|
|
# X-Hermes-Session-Id continues an existing session (history from state.db, not the body);
|
|
# requires a configured API key or any client could read history by guessing ids.
|
|
provided_session_id = request.headers.get("X-Hermes-Session-Id", "").strip()
|
|
if provided_session_id:
|
|
if not self._api_key:
|
|
logger.warning(
|
|
"Session continuation via X-Hermes-Session-Id rejected: "
|
|
"no API key configured. Set API_SERVER_KEY to enable "
|
|
"session continuity.")
|
|
return _error_response("Session continuation requires API key authentication. "
|
|
"Configure API_SERVER_KEY to enable this feature.", 403)
|
|
# Same guard as the native gateway: ids are interpolated into on-disk filenames.
|
|
from gateway.session import _is_path_unsafe
|
|
if re.search(r'[\r\n\x00]', provided_session_id) or _is_path_unsafe(provided_session_id):
|
|
return _invalid_request("Invalid session ID")
|
|
if len(provided_session_id) > self._MAX_SESSION_HEADER_LEN:
|
|
return _invalid_request("Session ID too long")
|
|
session_id = provided_session_id
|
|
try:
|
|
db = await self._ensure_session_db_async()
|
|
if db is not None:
|
|
# #98619/#13437: a client-addressed id from before a compression rotation
|
|
# must adopt the live continuation tip — history loads from it, the turn and
|
|
# the wake target bind it, and a detached delegation delivery row persisted
|
|
# on the tip is what this continuation consumes. Same canonical resolution
|
|
# the delivery writer (gateway/wake.py) and /v1/runs use; fails open.
|
|
from gateway.platforms.api_server_runs import _resolve_live_session_id
|
|
session_id = await _resolve_live_session_id(self, provided_session_id)
|
|
history = await asyncio.to_thread(db.get_messages_as_conversation, session_id)
|
|
except Exception as e:
|
|
logger.warning("Failed to load session history for %s: %s", session_id, e)
|
|
history = []
|
|
else:
|
|
# Stable id from the conversation fingerprint so Open WebUI-style clients map onto
|
|
# one Hermes session; namespaced by the routed profile (#123989).
|
|
first_user = next(
|
|
(cm.get("content", "") for cm in conversation_messages if cm.get("role") == "user"), "")
|
|
session_id = _derive_chat_session_id(system_prompt, first_user, _api_request_profile.get())
|
|
completion_id = f"chatcmpl-{uuid.uuid4().hex[:29]}"
|
|
model_name = body.get("model", self._model_name)
|
|
created = int(time.time())
|
|
route, agent_overrides, selection_error = self._select_request_route(
|
|
body, session_id=session_id, gateway_session_key=gateway_session_key,
|
|
model_alias=model_name)
|
|
if selection_error is not None:
|
|
return selection_error
|
|
run_kwargs = dict(
|
|
user_message=user_message, conversation_history=history,
|
|
ephemeral_system_prompt=system_prompt, session_id=session_id,
|
|
gateway_session_key=gateway_session_key, **agent_overrides, route=route,
|
|
relay_metadata=relay_metadata,
|
|
# #98619: only an explicitly provided X-Hermes-Session-Id is wake-capable (the
|
|
# header is 403-gated on API_SERVER_KEY, so the wake self-post can authenticate
|
|
# and the client can resume the session by sending it again). A fingerprint-derived
|
|
# id from a header-less client is NOT: delegate_task keeps its forced-sync fallback
|
|
# there — the wake would hard-fail or land in history that client never reloads.
|
|
session_history_delivery=("1" if provided_session_id else ""))
|
|
# This is presentation only. The ordinary API-key/session authorization
|
|
# above still applies; it grants no internal ingress or control authority.
|
|
if provided_session_id and body.get("hermes_notification_category") == "diagnostic":
|
|
run_kwargs["notification_category"] = "diagnostic"
|
|
if stream:
|
|
_stream_q = ThreadSafeAsyncQueue()
|
|
# tool_call_ids with an emitted "running": a "completed" without one (internal/
|
|
# filtered tools) is dropped rather than orphaned on the wire.
|
|
_started_tool_call_ids: set[str] = set()
|
|
|
|
def _on_tool_start(tool_call_id, function_name, function_args):
|
|
"""``hermes.tool.progress`` status=running; ``_``-prefixed tools stay off the wire."""
|
|
if not tool_call_id or function_name.startswith("_"):
|
|
return
|
|
_started_tool_call_ids.add(tool_call_id)
|
|
from agent.display import build_tool_preview, get_tool_emoji
|
|
label = build_tool_preview(function_name, function_args) or function_name
|
|
_stream_q.put_threadsafe(("__tool_progress__", {
|
|
"tool": function_name, "emoji": get_tool_emoji(function_name), "label": label,
|
|
"toolCallId": tool_call_id, "status": "running"}))
|
|
|
|
def _on_tool_complete(tool_call_id, function_name, function_args, function_result):
|
|
if not tool_call_id or tool_call_id not in _started_tool_call_ids:
|
|
return
|
|
_started_tool_call_ids.discard(tool_call_id)
|
|
_stream_q.put_threadsafe(("__tool_progress__", {
|
|
"tool": function_name, "toolCallId": tool_call_id, "status": "completed"}))
|
|
|
|
# tool_progress_callback deliberately NOT wired: it would duplicate the structured
|
|
# start/complete callbacks (which carry the tool_call id).
|
|
# The completion id doubles as the run id; on_done drops the approval mapping once the
|
|
# turn ends (POST /v1/runs/{id}/approval then answers 409) and sets the terminal status.
|
|
approval_notify, end_stream_run = self._register_stream_approval(
|
|
request, completion_id, _stream_q, session_id)
|
|
agent_task, agent_ref = self._spawn_stream_agent(
|
|
_stream_q, on_done=end_stream_run, tool_start_callback=_on_tool_start,
|
|
tool_complete_callback=_on_tool_complete, approval_notify_callback=approval_notify,
|
|
approval_session_key=completion_id, **run_kwargs)
|
|
# #13437 identity contract: an explicit-header client keeps addressing the id it
|
|
# sent; the response echoes that stable id while reads/writes adopt the live tip,
|
|
# so a rotation mid-turn (after these headers are prepared) never changes what the
|
|
# client should send next — it re-sends the same id and the tip resolution above
|
|
# finds whatever session is live by then.
|
|
return await self._write_sse_chat_completion(
|
|
request, completion_id, model_name, created, _stream_q,
|
|
agent_task, agent_ref, session_id=(provided_session_id or session_id),
|
|
gateway_session_key=gateway_session_key)
|
|
|
|
async def _compute_completion():
|
|
return await self._run_agent(**run_kwargs)
|
|
outcome, err = await self._run_idempotent(
|
|
request, body, _compute_completion, log_label="chat completions",
|
|
fingerprint_keys=["model", "provider", "model_options", "messages", "tools", "tool_choice", "stream",
|
|
"hermes_notification_category"],
|
|
route="chat_completions",
|
|
)
|
|
if err is not None:
|
|
return err
|
|
result, usage = outcome
|
|
presentation_muted = result.get("_notification_presentation_suppressed") is True
|
|
final_response = _resolve_media_to_data_urls(result.get("final_response") or "")
|
|
completed, is_partial, is_failed, err_msg = _result_flags(result)
|
|
if err_msg:
|
|
err_msg = _redact_api_error_text(err_msg)
|
|
finish_reason = _finish_reason(completed, is_partial, is_failed, err_msg)
|
|
# Same #13437 identity contract as the SSE path: an explicit-header client is echoed
|
|
# the stable id it sent; a fingerprint-derived (header-less) turn keeps reporting the
|
|
# id the agent actually resolved, so headerless clients still learn where the turn went.
|
|
response_headers = {"X-Hermes-Session-Id": (provided_session_id or result.get("session_id", session_id))}
|
|
if gateway_session_key:
|
|
response_headers["X-Hermes-Session-Key"] = gateway_session_key
|
|
# Hard fail (no usable text AND a real failure) -> 502 OpenAI error envelope so SDK
|
|
# clients raise instead of rendering the failure string as message.content.
|
|
if not final_response and (is_failed or is_partial):
|
|
err_body = _openai_error(
|
|
"" if presentation_muted else (err_msg or "Agent run did not produce a response."), err_type="server_error",
|
|
code="agent_incomplete")
|
|
err_body["error"]["hermes"] = {
|
|
"completed": completed, "partial": is_partial, "failed": is_failed}
|
|
response_headers["X-Hermes-Completed"] = "false"
|
|
response_headers["X-Hermes-Partial"] = "true" if is_partial else "false"
|
|
return web.json_response(err_body, status=502, headers=response_headers)
|
|
# Soft partial (some text, run incomplete): 200 + finish_reason="length"/Hermes extras.
|
|
response_data = {
|
|
"id": completion_id, "object": "chat.completion", "created": created,
|
|
"model": model_name,
|
|
"choices": [{"index": 0, "message": {"role": "assistant", "content": "" if presentation_muted else final_response},
|
|
"finish_reason": finish_reason}],
|
|
"usage": _chat_usage_payload(usage)}
|
|
# Non-streaming twin of ``delta.reasoning_content`` (#99552).
|
|
reasoning_text = _turn_reasoning_text(history, user_message, result)
|
|
if reasoning_text and not presentation_muted:
|
|
response_data["choices"][0]["message"]["reasoning_content"] = reasoning_text
|
|
if is_partial or is_failed or not completed:
|
|
response_data["hermes"] = _hermes_extras(
|
|
completed, is_partial, is_failed, "" if presentation_muted else err_msg, finish_reason)
|
|
response_headers["X-Hermes-Completed"] = "false"
|
|
response_headers["X-Hermes-Partial"] = "true" if is_partial else "false"
|
|
if err_msg and not presentation_muted:
|
|
response_headers["X-Hermes-Error"] = _redact_api_error_text(err_msg, limit=200)
|
|
return web.json_response(response_data, headers=response_headers)
|
|
|
|
async def _run_idempotent(
|
|
self, request: "web.Request", body: Dict[str, Any], compute, *,
|
|
log_label: str, fingerprint_keys: List[str], route: str) -> tuple:
|
|
"""Run ``compute()`` once per (principal scope, logical route, Idempotency-Key) + body fingerprint
|
|
-> ``((result, usage), None)`` or ``(None, 500 response)``.
|
|
|
|
``_idem_cache`` is process-global: under ``gateway.multiplex_profiles`` every profile's
|
|
``/p/<profile>/v1/...`` mirror shares it, so the key carries ``_run_idempotency_scope`` (the same
|
|
``sha256(profile, expected API key)`` namespace the durable ``/v1/runs`` API uses) — a client key
|
|
colliding across profiles, or a rotated API_SERVER_KEY, never replays another principal's response.
|
|
``route`` is the logical endpoint (``/v1/...`` and its ``/p/<profile>/v1/...`` alias are the same
|
|
route), folded into the key because the store keeps the fingerprint only as the slot's value.
|
|
"""
|
|
from gateway.platforms.api_server import _error_response, _idem_cache, _make_request_fingerprint
|
|
idempotency_key = request.headers.get("Idempotency-Key")
|
|
try:
|
|
if idempotency_key:
|
|
principal_scope = self._run_idempotency_scope(request)
|
|
scoped_key = f"{principal_scope}\0{route}\0{idempotency_key}"
|
|
fp = _make_request_fingerprint(body, keys=fingerprint_keys)
|
|
result, usage = await _idem_cache.get_or_set(scoped_key, fp, compute)
|
|
else:
|
|
result, usage = await compute()
|
|
return (result, usage), None
|
|
except Exception as e:
|
|
logger.error("Error running agent for %s: %s", log_label, e, exc_info=True)
|
|
message = "" if getattr(e, "_notification_presentation_suppressed", False) is True else f"Internal server error: {e}"
|
|
return None, _error_response(message, 500, err_type="server_error")
|
|
|
|
async def _prepare_sse_response(
|
|
self, request: "web.Request", session_id: Optional[str], gateway_session_key: Optional[str],
|
|
) -> "web.StreamResponse":
|
|
"""Open a prepared SSE StreamResponse with CORS + session headers (the CORS middleware
|
|
can't inject headers after ``prepare()`` flushes them, so they are resolved here)."""
|
|
sse_headers = self._sse_headers(request)
|
|
if session_id:
|
|
sse_headers["X-Hermes-Session-Id"] = session_id
|
|
if gateway_session_key:
|
|
sse_headers["X-Hermes-Session-Key"] = gateway_session_key
|
|
response = web.StreamResponse(status=200, headers=sse_headers)
|
|
await response.prepare(request)
|
|
return response
|
|
|
|
async def _write_sse_chat_completion(
|
|
self, request: "web.Request", completion_id: str, model: str,
|
|
created: int, stream_q, agent_task, agent_ref=None, session_id: str = None,
|
|
gateway_session_key: str = None) -> "web.StreamResponse":
|
|
"""Stream ``chat.completion.chunk`` frames from the agent's delta queue. On client
|
|
disconnect the agent is interrupted (stops LLM calls), then its task wrapper cancelled."""
|
|
from gateway.platforms.api_server import (
|
|
_abandon_agent_task, _chat_usage_payload, _resolve_media_to_data_urls, _sse_frame)
|
|
response = await self._prepare_sse_response(request, session_id, gateway_session_key)
|
|
|
|
def _chunk(delta: Dict[str, Any], finish_reason=None, **extra) -> Dict[str, Any]:
|
|
return {"id": completion_id, "object": "chat.completion.chunk", "created": created,
|
|
"model": model,
|
|
"choices": [{"index": 0, "delta": delta, "finish_reason": finish_reason}], **extra}
|
|
content_sent = False
|
|
try:
|
|
await response.write(_sse_frame(_chunk({"role": "assistant"})))
|
|
async for delta in _iter_stream_items(stream_q, agent_task, response):
|
|
if delta is None:
|
|
break
|
|
if isinstance(delta, tuple) and len(delta) == 2 and delta[0] == "__tool_progress__":
|
|
# Custom event: tool lifecycle for frontends without markers in history.
|
|
if not self._tool_progress_events:
|
|
continue # opted out for strict OpenAI clients (#12020)
|
|
await response.write(_sse_frame(delta[1], event="hermes.tool.progress"))
|
|
elif isinstance(delta, tuple) and len(delta) == 2 and delta[0] == "__reasoning__":
|
|
# DeepSeek-style ``delta.reasoning_content`` (#99552), the field Open WebUI,
|
|
# opencode and the Vercel AI SDK render as a thinking block.
|
|
await response.write(_sse_frame(_chunk({"reasoning_content": delta[1]})))
|
|
elif isinstance(delta, tuple) and len(delta) == 2 and delta[0] == "__status__":
|
|
await response.write(_sse_frame(delta[1], event="hermes.status"))
|
|
elif isinstance(delta, tuple) and len(delta) == 2 and delta[0] == "__approval__":
|
|
await response.write(_sse_frame(delta[1], event="approval.request"))
|
|
else:
|
|
if delta:
|
|
content_sent = True
|
|
await response.write(_sse_frame(_chunk({"content": delta})))
|
|
# The agent can fail after the queue drains (task raises / result flagged failed or
|
|
# partial): surface a non-"stop" finish_reason like the non-streaming path.
|
|
usage = {"input_tokens": 0, "output_tokens": 0, "total_tokens": 0}
|
|
result = agent_error = None
|
|
try:
|
|
result, agent_usage = await agent_task
|
|
usage = agent_usage or usage
|
|
except Exception as exc:
|
|
agent_error = exc
|
|
logger.error("Agent task %s failed during SSE streaming: %s", completion_id, exc)
|
|
completed, is_partial, is_failed, err_msg = _result_flags(result)
|
|
if agent_error is not None:
|
|
is_failed = True
|
|
err_msg = err_msg or str(agent_error)
|
|
finish_reason = _finish_reason(completed, is_partial, is_failed, err_msg, agent_error)
|
|
finish_chunk = _chunk({}, finish_reason, usage=_chat_usage_payload(usage))
|
|
presentation_muted = (
|
|
(isinstance(result, dict) and result.get("_notification_presentation_suppressed") is True)
|
|
or getattr(agent_error, "_notification_presentation_suppressed", False) is True
|
|
)
|
|
# Recovery paths (guardrail halt, partial_stream_recovery, fallback prior-turn
|
|
# content) can return a final_response without firing any content delta; emit it
|
|
# once so the client does not see an empty stream (#31449). Mirrors
|
|
# _ResponsesStream.collect_result for /v1/responses.
|
|
if not content_sent and not presentation_muted and isinstance(result, dict):
|
|
fallback_text = _resolve_media_to_data_urls(result.get("final_response") or "")
|
|
if fallback_text:
|
|
await response.write(_sse_frame(_chunk({"content": fallback_text})))
|
|
elif not presentation_muted:
|
|
# Chat chunks can only append: a non-append rewrite follows the streamed text (as in the CLI).
|
|
tail, appended = _post_stream_transform(result)
|
|
if tail:
|
|
await response.write(_sse_frame(_chunk({"content": tail if appended else _transformed_notice() + tail})))
|
|
if finish_reason != "stop":
|
|
if err_msg and not presentation_muted:
|
|
finish_chunk["error"] = {
|
|
"message": err_msg,
|
|
"type": type(agent_error).__name__ if agent_error else "agent_error"}
|
|
finish_chunk["hermes"] = _hermes_extras(
|
|
completed, is_partial, is_failed, "" if presentation_muted else err_msg, finish_reason)
|
|
await response.write(_sse_frame(finish_chunk))
|
|
await response.write(b"data: [DONE]\n\n")
|
|
except (ConnectionResetError, ConnectionAbortedError, BrokenPipeError, OSError):
|
|
await _abandon_agent_task(agent_ref, agent_task, "SSE client disconnected")
|
|
logger.info("SSE client disconnected; interrupted agent task %s", completion_id)
|
|
except Exception:
|
|
# Agent crashed mid-stream: an error chunk beats a TransferEncodingError.
|
|
import traceback as _tb
|
|
logger.error("Agent crashed mid-stream for %s: %s", completion_id, _tb.format_exc()[:300])
|
|
with suppress(Exception):
|
|
await response.write(_sse_frame(_chunk({}, "error")))
|
|
await response.write(b"data: [DONE]\n\n")
|
|
return response
|
|
|
|
async def _write_sse_responses(
|
|
self, request: "web.Request", response_id: str, model: str, created_at: int, stream_q,
|
|
agent_task, agent_ref, conversation_history: List[Dict[str, str]], user_message: str,
|
|
instructions: Optional[str], conversation: Optional[str], store: bool, session_id: str,
|
|
gateway_session_key: Optional[str] = None) -> "web.StreamResponse":
|
|
"""Write the SSE stream for POST /v1/responses.
|
|
|
|
Events: ``response.created`` -> ``output_text.delta/done`` + ``output_item.added/done``
|
|
(reasoning / function_call / function_call_output) + ``reasoning_summary_part/text.*``
|
|
-> ``response.completed`` (non-streaming envelope)
|
|
or ``response.failed``. On disconnect the agent is interrupted and, with ``store=True``,
|
|
an ``incomplete`` snapshot replaces ``in_progress`` so GET / chaining still work.
|
|
"""
|
|
from gateway.platforms.api_server import _abandon_agent_task, _redact_api_error_text
|
|
response = await self._prepare_sse_response(request, session_id, gateway_session_key)
|
|
st = _ResponsesStream(
|
|
self, response, response_id=response_id, model=model, created_at=created_at,
|
|
conversation_history=conversation_history, user_message=user_message,
|
|
instructions=instructions, conversation=conversation, store=store, session_id=session_id)
|
|
try:
|
|
await st.emit_created()
|
|
async for item in _iter_stream_items(stream_q, agent_task, response):
|
|
if item is None: # EOS sentinel
|
|
st.cancel_batch_timer()
|
|
await st.flush_batch()
|
|
break
|
|
await st.dispatch(item)
|
|
await st.flush_batch()
|
|
await st.collect_result(agent_task)
|
|
await st.close_message_item()
|
|
if st.agent_error:
|
|
await st.emit_failed()
|
|
else:
|
|
await st.emit_completed()
|
|
except (ConnectionResetError, ConnectionAbortedError, BrokenPipeError, OSError):
|
|
st.persist_incomplete_if_needed()
|
|
await _abandon_agent_task(agent_ref, agent_task, "SSE client disconnected")
|
|
logger.info("SSE client disconnected; interrupted agent task %s", response_id)
|
|
except asyncio.CancelledError:
|
|
# Server-side cancellation (shutdown, timeout): persist incomplete, then re-raise.
|
|
st.persist_incomplete_if_needed()
|
|
await _abandon_agent_task(
|
|
agent_ref, agent_task, "SSE task cancelled",
|
|
reap_source="api_server_sse_cancelled", await_cancel=False)
|
|
logger.info("SSE task cancelled; persisted incomplete snapshot for %s", response_id)
|
|
raise
|
|
except Exception as exc:
|
|
# Unhandled agent error (BadRequestError, AuthenticationError, ...): emit
|
|
# response.failed and end the stream cleanly (no TransferEncodingError).
|
|
import traceback as _tb
|
|
st.persist_incomplete_if_needed()
|
|
st.agent_error = _redact_api_error_text(_tb.format_exc())
|
|
with suppress(Exception):
|
|
await st.emit_crash(exc)
|
|
logger.error("Agent crashed mid-stream for %s: %s", response_id, str(st.agent_error)[:300])
|
|
return response
|
|
|
|
async def _handle_responses(self, request: "web.Request") -> "web.Response":
|
|
"""POST /v1/responses — OpenAI Responses API format."""
|
|
from gateway.platforms.api_server import (
|
|
ThreadSafeAsyncQueue, _auto_truncate_response_history, _coerce_request_bool,
|
|
_content_has_visible_payload, _error_response, _invalid_request,
|
|
_multimodal_validation_error, _normalize_multimodal_content, _redact_api_error_text,
|
|
_resolve_media_to_data_urls, _responses_usage_payload)
|
|
# Bound total in-flight agent runs (configurable; #7483).
|
|
limited = self._concurrency_limited_response()
|
|
if limited is not None:
|
|
return limited
|
|
gateway_session_key, key_err = self._parse_session_key_header(request)
|
|
if key_err is not None:
|
|
return key_err
|
|
try:
|
|
body = await request.json()
|
|
except Exception:
|
|
return _invalid_request("Invalid JSON in request body")
|
|
from gateway.platforms.api_server import _request_relay_metadata
|
|
relay_metadata = _request_relay_metadata(body)
|
|
raw_input = body.get("input")
|
|
if raw_input is None:
|
|
return _error_response("Missing 'input' field", 400)
|
|
instructions = body.get("instructions")
|
|
previous_response_id = body.get("previous_response_id")
|
|
conversation = body.get("conversation")
|
|
store = _coerce_request_bool(body.get("store"), default=True)
|
|
if conversation and previous_response_id:
|
|
return _error_response("Cannot use both 'conversation' and 'previous_response_id'", 400)
|
|
if conversation:
|
|
# A conversation name resolves to its latest response_id (unknown = new conversation).
|
|
previous_response_id = self._current_response_store().get_conversation(conversation)
|
|
|
|
input_messages: List[Dict[str, Any]] = []
|
|
if isinstance(raw_input, str):
|
|
input_messages = [{"role": "user", "content": raw_input}]
|
|
elif isinstance(raw_input, list):
|
|
for idx, item in enumerate(raw_input):
|
|
if isinstance(item, str):
|
|
input_messages.append({"role": "user", "content": item})
|
|
elif _is_reasoning_input_item(item):
|
|
continue
|
|
elif isinstance(item, dict):
|
|
try:
|
|
content = _normalize_multimodal_content(item.get("content", ""))
|
|
except ValueError as exc:
|
|
return _multimodal_validation_error(exc, param=f"input[{idx}].content")
|
|
input_messages.append({"role": item.get("role", "user"), "content": content})
|
|
else:
|
|
return _error_response("'input' must be a string or array", 400)
|
|
|
|
# Explicit conversation_history (stateless clients) beats previous_response_id chaining.
|
|
conversation_history: List[Dict[str, Any]] = []
|
|
raw_history = body.get("conversation_history")
|
|
if raw_history:
|
|
if not isinstance(raw_history, list):
|
|
return _error_response("'conversation_history' must be an array of message objects", 400)
|
|
for i, entry in enumerate(raw_history):
|
|
if _is_reasoning_input_item(entry):
|
|
continue
|
|
if not isinstance(entry, dict) or "role" not in entry or "content" not in entry:
|
|
return _error_response(f"conversation_history[{i}] must have 'role' and 'content' fields", 400)
|
|
try:
|
|
entry_content = _normalize_multimodal_content(entry["content"])
|
|
except ValueError as exc:
|
|
return _multimodal_validation_error(exc, param=f"conversation_history[{i}].content")
|
|
conversation_history.append({"role": str(entry["role"]), "content": entry_content})
|
|
if previous_response_id:
|
|
logger.debug("Both conversation_history and previous_response_id provided; using conversation_history")
|
|
stored_session_id = None
|
|
if not conversation_history and previous_response_id:
|
|
stored = self._current_response_store().get(previous_response_id)
|
|
if stored is None:
|
|
return _error_response(f"Previous response not found: {previous_response_id}", 404)
|
|
conversation_history = list(stored.get("conversation_history", []))
|
|
stored_session_id = stored.get("session_id")
|
|
if instructions is None:
|
|
instructions = stored.get("instructions")
|
|
# All input messages but the last become history; the last is the user message.
|
|
conversation_history.extend(input_messages[:-1])
|
|
user_message: Any = input_messages[-1].get("content", "") if input_messages else ""
|
|
if not _content_has_visible_payload(user_message):
|
|
return _error_response("No user message found in input", 400)
|
|
if body.get("truncation") == "auto":
|
|
conversation_history = _auto_truncate_response_history(conversation_history)
|
|
|
|
# Session precedence: previous_response_id chain > declared X-Hermes-Session-Key > fresh
|
|
# id. Binding the declared key follows the same precedence: a chain-selected session must
|
|
# not have its routing key rewritten to this header.
|
|
_declared_selected = not stored_session_id and bool(gateway_session_key)
|
|
session_id = (
|
|
stored_session_id
|
|
or await asyncio.to_thread(self._declared_conversation_session, gateway_session_key)
|
|
or str(uuid.uuid4()))
|
|
stream = _coerce_request_bool(body.get("stream"), default=False)
|
|
route, agent_overrides, selection_error = self._select_request_route(
|
|
body, session_id=session_id, gateway_session_key=gateway_session_key,
|
|
model_alias=body.get("model"))
|
|
if selection_error is not None:
|
|
return selection_error
|
|
run_kwargs = dict(
|
|
user_message=user_message, conversation_history=conversation_history,
|
|
ephemeral_system_prompt=instructions, session_id=session_id,
|
|
gateway_session_key=gateway_session_key, bind_declared_conversation=_declared_selected,
|
|
**agent_overrides, route=route, relay_metadata=relay_metadata)
|
|
if stream:
|
|
_stream_q = ThreadSafeAsyncQueue()
|
|
|
|
def _on_tool_progress(event_type, name, preview, args, **kwargs):
|
|
return # structured start/complete callbacks carry the call id; progress ignored
|
|
|
|
def _on_tool_start(tool_call_id, function_name, function_args):
|
|
_stream_q.put_threadsafe(("__tool_started__", {
|
|
"tool_call_id": tool_call_id, "name": function_name,
|
|
"arguments": function_args or {}}))
|
|
|
|
def _on_tool_complete(tool_call_id, function_name, function_args, function_result):
|
|
_stream_q.put_threadsafe(("__tool_completed__", {
|
|
"tool_call_id": tool_call_id, "name": function_name,
|
|
"arguments": function_args or {}, "result": function_result}))
|
|
|
|
def _on_commentary(text, *, already_streamed: bool = False):
|
|
# Already-streamed text went out as output_text.delta of the final item; a second
|
|
# copy as a commentary item would duplicate it.
|
|
if not already_streamed and isinstance(text, str) and text.strip():
|
|
_stream_q.put_threadsafe(("__commentary__", {"text": text}))
|
|
agent_task, agent_ref = self._spawn_stream_agent(
|
|
_stream_q, tool_progress_callback=_on_tool_progress,
|
|
tool_start_callback=_on_tool_start, tool_complete_callback=_on_tool_complete,
|
|
interim_assistant_callback=_on_commentary, **run_kwargs)
|
|
return await self._write_sse_responses(
|
|
request=request, response_id=f"resp_{uuid.uuid4().hex[:28]}",
|
|
model=body.get("model", self._model_name), created_at=int(time.time()),
|
|
stream_q=_stream_q, agent_task=agent_task, agent_ref=agent_ref,
|
|
conversation_history=conversation_history, user_message=user_message,
|
|
instructions=instructions, conversation=conversation, store=store,
|
|
session_id=session_id, gateway_session_key=gateway_session_key)
|
|
|
|
async def _compute_response():
|
|
return await self._run_agent(**run_kwargs)
|
|
outcome, err = await self._run_idempotent(
|
|
request, body, _compute_response, log_label="responses",
|
|
fingerprint_keys=["input", "instructions", "previous_response_id", "conversation", "model", "provider", "model_options", "tools"],
|
|
route="responses",
|
|
)
|
|
if err is not None:
|
|
return err
|
|
result, usage = outcome
|
|
final_response = _resolve_media_to_data_urls(result.get("final_response", ""))
|
|
if not final_response:
|
|
final_response = _redact_api_error_text(result.get("error", "(No response generated)"))
|
|
response_id = f"resp_{uuid.uuid4().hex[:28]}"
|
|
created_at = int(time.time())
|
|
full_history = self._build_response_conversation_history(
|
|
conversation_history, user_message, result, final_response,
|
|
tool_output_max_chars=self._history_tool_output_max_chars)
|
|
# _run_agent's effective session id carries compression rotations; storing it keeps
|
|
# previous_response_id chaining off the pre-rotation session (else compression re-fires).
|
|
_result_sid = result.get("session_id") if isinstance(result, dict) else None
|
|
_effective_session_id = (
|
|
_result_sid if isinstance(_result_sid, str) and _result_sid else session_id)
|
|
# Output items = current turn only (AIAgent returns a full transcript; mocked paths
|
|
# only the current-turn suffix).
|
|
output_start_index = self._response_messages_turn_start_index(
|
|
conversation_history, user_message, result)
|
|
response_data = {
|
|
"id": response_id, "object": "response", "status": "completed",
|
|
"created_at": created_at, "model": body.get("model", self._model_name),
|
|
"output": self._extract_output_items(result, start_index=output_start_index),
|
|
"usage": _responses_usage_payload(usage)}
|
|
if store:
|
|
response_store = self._current_response_store()
|
|
response_store.put(response_id, {
|
|
"response": response_data, "conversation_history": full_history,
|
|
"instructions": instructions, "session_id": _effective_session_id})
|
|
if conversation:
|
|
response_store.set_conversation(conversation, response_id)
|
|
response_headers = {"X-Hermes-Session-Id": _effective_session_id}
|
|
if gateway_session_key:
|
|
response_headers["X-Hermes-Session-Key"] = gateway_session_key
|
|
return web.json_response(response_data, headers=response_headers)
|
|
|
|
async def _handle_get_response(self, request: "web.Request") -> "web.Response":
|
|
"""GET /v1/responses/{response_id} — retrieve a stored response."""
|
|
from gateway.platforms.api_server import _error_response
|
|
auth_err = self._check_auth(request)
|
|
if auth_err:
|
|
return auth_err
|
|
response_id = request.match_info["response_id"]
|
|
stored = self._current_response_store().get(response_id)
|
|
if stored is None:
|
|
return _error_response(f"Response not found: {response_id}", 404)
|
|
return web.json_response(stored["response"])
|
|
|
|
async def _handle_delete_response(self, request: "web.Request") -> "web.Response":
|
|
"""DELETE /v1/responses/{response_id} — delete a stored response."""
|
|
from gateway.platforms.api_server import _error_response
|
|
auth_err = self._check_auth(request)
|
|
if auth_err:
|
|
return auth_err
|
|
response_id = request.match_info["response_id"]
|
|
if not self._current_response_store().delete(response_id):
|
|
return _error_response(f"Response not found: {response_id}", 404)
|
|
return web.json_response({"id": response_id, "object": "response", "deleted": True})
|
|
|
|
@staticmethod
|
|
def _build_response_conversation_history(
|
|
conversation_history: List[Dict[str, Any]], user_message: Any, result: Dict[str, Any],
|
|
final_response: Any, *, tool_output_max_chars: int = 0) -> List[Dict[str, Any]]:
|
|
"""Build the stored Responses transcript without duplicating history.
|
|
|
|
A compressed transcript (``result["_compressed"]``) shares no input-history prefix, so
|
|
turn-start detection fails; prepending the uncompressed history would bloat the stored
|
|
context and re-trigger compression every request — it is stored as-is instead.
|
|
|
|
``tool_output_max_chars`` > 0 caps tool outputs / tool-call argument blobs in the
|
|
stored copy (gateway.api_server.history_tool_output_max_chars; 0 = store verbatim).
|
|
"""
|
|
from gateway.platforms.api_server import APIServerAdapter
|
|
prior = list(conversation_history)
|
|
current_user = {"role": "user", "content": user_message}
|
|
agent_messages = result.get("messages") if isinstance(result, dict) else None
|
|
if isinstance(agent_messages, list) and agent_messages:
|
|
turn_start = APIServerAdapter._response_messages_turn_start_index(
|
|
conversation_history, user_message, result)
|
|
# turn_start == 0: compression rewrote the transcript or agent_messages is turn-only.
|
|
if turn_start or result.get("_compressed"):
|
|
history = list(agent_messages)
|
|
else:
|
|
history = prior + [current_user] + agent_messages
|
|
else:
|
|
history = prior + [current_user, {"role": "assistant", "content": final_response}]
|
|
return _cap_history_tool_outputs(history, tool_output_max_chars)
|
|
|
|
@staticmethod
|
|
def _response_messages_turn_start_index(
|
|
conversation_history: List[Dict[str, Any]], user_message: Any, result: Dict[str, Any],
|
|
) -> int:
|
|
"""Index where this turn starts in a transcript-shaped result["messages"] (0 = all)."""
|
|
from gateway.platforms.api_server_turn_boundary import response_turn_start_index
|
|
return response_turn_start_index(conversation_history, user_message, result)
|
|
|
|
@classmethod
|
|
def _turn_transcript_messages(
|
|
cls, conversation_history: List[Dict[str, Any]], user_message: Any, result: Dict[str, Any],
|
|
) -> List[Dict[str, Any]]:
|
|
"""This turn's assistant/tool messages in client-safe shape: clients accumulating
|
|
``assistant.delta`` into one buffer cannot reconstruct assistant segments that preceded
|
|
tool calls, so ``run.completed`` carries the authoritative per-turn transcript.
|
|
|
|
Emitting the authoritative per-turn transcript on ``run.completed`` lets any SSE consumer reconcile
|
|
its live view against ground truth without a separate ``GET /messages`` round-trip. Purely additive:
|
|
clients that ignore the field are unaffected. Refs #34703.
|
|
"""
|
|
agent_messages = result.get("messages") if isinstance(result, dict) else None
|
|
if not isinstance(agent_messages, list) or not agent_messages:
|
|
return []
|
|
start = cls._response_messages_turn_start_index(conversation_history, user_message, result)
|
|
out: List[Dict[str, Any]] = []
|
|
for msg in agent_messages[start:]:
|
|
if not isinstance(msg, dict) or msg.get("role") not in {"assistant", "tool"}:
|
|
continue
|
|
# _message_response projects compaction scaffolding; pure handoffs are "hidden".
|
|
projected = cls._message_response(msg)
|
|
if projected.get("display_kind") != "hidden":
|
|
out.append(projected)
|
|
return out
|
|
|
|
@staticmethod
|
|
def _extract_output_items(result: Dict[str, Any], start_index: int = 0) -> List[Dict[str, Any]]:
|
|
"""Output items from ``result["messages"][start_index:]``: ``function_call`` per assistant
|
|
tool_call, ``function_call_output`` per tool message, then the final ``message``."""
|
|
from gateway.platforms.api_server import _redact_api_error_text
|
|
items: List[Dict[str, Any]] = []
|
|
messages = result.get("messages", [])
|
|
if start_index > 0:
|
|
messages = messages[start_index:]
|
|
for msg in messages:
|
|
role = msg.get("role")
|
|
reasoning = msg.get("reasoning") if role == "assistant" else None
|
|
if isinstance(reasoning, str) and reasoning.strip():
|
|
# Precedes this message's function_call items, like the SSE writer closes a
|
|
# thinking burst before the next tool item opens (#99552).
|
|
items.append(_reasoning_item(reasoning))
|
|
if role == "assistant" and msg.get("tool_calls"):
|
|
for tc in msg["tool_calls"]:
|
|
func = tc.get("function", {})
|
|
# Already executed server-side; replayed for structured tool UI only, so
|
|
# marked completed (matching the SSE path) — never pending client calls.
|
|
items.append({
|
|
"id": f"fc_{uuid.uuid4().hex[:24]}", "type": "function_call",
|
|
"status": "completed", "name": func.get("name", ""),
|
|
"arguments": func.get("arguments", ""),
|
|
"call_id": tc.get("id", "")})
|
|
elif role == "tool":
|
|
items.append({
|
|
"id": f"fco_{uuid.uuid4().hex[:24]}", "type": "function_call_output",
|
|
"status": "completed", "call_id": msg.get("tool_call_id", ""),
|
|
"output": msg.get("content", "")})
|
|
final = result.get("final_response", "") or _redact_api_error_text(
|
|
result.get("error", "(No response generated)"))
|
|
items.append(_message_item(final))
|
|
return items
|