hermes-trace/HERMES_PLUGIN_RESEARCH.md

8 KiB

Hermes Plugin System — Research Notes for hermes-trace

Findings from reading the Hermes Agent source code and official documentation.

Correct hook choice: pre_api_request / post_api_request

These are the official hooks for per-LLM-call telemetry. Not middleware, not pre_llm_call.

Evidence: Hermes Agent's own bundled Langfuse plugin at plugins/observability/langfuse/__init__.py uses exactly these hooks.

From register() in the Langfuse plugin (L1026-1035):

def register(ctx) -> None:
    # Register for both hook name variants so the plugin works across
    # Hermes versions.  pre_api_request / post_api_request fire per API
    # call (preferred); pre_llm_call / post_llm_call fire once per turn.
    ctx.register_hook("pre_api_request", on_pre_llm_request)
    ctx.register_hook("post_api_request", on_post_llm_call)
    ctx.register_hook("pre_llm_call", on_pre_llm_call)
    ctx.register_hook("post_llm_call", on_post_llm_call)
    ctx.register_hook("pre_tool_call", on_pre_tool_call)
    ctx.register_hook("post_tool_call", on_post_tool_call)

Key insight: pre_api_request / post_api_request fire per individual LLM API call — every time the agent calls the provider, including retries and interim calls inside the tool loop. This is exactly what we need for telemetry, and it's the preferred mechanism per the code comments.

Why not middleware

ctx.register_middleware("llm_execution", callback) technically exists in the code (hermes_cli/plugins.py) but is not documented in any official Hermes documentation (website/docs/). Using it would be relying on an undocumented internal mechanism.

Callback signatures — from source code

pre_api_request

Called at agent/conversation_loop.py L1012-1058, every API call, before the call is made.

def on_pre_api_request(
    *,
    task_id: str,
    turn_id: str,
    api_request_id: str,
    session_id: str,
    user_message: str,
    conversation_history: list,     # full conversation so far
    platform: str,
    model: str,
    provider: str,
    base_url: str,
    api_mode: str,
    api_call_count: int,            # which call in this turn (0, 1, 2...)
    request_messages: list,         # the actual messages sent to the LLM
    message_count: int,             # total messages in request
    tool_count: int,
    approx_input_tokens: int,
    request_char_count: int,
    max_tokens: int,
    started_at: float,              # unix timestamp
    **_: Any,
) -> None:

The request_messages list contains the full prompt sent to the LLM, including system prompt + conversation history + current user message. This is where we can detect background review forks by scanning for the review prompt text.

post_api_request

Called at agent/conversation_loop.py L3553-3588, every API call, after the response is received.

def on_post_api_request(
    *,
    task_id: str,
    turn_id: str,
    api_request_id: str,
    session_id: str,
    platform: str,
    model: str,
    provider: str,
    base_url: str,
    api_mode: str,
    api_call_count: int,
    api_duration: float,            # seconds
    started_at: float,
    ended_at: float,
    finish_reason: str,             # "stop", "tool_calls", "length", ...
    message_count: int,
    response_model: str,
    response: dict,                 # sanitized response payload (via _api_response_payload_for_hook)
    usage: dict,                    # token usage summary (via _usage_summary_for_api_request_hook)
    assistant_message: object,      # the message object — has .content, .tool_calls, .reasoning
    assistant_content_chars: int,
    assistant_tool_call_count: int,
    **_: Any,
) -> None:

The usage dict contains (from _usage_summary_for_api_request_hook):

  • input_tokens / prompt_tokens
  • output_tokens / completion_tokens
  • total_tokens
  • cache_read_tokens
  • cache_write_tokens
  • reasoning_tokens

The response dict (from _api_response_payload_for_hook) contains a sanitized view with:

  • Content / tool calls from the choice
  • Finish reason
  • But NOT raw usage (that's in the separate usage param)

The assistant_message object has:

  • .content — the text response
  • .reasoning — reasoning content (deepseek-v4, etc.)
  • .tool_calls — list of tool call objects with .id, .function.name, .function.arguments

pre_tool_call

Called before each tool executes. Gets tool_name, args, task_id, session_id, tool_call_id.

post_tool_call

Called after each tool returns. Gets tool_name, args, result, task_id, session_id, tool_call_id, duration_ms.

Detecting background review forks

In pre_api_request, scan request_messages for the _SKILL_REVIEW_PROMPT text:

def on_pre_api_request(*, request_messages, session_id, model, **kw):
    for msg in request_messages:
        content = msg.get("content", "") or ""
        if (
            isinstance(content, str)
            and "Review the conversation above and update the skill library" in content
        ):
            # This is a background review fork!
            turn_type = "background_review"
            break

Langfuse plugin reference

The canonical reference implementation is at:

~/.hermes/hermes-agent/plugins/observability/langfuse/__init__.py

This is Hermes Agent's bundled Langfuse observability plugin. Key sections:

Lines Content
1-22 Module docstring + env var docs
43-56 TraceState dataclass — trace tree state management
712-750 on_pre_llm_call — turn-scoped hook handler
752-821 on_pre_llm_request — per-API-call hook handler (our model)
824-949 on_post_llm_call — per-API-call post hook handler (our model)
952-976 on_pre_tool_call — per-tool pre hook
978-1023 on_post_tool_call — per-tool post hook
1026-1035 register() — hook registration

Note: the Langfuse plugin has the naming confusing where on_pre_llm_request (per call) and on_pre_llm_call (per turn) are different things. We should follow the naming more clearly for hermes-trace.

Hermes plugin structure (hermes-trace)

~/.hermes/plugins/hermes-trace/
├── plugin.yaml              # Manifest: name, version, description
├── PLAN.md                  # Design document
├── HERMES_PLUGIN_RESEARCH.md # this file
├── __init__.py              # register(ctx): wire hooks
├── db.py                    # SQLite schema + write/read
├── app.py                   # Granian + FastAPI + Jinja2 frontend
└── templates/               # Jinja2 HTML templates

Plugin lifecycle notes

  • register(ctx) is called exactly once at Hermes startup (gateway process or CLI).
  • Daemon threads (threading.Thread(daemon=True)) spawned inside register() live as long as the Hermes process.
  • This is safe for: background cleanup loops, the Granian frontend server, or any async housekeeping.
  • If the plugin fails to load (exception in register()), Hermes logs a warning and continues; the plugin is skipped.

Source files consulted

File Lines Content
plugins/observability/langfuse/__init__.py 1-1035 Bundled Langfuse plugin — our reference implementation
plugins/observability/nemo_relay/__init__.py 1-500+ Another observability plugin using same hooks
agent/conversation_loop.py 1008-1058 pre_api_request hook call site
agent/conversation_loop.py 3545-3588 post_api_request hook call site
hermes_cli/plugins.py 290-550 PluginContext class — all ctx.register_* methods
hermes_cli/plugins.py 997-1030 register_hook implementation
agent/background_review.py 428-710 Fork mechanism
agent/turn_finalizer.py 375-401 Review trigger
website/docs/guides/build-a-hermes-plugin.md 1-1254 Official plugin guide

Frontend Research (added 2026-06-18)

See AGENTS.md for current frontend architecture (flat-table + overlay trace viewer with HTMX + Alpine patterns).