200 lines
8 KiB
Markdown
200 lines
8 KiB
Markdown
# Hermes Plugin System — Research Notes for hermes-trace
|
|
|
|
> Findings from reading the Hermes Agent source code and official documentation.
|
|
|
|
## Correct hook choice: `pre_api_request` / `post_api_request`
|
|
|
|
**These are the official hooks for per-LLM-call telemetry.** Not middleware, not `pre_llm_call`.
|
|
|
|
Evidence: Hermes Agent's own bundled Langfuse plugin at `plugins/observability/langfuse/__init__.py` uses exactly these hooks.
|
|
|
|
From `register()` in the Langfuse plugin (L1026-1035):
|
|
|
|
```python
|
|
def register(ctx) -> None:
|
|
# Register for both hook name variants so the plugin works across
|
|
# Hermes versions. pre_api_request / post_api_request fire per API
|
|
# call (preferred); pre_llm_call / post_llm_call fire once per turn.
|
|
ctx.register_hook("pre_api_request", on_pre_llm_request)
|
|
ctx.register_hook("post_api_request", on_post_llm_call)
|
|
ctx.register_hook("pre_llm_call", on_pre_llm_call)
|
|
ctx.register_hook("post_llm_call", on_post_llm_call)
|
|
ctx.register_hook("pre_tool_call", on_pre_tool_call)
|
|
ctx.register_hook("post_tool_call", on_post_tool_call)
|
|
```
|
|
|
|
**Key insight:** `pre_api_request` / `post_api_request` fire **per individual LLM API call** — every time the agent calls the provider, including retries and interim calls inside the tool loop. This is exactly what we need for telemetry, and it's the **preferred** mechanism per the code comments.
|
|
|
|
## Why not middleware
|
|
|
|
`ctx.register_middleware("llm_execution", callback)` technically exists in the code (`hermes_cli/plugins.py`) but is **not documented** in any official Hermes documentation (`website/docs/`). Using it would be relying on an undocumented internal mechanism.
|
|
|
|
## Callback signatures — from source code
|
|
|
|
### `pre_api_request`
|
|
|
|
Called at `agent/conversation_loop.py` L1012-1058, **every API call, before the call is made.**
|
|
|
|
```python
|
|
def on_pre_api_request(
|
|
*,
|
|
task_id: str,
|
|
turn_id: str,
|
|
api_request_id: str,
|
|
session_id: str,
|
|
user_message: str,
|
|
conversation_history: list, # full conversation so far
|
|
platform: str,
|
|
model: str,
|
|
provider: str,
|
|
base_url: str,
|
|
api_mode: str,
|
|
api_call_count: int, # which call in this turn (0, 1, 2...)
|
|
request_messages: list, # the actual messages sent to the LLM
|
|
message_count: int, # total messages in request
|
|
tool_count: int,
|
|
approx_input_tokens: int,
|
|
request_char_count: int,
|
|
max_tokens: int,
|
|
started_at: float, # unix timestamp
|
|
**_: Any,
|
|
) -> None:
|
|
```
|
|
|
|
The `request_messages` list contains the full prompt sent to the LLM, including system prompt + conversation history + current user message. This is where we can detect background review forks by scanning for the review prompt text.
|
|
|
|
### `post_api_request`
|
|
|
|
Called at `agent/conversation_loop.py` L3553-3588, **every API call, after the response is received.**
|
|
|
|
```python
|
|
def on_post_api_request(
|
|
*,
|
|
task_id: str,
|
|
turn_id: str,
|
|
api_request_id: str,
|
|
session_id: str,
|
|
platform: str,
|
|
model: str,
|
|
provider: str,
|
|
base_url: str,
|
|
api_mode: str,
|
|
api_call_count: int,
|
|
api_duration: float, # seconds
|
|
started_at: float,
|
|
ended_at: float,
|
|
finish_reason: str, # "stop", "tool_calls", "length", ...
|
|
message_count: int,
|
|
response_model: str,
|
|
response: dict, # sanitized response payload (via _api_response_payload_for_hook)
|
|
usage: dict, # token usage summary (via _usage_summary_for_api_request_hook)
|
|
assistant_message: object, # the message object — has .content, .tool_calls, .reasoning
|
|
assistant_content_chars: int,
|
|
assistant_tool_call_count: int,
|
|
**_: Any,
|
|
) -> None:
|
|
```
|
|
|
|
The `usage` dict contains (from `_usage_summary_for_api_request_hook`):
|
|
- `input_tokens` / `prompt_tokens`
|
|
- `output_tokens` / `completion_tokens`
|
|
- `total_tokens`
|
|
- `cache_read_tokens`
|
|
- `cache_write_tokens`
|
|
- `reasoning_tokens`
|
|
|
|
The `response` dict (from `_api_response_payload_for_hook`) contains a sanitized view with:
|
|
- Content / tool calls from the choice
|
|
- Finish reason
|
|
- But NOT raw usage (that's in the separate `usage` param)
|
|
|
|
The `assistant_message` object has:
|
|
- `.content` — the text response
|
|
- `.reasoning` — reasoning content (deepseek-v4, etc.)
|
|
- `.tool_calls` — list of tool call objects with `.id`, `.function.name`, `.function.arguments`
|
|
|
|
### `pre_tool_call`
|
|
|
|
Called before each tool executes. Gets `tool_name`, `args`, `task_id`, `session_id`, `tool_call_id`.
|
|
|
|
### `post_tool_call`
|
|
|
|
Called after each tool returns. Gets `tool_name`, `args`, `result`, `task_id`, `session_id`, `tool_call_id`, `duration_ms`.
|
|
|
|
## Detecting background review forks
|
|
|
|
In `pre_api_request`, scan `request_messages` for the `_SKILL_REVIEW_PROMPT` text:
|
|
|
|
```python
|
|
def on_pre_api_request(*, request_messages, session_id, model, **kw):
|
|
for msg in request_messages:
|
|
content = msg.get("content", "") or ""
|
|
if (
|
|
isinstance(content, str)
|
|
and "Review the conversation above and update the skill library" in content
|
|
):
|
|
# This is a background review fork!
|
|
turn_type = "background_review"
|
|
break
|
|
```
|
|
|
|
## Langfuse plugin reference
|
|
|
|
The canonical reference implementation is at:
|
|
|
|
```
|
|
~/.hermes/hermes-agent/plugins/observability/langfuse/__init__.py
|
|
```
|
|
|
|
This is Hermes Agent's **bundled** Langfuse observability plugin. Key sections:
|
|
|
|
| Lines | Content |
|
|
|-------|---------|
|
|
| 1-22 | Module docstring + env var docs |
|
|
| 43-56 | `TraceState` dataclass — trace tree state management |
|
|
| 712-750 | `on_pre_llm_call` — turn-scoped hook handler |
|
|
| 752-821 | `on_pre_llm_request` — **per-API-call hook handler (our model)** |
|
|
| 824-949 | `on_post_llm_call` — **per-API-call post hook handler (our model)** |
|
|
| 952-976 | `on_pre_tool_call` — per-tool pre hook |
|
|
| 978-1023 | `on_post_tool_call` — per-tool post hook |
|
|
| 1026-1035 | `register()` — hook registration |
|
|
|
|
Note: the Langfuse plugin has the naming confusing where `on_pre_llm_request` (per call) and `on_pre_llm_call` (per turn) are different things. We should follow the naming more clearly for hermes-trace.
|
|
|
|
## Hermes plugin structure (hermes-trace)
|
|
|
|
```
|
|
~/.hermes/plugins/hermes-trace/
|
|
├── plugin.yaml # Manifest: name, version, description
|
|
├── PLAN.md # Design document
|
|
├── HERMES_PLUGIN_RESEARCH.md # this file
|
|
├── __init__.py # register(ctx): wire hooks
|
|
├── db.py # SQLite schema + write/read
|
|
├── app.py # Granian + FastAPI + Jinja2 frontend
|
|
└── templates/ # Jinja2 HTML templates
|
|
```
|
|
|
|
## Plugin lifecycle notes
|
|
|
|
- `register(ctx)` is called exactly once at Hermes startup (gateway process or CLI).
|
|
- Daemon threads (`threading.Thread(daemon=True)`) spawned inside `register()` live as long as the Hermes process.
|
|
- This is safe for: background cleanup loops, the Granian frontend server, or any async housekeeping.
|
|
- If the plugin fails to load (exception in `register()`), Hermes logs a warning and continues; the plugin is skipped.
|
|
|
|
## Source files consulted
|
|
|
|
| File | Lines | Content |
|
|
|------|-------|---------|
|
|
| `plugins/observability/langfuse/__init__.py` | 1-1035 | **Bundled Langfuse plugin — our reference implementation** |
|
|
| `plugins/observability/nemo_relay/__init__.py` | 1-500+ | Another observability plugin using same hooks |
|
|
| `agent/conversation_loop.py` | 1008-1058 | `pre_api_request` hook call site |
|
|
| `agent/conversation_loop.py` | 3545-3588 | `post_api_request` hook call site |
|
|
| `hermes_cli/plugins.py` | 290-550 | `PluginContext` class — all `ctx.register_*` methods |
|
|
| `hermes_cli/plugins.py` | 997-1030 | `register_hook` implementation |
|
|
| `agent/background_review.py` | 428-710 | Fork mechanism |
|
|
| `agent/turn_finalizer.py` | 375-401 | Review trigger |
|
|
| `website/docs/guides/build-a-hermes-plugin.md` | 1-1254 | Official plugin guide |
|
|
|
|
## Frontend Research (added 2026-06-18)
|
|
|
|
See AGENTS.md for current frontend architecture (flat-table + overlay trace viewer with HTMX + Alpine patterns).
|