13 KiB
hermes-trace — Local LLM Turn Telemetry
Capture every Hermes LLM API call (main session, background review forks, subagents) to a local SQLite store, with a lightweight queryable frontend.
Problem
- Background review fork (平行玖璃) conversations are volatile — the fork's LLM turns exist only in memory during
run_conversation, then the thread dies and the conversation is lost. state.dbdoes not store fork messages (_session_db is Noneon the fork agent).- JSON session files (
sessions/session_*.json) were the only persistent record, but the current code makes the fork share the parent'ssession_idand the size guard blocks JSON overwrite. - Langfuse free tier (50k events/month) is exhausted; self-hosting Langfuse is too heavy (several GB + PostgreSQL + Node.js).
- No way to inspect the fork's reasoning, context, or decision process after the fact.
Approach
Capture layer: Hermes plugin using official hooks
Use three official, documented Hermes plugin hooks:
| Hook | Fires | What we capture |
|---|---|---|
pre_api_request |
Every LLM API call, before call | Messages sent, model, approx tokens, metadata |
post_api_request |
Every LLM API call, after response | Response content, token usage, latency, finish_reason, reasoning |
post_tool_call |
Every tool result | Tool name, arguments, result, duration_ms |
pre_tool_call is not implemented: post_tool_call already provides the tool name and arguments along with the result and duration, so a separate pre-hook is unnecessary.
These hooks fire per individual API call (including retries and interim calls inside the tool loop), not per user turn. The Langfuse plugin uses the same mechanism — confirmed in source:
# ~/.hermes/hermes-agent/plugins/observability/langfuse/__init__.py L1026-1035
def register(ctx) -> None:
ctx.register_hook("pre_api_request", on_pre_llm_request)
ctx.register_hook("post_api_request", on_post_llm_call)
...
Capture scope: full payload always. full_request and full_response JSON are stored for every API call. No conditional toggle — if storage is a concern, lower the retention period.
Storage layer: Local SQLite, two tables
CREATE TABLE turns (
id INTEGER PRIMARY KEY AUTOINCREMENT,
session_id TEXT NOT NULL,
-- Call metadata
model TEXT,
turn_type TEXT DEFAULT 'main', -- reserved for future classification
api_call_count INTEGER, -- which call in this turn (0, 1, 2...)
-- Previews (for fast listing)
request_preview TEXT, -- first 300 chars of user/request
response_preview TEXT, -- first 300 chars of assistant response
reasoning TEXT, -- model's reasoning/thinking text
finish_reason TEXT, -- "stop", "tool_calls", "length", ...
-- Token usage (from post_api_request `usage` dict)
input_tokens INTEGER,
output_tokens INTEGER,
total_tokens INTEGER,
cache_read_tokens INTEGER,
cache_write_tokens INTEGER,
reasoning_tokens INTEGER,
-- Performance
elapsed_ms INTEGER,
tool_calls_count INTEGER DEFAULT 0,
-- Full payload (always stored)
full_request TEXT, -- JSON: full messages sent to the LLM
full_response TEXT, -- JSON: full response from the LLM
-- Timestamp
created_at TEXT DEFAULT (datetime('now'))
);
CREATE TABLE tool_calls (
id INTEGER PRIMARY KEY AUTOINCREMENT,
turn_id INTEGER NOT NULL REFERENCES turns(id) ON DELETE CASCADE,
call_order INTEGER, -- order within the turn
name TEXT NOT NULL, -- tool name
args TEXT, -- full arguments JSON
result TEXT, -- full result JSON
duration_ms INTEGER,
error TEXT, -- non-null if tool raised an error
created_at TEXT DEFAULT (datetime('now'))
);
CREATE INDEX idx_turns_session_created ON turns(session_id, created_at DESC, id DESC);
CREATE INDEX idx_toolcalls_turn ON tool_calls(turn_id);
No FTS5 index for MVP — full-text search is deferred.
Retention & cleanup
| Property | Value |
|---|---|
| Retention | N days, configurable via HERMES_TRACE_RETENTION_DAYS env var |
| Default | 7 days |
| Size limit | None (retention is the only guard) |
| Cleanup mechanism | Non-blocking daemon thread, spawned in register() |
| Cleanup interval | Every hour; deletes WHERE created_at < datetime('now', '-N days') |
Frontend: Granian auto-spawned by plugin
The plugin spawns a Granian daemon thread inside register(). No manual start needed.
| Property | Value |
|---|---|
| Server | Granian (RSGI — granian --interface asgi) |
| Framework | FastAPI + Jinja2 templates |
| Frontend | HTMX (server round-trips) + Alpine.js (client UI state) |
| Port | 7788, configurable via HERMES_TRACE_PORT env var |
| Auto-start | ✅ Yes — daemon thread in register() |
| Zero build step | ✅ Pure server-rendered HTML |
MVP pages (6 routes)
| Route | Type | Content |
|---|---|---|
GET / |
Full page | Session list, sorted by recency; each row shows session_id, turn count, last activity, model |
GET /session/{id} |
Full page | Flat turn timeline table; clicking a row opens the right-side detail overlay |
GET /session/{id}/waterfall/{turn_id} |
HTMX fragment | Legacy waterfall tree fragment with no live navigation path in the UI |
GET /turn/{id} |
HTMX fragment | Full turn detail with metadata badges, user input/response, and technical details in the overlay |
GET /turn/{id}?focus_tc=N |
HTMX fragment | Same as above, auto-scrolls to tool call #N (for multi-tool-click in waterfall) |
GET /turn/{id}/payload?type=request|response |
HTMX fragment | Lazy-loaded full JSON content in <textarea readonly> (crash-safe for large payloads) |
Deferred: full-text search, statistics/analytics, auto-refresh, turn type filtering.
Design Decisions (grill, 2026-06-18)
| # | Decision | Conclusion |
|---|---|---|
| 1 | Capture scope | Full payload always — full_request, full_response, tool call args + result all stored unconditionally |
| 2 | Retention | Configurable N days (retention_days), default 7 days; no size limit |
| 3 | Cleanup timing | Async daemon thread — spawned in register(), runs every hour, deletes expired rows |
| 4 | Turn type classification | Deferred — not needed for MVP, all turns stored indiscriminately |
| 5 | Frontend startup | Plugin auto-spawn Granian in a daemon thread at register time |
| 6 | Frontend port | 7788, configurable via plugins_config |
| 7 | DB schema | Two tables: turns (per-API-call) + tool_calls (per-tool-invocation), FK cascade |
| 8 | MVP scope | 3 pages: session list → turn timeline → turn detail (HTMX expandable) |
Architecture
┌────────────────────────────────────────────┐
│ Hermes Plugin (hermes-trace) │
│ pre/post_api_request hooks │
│ post_tool_call hook │
│ (writes every LLM API call to telemetry) │
│ + cleanup daemon thread (every hour) │
│ + Granian daemon thread (:7788) │
└────────────┬───────────────────────────────┘
│ writes & reads
┌────────────▼───────────────────────────────┐
│ $HERMES_HOME/telemetry/telemetry.db │
│ Tables: turns, tool_calls │
└────────────────────────────────────────────┘
│ reads
┌────────────▼───────────────────────────────┐
│ Granian → FastAPI + Jinja2 templates │
│ Routes: /, /session/{id}, /session/{id}/waterfall/{turn_id}, /turn/{id}, /turn/{id}/payload │
└────────────┬───────────────────────────────┘
│ serves HTML
┌────────────▼───────────────────────────────┐
│ Browser (HTMX + Alpine.js) │
│ - HTMX: server round-trips (list, nav) │
│ - Alpine.js: client UI state (expand) │
└────────────────────────────────────────────┘
Configuration
Configuration is read from environment variables (not plugins_config):
| Env var | Default | Description |
|---|---|---|
HERMES_TRACE_DB_PATH |
$HERMES_HOME/telemetry/telemetry.db |
SQLite database path |
HERMES_TRACE_HOST |
127.0.0.1 |
Granian bind address |
HERMES_TRACE_PORT |
7788 |
Frontend server port |
HERMES_TRACE_RETENTION_DAYS |
7 |
Days to keep turn data |
In ~/.hermes/config.yaml, the plugin can be enabled/disabled but runtime config uses env vars:
plugins:
enabled:
- hermes-trace
Reference: bundled Langfuse plugin
The canonical reference for this approach is Hermes Agent's bundled Langfuse observability plugin:
Location: ~/.hermes/hermes-agent/plugins/observability/langfuse/__init__.py
| Lines | What to look at |
|---|---|
| 752-821 | on_pre_llm_request — per-API-call pre-hook handler |
| 824-949 | on_post_llm_call — per-API-call post-hook handler (note: confusing naming) |
| 952-976 | on_pre_tool_call — bundled Langfuse pre-hook; not implemented in hermes-trace because post_tool_call includes the arguments |
| 978-1023 | on_post_tool_call — tool result tracking |
| 1026-1035 | register() — how hooks are wired |
| 43-56 | TraceState — state management across hooks |
| 503-562 | _usage_and_cost — token/cost extraction |
| 432-444 | _coerce_request_messages — message normalization |
Also useful: plugins/observability/nemo_relay/__init__.py — another observability plugin using the same hooks.
Files
hermes-trace/
├── .gitignore # Python project ignores
├── plugin.yaml # Hermes plugin manifest
├── PLAN.md # this file
├── HERMES_PLUGIN_RESEARCH.md # Detailed research notes
├── __init__.py # register(ctx): wire hooks + spawn daemons
├── db.py # SQLite schema + write/read abstraction
├── app.py # Granian + FastAPI + Jinja2 frontend
├── templates/
│ ├── layout.html # Base layout + all CSS + HTMX/Alpine CDN
│ ├── index.html # Session list page
│ ├── session.html # Flat timeline table + right-side overlay
│ ├── waterfall.html # Legacy unlinked waterfall tree fragment
│ ├── turn_detail.html # Turn detail fragment (overlay panel)
│ └── payload.html # Lazy-loaded JSON textarea fragment
Implementation Order
db.py— SQLite schema (turns+tool_calls), write/read helpers, cleanup query__init__.py—register(): wire 3 hooks + spawn cleanup daemon + spawn Granian subprocessapp.py— FastAPI app, Jinja2 templates, 6 routestemplates/— layout, index, session, waterfall, turn_detail, payload- Test — with real fork scenarios
Status
- Investigate background review fork persistence gap
- Research correct capture mechanism:
pre_api_request/post_api_requesthooks - Read Langfuse bundled plugin as reference implementation
- Decide frontend: HTMX + Alpine.js + Jinja2
- Design decisions locked (grill, 2026-06-18)
- Document all findings in HERMES_PLUGIN_RESEARCH.md
- Git init + .gitignore
- Write plugin DB schema (
db.py) - Write plugin hooks (
__init__.py) - Write Granian frontend (
app.py+ templates: layout, index, session, waterfall, turn_detail, payload) - Flat-table + overlay trace viewer: timeline table with a right-side resizable detail panel
- Crash-safe large JSON display (
<textarea readonly>instead of<pre>) - Token usage breakdown tooltip (Langfuse-style)
- Table row → right-side overlay detail navigation
- JSON pretty/raw toggle on tool call args/result and Full Request/Response
- Full-text search across turns
- Statistics / analytics dashboard
- Turn type filtering (main / background_review / subagent)