hermes-trace/PLAN.md

13 KiB

hermes-trace — Local LLM Turn Telemetry

Capture every Hermes LLM API call (main session, background review forks, subagents) to a local SQLite store, with a lightweight queryable frontend.

Problem

  • Background review fork (平行玖璃) conversations are volatile — the fork's LLM turns exist only in memory during run_conversation, then the thread dies and the conversation is lost.
  • state.db does not store fork messages (_session_db is None on the fork agent).
  • JSON session files (sessions/session_*.json) were the only persistent record, but the current code makes the fork share the parent's session_id and the size guard blocks JSON overwrite.
  • Langfuse free tier (50k events/month) is exhausted; self-hosting Langfuse is too heavy (several GB + PostgreSQL + Node.js).
  • No way to inspect the fork's reasoning, context, or decision process after the fact.

Approach

Capture layer: Hermes plugin using official hooks

Use three official, documented Hermes plugin hooks:

Hook Fires What we capture
pre_api_request Every LLM API call, before call Messages sent, model, approx tokens, metadata
post_api_request Every LLM API call, after response Response content, token usage, latency, finish_reason, reasoning
post_tool_call Every tool result Tool name, arguments, result, duration_ms

pre_tool_call is not implemented: post_tool_call already provides the tool name and arguments along with the result and duration, so a separate pre-hook is unnecessary.

These hooks fire per individual API call (including retries and interim calls inside the tool loop), not per user turn. The Langfuse plugin uses the same mechanism — confirmed in source:

# ~/.hermes/hermes-agent/plugins/observability/langfuse/__init__.py L1026-1035
def register(ctx) -> None:
    ctx.register_hook("pre_api_request", on_pre_llm_request)
    ctx.register_hook("post_api_request", on_post_llm_call)
    ...

Capture scope: full payload always. full_request and full_response JSON are stored for every API call. No conditional toggle — if storage is a concern, lower the retention period.

Storage layer: Local SQLite, two tables

CREATE TABLE turns (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    session_id TEXT NOT NULL,

    -- Call metadata
    model TEXT,
    turn_type TEXT DEFAULT 'main',          -- reserved for future classification
    api_call_count INTEGER,                 -- which call in this turn (0, 1, 2...)

    -- Previews (for fast listing)
    request_preview TEXT,                   -- first 300 chars of user/request
    response_preview TEXT,                  -- first 300 chars of assistant response
    reasoning TEXT,                         -- model's reasoning/thinking text
    finish_reason TEXT,                     -- "stop", "tool_calls", "length", ...

    -- Token usage (from post_api_request `usage` dict)
    input_tokens INTEGER,
    output_tokens INTEGER,
    total_tokens INTEGER,
    cache_read_tokens INTEGER,
    cache_write_tokens INTEGER,
    reasoning_tokens INTEGER,

    -- Performance
    elapsed_ms INTEGER,
    tool_calls_count INTEGER DEFAULT 0,

    -- Full payload (always stored)
    full_request TEXT,                      -- JSON: full messages sent to the LLM
    full_response TEXT,                     -- JSON: full response from the LLM

    -- Timestamp
    created_at TEXT DEFAULT (datetime('now'))
);

CREATE TABLE tool_calls (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    turn_id INTEGER NOT NULL REFERENCES turns(id) ON DELETE CASCADE,
    call_order INTEGER,                     -- order within the turn
    name TEXT NOT NULL,                     -- tool name
    args TEXT,                              -- full arguments JSON
    result TEXT,                            -- full result JSON
    duration_ms INTEGER,
    error TEXT,                             -- non-null if tool raised an error
    created_at TEXT DEFAULT (datetime('now'))
);

CREATE INDEX idx_turns_session_created ON turns(session_id, created_at DESC, id DESC);
CREATE INDEX idx_toolcalls_turn ON tool_calls(turn_id);

No FTS5 index for MVP — full-text search is deferred.

Retention & cleanup

Property Value
Retention N days, configurable via HERMES_TRACE_RETENTION_DAYS env var
Default 7 days
Size limit None (retention is the only guard)
Cleanup mechanism Non-blocking daemon thread, spawned in register()
Cleanup interval Every hour; deletes WHERE created_at < datetime('now', '-N days')

Frontend: Granian auto-spawned by plugin

The plugin spawns a Granian daemon thread inside register(). No manual start needed.

Property Value
Server Granian (RSGI — granian --interface asgi)
Framework FastAPI + Jinja2 templates
Frontend HTMX (server round-trips) + Alpine.js (client UI state)
Port 7788, configurable via HERMES_TRACE_PORT env var
Auto-start ✅ Yes — daemon thread in register()
Zero build step ✅ Pure server-rendered HTML

MVP pages (6 routes)

Route Type Content
GET / Full page Session list, sorted by recency; each row shows session_id, turn count, last activity, model
GET /session/{id} Full page Flat turn timeline table; clicking a row opens the right-side detail overlay
GET /session/{id}/waterfall/{turn_id} HTMX fragment Legacy waterfall tree fragment with no live navigation path in the UI
GET /turn/{id} HTMX fragment Full turn detail with metadata badges, user input/response, and technical details in the overlay
GET /turn/{id}?focus_tc=N HTMX fragment Same as above, auto-scrolls to tool call #N (for multi-tool-click in waterfall)
GET /turn/{id}/payload?type=request|response HTMX fragment Lazy-loaded full JSON content in <textarea readonly> (crash-safe for large payloads)

Deferred: full-text search, statistics/analytics, auto-refresh, turn type filtering.

Design Decisions (grill, 2026-06-18)

# Decision Conclusion
1 Capture scope Full payload always — full_request, full_response, tool call args + result all stored unconditionally
2 Retention Configurable N days (retention_days), default 7 days; no size limit
3 Cleanup timing Async daemon thread — spawned in register(), runs every hour, deletes expired rows
4 Turn type classification Deferred — not needed for MVP, all turns stored indiscriminately
5 Frontend startup Plugin auto-spawn Granian in a daemon thread at register time
6 Frontend port 7788, configurable via plugins_config
7 DB schema Two tables: turns (per-API-call) + tool_calls (per-tool-invocation), FK cascade
8 MVP scope 3 pages: session list → turn timeline → turn detail (HTMX expandable)

Architecture

┌────────────────────────────────────────────┐
│  Hermes Plugin (hermes-trace)              │
│  pre/post_api_request hooks                │
│  post_tool_call hook                       │
│  (writes every LLM API call to telemetry)  │
│  + cleanup daemon thread (every hour)      │
│  + Granian daemon thread (:7788)           │
└────────────┬───────────────────────────────┘
             │ writes & reads
┌────────────▼───────────────────────────────┐
│  $HERMES_HOME/telemetry/telemetry.db          │
│  Tables: turns, tool_calls                 │
└────────────────────────────────────────────┘
             │ reads
┌────────────▼───────────────────────────────┐
│  Granian → FastAPI + Jinja2 templates      │
│  Routes: /, /session/{id}, /session/{id}/waterfall/{turn_id}, /turn/{id}, /turn/{id}/payload │
└────────────┬───────────────────────────────┘
             │ serves HTML
┌────────────▼───────────────────────────────┐
│  Browser (HTMX + Alpine.js)                │
│  - HTMX: server round-trips (list, nav)    │
│  - Alpine.js: client UI state (expand)     │
└────────────────────────────────────────────┘

Configuration

Configuration is read from environment variables (not plugins_config):

Env var Default Description
HERMES_TRACE_DB_PATH $HERMES_HOME/telemetry/telemetry.db SQLite database path
HERMES_TRACE_HOST 127.0.0.1 Granian bind address
HERMES_TRACE_PORT 7788 Frontend server port
HERMES_TRACE_RETENTION_DAYS 7 Days to keep turn data

In ~/.hermes/config.yaml, the plugin can be enabled/disabled but runtime config uses env vars:

plugins:
  enabled:
    - hermes-trace

Reference: bundled Langfuse plugin

The canonical reference for this approach is Hermes Agent's bundled Langfuse observability plugin:

Location: ~/.hermes/hermes-agent/plugins/observability/langfuse/__init__.py

Lines What to look at
752-821 on_pre_llm_request — per-API-call pre-hook handler
824-949 on_post_llm_call — per-API-call post-hook handler (note: confusing naming)
952-976 on_pre_tool_call — bundled Langfuse pre-hook; not implemented in hermes-trace because post_tool_call includes the arguments
978-1023 on_post_tool_call — tool result tracking
1026-1035 register() — how hooks are wired
43-56 TraceState — state management across hooks
503-562 _usage_and_cost — token/cost extraction
432-444 _coerce_request_messages — message normalization

Also useful: plugins/observability/nemo_relay/__init__.py — another observability plugin using the same hooks.

Files

hermes-trace/
├── .gitignore                  # Python project ignores
├── plugin.yaml                 # Hermes plugin manifest
├── PLAN.md                     # this file
├── HERMES_PLUGIN_RESEARCH.md   # Detailed research notes
├── __init__.py                 # register(ctx): wire hooks + spawn daemons
├── db.py                       # SQLite schema + write/read abstraction
├── app.py                      # Granian + FastAPI + Jinja2 frontend
├── templates/
│   ├── layout.html              # Base layout + all CSS + HTMX/Alpine CDN
│   ├── index.html               # Session list page
│   ├── session.html             # Flat timeline table + right-side overlay
│   ├── waterfall.html           # Legacy unlinked waterfall tree fragment
│   ├── turn_detail.html         # Turn detail fragment (overlay panel)
│   └── payload.html             # Lazy-loaded JSON textarea fragment

Implementation Order

  1. db.py — SQLite schema (turns + tool_calls), write/read helpers, cleanup query
  2. __init__.py — register(): wire 3 hooks + spawn cleanup daemon + spawn Granian subprocess
  3. app.py — FastAPI app, Jinja2 templates, 6 routes
  4. templates/ — layout, index, session, waterfall, turn_detail, payload
  5. Test — with real fork scenarios

Status

  • Investigate background review fork persistence gap
  • Research correct capture mechanism: pre_api_request / post_api_request hooks
  • Read Langfuse bundled plugin as reference implementation
  • Decide frontend: HTMX + Alpine.js + Jinja2
  • Design decisions locked (grill, 2026-06-18)
  • Document all findings in HERMES_PLUGIN_RESEARCH.md
  • Git init + .gitignore
  • Write plugin DB schema (db.py)
  • Write plugin hooks (__init__.py)
  • Write Granian frontend (app.py + templates: layout, index, session, waterfall, turn_detail, payload)
  • Flat-table + overlay trace viewer: timeline table with a right-side resizable detail panel
  • Crash-safe large JSON display (<textarea readonly> instead of <pre>)
  • Token usage breakdown tooltip (Langfuse-style)
  • Table row → right-side overlay detail navigation
  • JSON pretty/raw toggle on tool call args/result and Full Request/Response
  • Full-text search across turns
  • Statistics / analytics dashboard
  • Turn type filtering (main / background_review / subagent)