hermes-trace/PLAN.md

258 lines
13 KiB
Markdown

# hermes-trace — Local LLM Turn Telemetry
> Capture every Hermes LLM API call (main session, background review forks, subagents) to a local SQLite store, with a lightweight queryable frontend.
## Problem
- Background review fork (平行玖璃) conversations are **volatile** — the fork's LLM turns exist only in memory during `run_conversation`, then the thread dies and the conversation is lost.
- `state.db` does not store fork messages (`_session_db is None` on the fork agent).
- JSON session files (`sessions/session_*.json`) were the only persistent record, but the current code makes the fork share the parent's `session_id` and the size guard blocks JSON overwrite.
- Langfuse free tier (50k events/month) is exhausted; self-hosting Langfuse is too heavy (several GB + PostgreSQL + Node.js).
- No way to inspect the fork's **reasoning**, **context**, or **decision process** after the fact.
## Approach
### Capture layer: Hermes plugin using official hooks
Use three **official, documented Hermes plugin hooks**:
| Hook | Fires | What we capture |
|------|-------|-----------------|
| `pre_api_request` | Every LLM API call, before call | Messages sent, model, approx tokens, metadata |
| `post_api_request` | Every LLM API call, after response | Response content, token usage, latency, finish_reason, reasoning |
| `post_tool_call` | Every tool result | Tool name, arguments, result, duration_ms |
`pre_tool_call` is not implemented: `post_tool_call` already provides the tool name and arguments along with the result and duration, so a separate pre-hook is unnecessary.
**These hooks fire per individual API call** (including retries and interim calls inside the tool loop), not per user turn. The Langfuse plugin uses the same mechanism — confirmed in source:
```python
# ~/.hermes/hermes-agent/plugins/observability/langfuse/__init__.py L1026-1035
def register(ctx) -> None:
ctx.register_hook("pre_api_request", on_pre_llm_request)
ctx.register_hook("post_api_request", on_post_llm_call)
...
```
**Capture scope: full payload always.** `full_request` and `full_response` JSON are stored for every API call. No conditional toggle — if storage is a concern, lower the retention period.
### Storage layer: Local SQLite, two tables
```sql
CREATE TABLE turns (
id INTEGER PRIMARY KEY AUTOINCREMENT,
session_id TEXT NOT NULL,
-- Call metadata
model TEXT,
turn_type TEXT DEFAULT 'main', -- reserved for future classification
api_call_count INTEGER, -- which call in this turn (0, 1, 2...)
-- Previews (for fast listing)
request_preview TEXT, -- first 300 chars of user/request
response_preview TEXT, -- first 300 chars of assistant response
reasoning TEXT, -- model's reasoning/thinking text
finish_reason TEXT, -- "stop", "tool_calls", "length", ...
-- Token usage (from post_api_request `usage` dict)
input_tokens INTEGER,
output_tokens INTEGER,
total_tokens INTEGER,
cache_read_tokens INTEGER,
cache_write_tokens INTEGER,
reasoning_tokens INTEGER,
-- Performance
elapsed_ms INTEGER,
tool_calls_count INTEGER DEFAULT 0,
-- Full payload (always stored)
full_request TEXT, -- JSON: full messages sent to the LLM
full_response TEXT, -- JSON: full response from the LLM
-- Timestamp
created_at TEXT DEFAULT (datetime('now'))
);
CREATE TABLE tool_calls (
id INTEGER PRIMARY KEY AUTOINCREMENT,
turn_id INTEGER NOT NULL REFERENCES turns(id) ON DELETE CASCADE,
call_order INTEGER, -- order within the turn
name TEXT NOT NULL, -- tool name
args TEXT, -- full arguments JSON
result TEXT, -- full result JSON
duration_ms INTEGER,
error TEXT, -- non-null if tool raised an error
created_at TEXT DEFAULT (datetime('now'))
);
CREATE INDEX idx_turns_session_created ON turns(session_id, created_at DESC, id DESC);
CREATE INDEX idx_toolcalls_turn ON tool_calls(turn_id);
```
No FTS5 index for MVP — full-text search is deferred.
### Retention & cleanup
| Property | Value |
|----------|-------|
| Retention | **N days**, configurable via `HERMES_TRACE_RETENTION_DAYS` env var |
| Default | **7 days** |
| Size limit | None (retention is the only guard) |
| Cleanup mechanism | **Non-blocking daemon thread**, spawned in `register()` |
| Cleanup interval | Every hour; deletes `WHERE created_at < datetime('now', '-N days')` |
### Frontend: Granian auto-spawned by plugin
The plugin spawns a **Granian daemon thread** inside `register()`. No manual start needed.
| Property | Value |
|----------|-------|
| Server | Granian (RSGI — `granian --interface asgi`) |
| Framework | FastAPI + Jinja2 templates |
| Frontend | HTMX (server round-trips) + Alpine.js (client UI state) |
| Port | **7788**, configurable via `HERMES_TRACE_PORT` env var |
| Auto-start | ✅ Yes — daemon thread in `register()` |
| Zero build step | ✅ Pure server-rendered HTML |
### MVP pages (6 routes)
| Route | Type | Content |
|-------|------|---------|
| `GET /` | Full page | Session list, sorted by recency; each row shows session_id, turn count, last activity, model |
| `GET /session/{id}` | Full page | Flat turn timeline table; clicking a row opens the right-side detail overlay |
| `GET /session/{id}/waterfall/{turn_id}` | HTMX fragment | Legacy waterfall tree fragment with no live navigation path in the UI |
| `GET /turn/{id}` | HTMX fragment | Full turn detail with metadata badges, user input/response, and technical details in the overlay |
| `GET /turn/{id}?focus_tc=N` | HTMX fragment | Same as above, auto-scrolls to tool call #N (for multi-tool-click in waterfall) |
| `GET /turn/{id}/payload?type=request\|response` | HTMX fragment | Lazy-loaded full JSON content in `<textarea readonly>` (crash-safe for large payloads) |
Deferred: full-text search, statistics/analytics, auto-refresh, turn type filtering.
## Design Decisions (grill, 2026-06-18)
| # | Decision | Conclusion |
|---|----------|-----------|
| 1 | Capture scope | **Full payload always** — full_request, full_response, tool call args + result all stored unconditionally |
| 2 | Retention | **Configurable N days** (`retention_days`), default **7 days**; no size limit |
| 3 | Cleanup timing | **Async daemon thread** — spawned in `register()`, runs every hour, deletes expired rows |
| 4 | Turn type classification | **Deferred** — not needed for MVP, all turns stored indiscriminately |
| 5 | Frontend startup | **Plugin auto-spawn** Granian in a daemon thread at register time |
| 6 | Frontend port | **7788**, configurable via `plugins_config` |
| 7 | DB schema | **Two tables**: `turns` (per-API-call) + `tool_calls` (per-tool-invocation), FK cascade |
| 8 | MVP scope | 3 pages: session list → turn timeline → turn detail (HTMX expandable) |
## Architecture
```
┌────────────────────────────────────────────┐
│ Hermes Plugin (hermes-trace) │
│ pre/post_api_request hooks │
│ post_tool_call hook │
│ (writes every LLM API call to telemetry) │
│ + cleanup daemon thread (every hour) │
│ + Granian daemon thread (:7788) │
└────────────┬───────────────────────────────┘
│ writes & reads
┌────────────▼───────────────────────────────┐
│ $HERMES_HOME/telemetry/telemetry.db │
│ Tables: turns, tool_calls │
└────────────────────────────────────────────┘
│ reads
┌────────────▼───────────────────────────────┐
│ Granian → FastAPI + Jinja2 templates │
│ Routes: /, /session/{id}, /session/{id}/waterfall/{turn_id}, /turn/{id}, /turn/{id}/payload │
└────────────┬───────────────────────────────┘
│ serves HTML
┌────────────▼───────────────────────────────┐
│ Browser (HTMX + Alpine.js) │
│ - HTMX: server round-trips (list, nav) │
│ - Alpine.js: client UI state (expand) │
└────────────────────────────────────────────┘
```
## Configuration
Configuration is read from **environment variables** (not `plugins_config`):
| Env var | Default | Description |
|---------|---------|-------------|
| `HERMES_TRACE_DB_PATH` | `$HERMES_HOME/telemetry/telemetry.db` | SQLite database path |
| `HERMES_TRACE_HOST` | `127.0.0.1` | Granian bind address |
| `HERMES_TRACE_PORT` | `7788` | Frontend server port |
| `HERMES_TRACE_RETENTION_DAYS` | `7` | Days to keep turn data |
In `~/.hermes/config.yaml`, the plugin can be enabled/disabled but runtime config uses env vars:
```yaml
plugins:
enabled:
- hermes-trace
```
## Reference: bundled Langfuse plugin
The canonical reference for this approach is Hermes Agent's bundled Langfuse observability plugin:
**Location:** `~/.hermes/hermes-agent/plugins/observability/langfuse/__init__.py`
| Lines | What to look at |
|-------|-----------------|
| 752-821 | `on_pre_llm_request` — per-API-call pre-hook handler |
| 824-949 | `on_post_llm_call` — per-API-call post-hook handler (note: confusing naming) |
| 952-976 | `on_pre_tool_call` — bundled Langfuse pre-hook; not implemented in hermes-trace because `post_tool_call` includes the arguments |
| 978-1023 | `on_post_tool_call` — tool result tracking |
| 1026-1035 | `register()` — how hooks are wired |
| 43-56 | `TraceState` — state management across hooks |
| 503-562 | `_usage_and_cost` — token/cost extraction |
| 432-444 | `_coerce_request_messages` — message normalization |
Also useful: `plugins/observability/nemo_relay/__init__.py` — another observability plugin using the same hooks.
## Files
```
hermes-trace/
├── .gitignore # Python project ignores
├── plugin.yaml # Hermes plugin manifest
├── PLAN.md # this file
├── HERMES_PLUGIN_RESEARCH.md # Detailed research notes
├── __init__.py # register(ctx): wire hooks + spawn daemons
├── db.py # SQLite schema + write/read abstraction
├── app.py # Granian + FastAPI + Jinja2 frontend
├── templates/
│ ├── layout.html # Base layout + all CSS + HTMX/Alpine CDN
│ ├── index.html # Session list page
│ ├── session.html # Flat timeline table + right-side overlay
│ ├── waterfall.html # Legacy unlinked waterfall tree fragment
│ ├── turn_detail.html # Turn detail fragment (overlay panel)
│ └── payload.html # Lazy-loaded JSON textarea fragment
```
## Implementation Order
1. **`db.py`** — SQLite schema (`turns` + `tool_calls`), write/read helpers, cleanup query
2. **`__init__.py`** — `register()`: wire 3 hooks + spawn cleanup daemon + spawn Granian subprocess
3. **`app.py`** — FastAPI app, Jinja2 templates, 6 routes
4. **`templates/`** — layout, index, session, waterfall, turn_detail, payload
5. **Test** — with real fork scenarios
## Status
- [x] Investigate background review fork persistence gap
- [x] Research correct capture mechanism: `pre_api_request` / `post_api_request` hooks
- [x] Read Langfuse bundled plugin as reference implementation
- [x] Decide frontend: HTMX + Alpine.js + Jinja2
- [x] Design decisions locked (grill, 2026-06-18)
- [x] Document all findings in HERMES_PLUGIN_RESEARCH.md
- [x] Git init + .gitignore
- [x] Write plugin DB schema (`db.py`)
- [x] Write plugin hooks (`__init__.py`)
- [x] Write Granian frontend (`app.py` + templates: layout, index, session, waterfall, turn_detail, payload)
- [x] Flat-table + overlay trace viewer: timeline table with a right-side resizable detail panel
- [x] Crash-safe large JSON display (`<textarea readonly>` instead of `<pre>`)
- [x] Token usage breakdown tooltip (Langfuse-style)
- [x] Table row → right-side overlay detail navigation
- [x] JSON pretty/raw toggle on tool call args/result and Full Request/Response
- [ ] Full-text search across turns
- [ ] Statistics / analytics dashboard
- [ ] Turn type filtering (main / background_review / subagent)