258 lines
13 KiB
Markdown
258 lines
13 KiB
Markdown
# hermes-trace — Local LLM Turn Telemetry
|
|
|
|
> Capture every Hermes LLM API call (main session, background review forks, subagents) to a local SQLite store, with a lightweight queryable frontend.
|
|
|
|
## Problem
|
|
|
|
- Background review fork (平行玖璃) conversations are **volatile** — the fork's LLM turns exist only in memory during `run_conversation`, then the thread dies and the conversation is lost.
|
|
- `state.db` does not store fork messages (`_session_db is None` on the fork agent).
|
|
- JSON session files (`sessions/session_*.json`) were the only persistent record, but the current code makes the fork share the parent's `session_id` and the size guard blocks JSON overwrite.
|
|
- Langfuse free tier (50k events/month) is exhausted; self-hosting Langfuse is too heavy (several GB + PostgreSQL + Node.js).
|
|
- No way to inspect the fork's **reasoning**, **context**, or **decision process** after the fact.
|
|
|
|
## Approach
|
|
|
|
### Capture layer: Hermes plugin using official hooks
|
|
|
|
Use three **official, documented Hermes plugin hooks**:
|
|
|
|
| Hook | Fires | What we capture |
|
|
|------|-------|-----------------|
|
|
| `pre_api_request` | Every LLM API call, before call | Messages sent, model, approx tokens, metadata |
|
|
| `post_api_request` | Every LLM API call, after response | Response content, token usage, latency, finish_reason, reasoning |
|
|
| `post_tool_call` | Every tool result | Tool name, arguments, result, duration_ms |
|
|
|
|
`pre_tool_call` is not implemented: `post_tool_call` already provides the tool name and arguments along with the result and duration, so a separate pre-hook is unnecessary.
|
|
|
|
**These hooks fire per individual API call** (including retries and interim calls inside the tool loop), not per user turn. The Langfuse plugin uses the same mechanism — confirmed in source:
|
|
|
|
```python
|
|
# ~/.hermes/hermes-agent/plugins/observability/langfuse/__init__.py L1026-1035
|
|
def register(ctx) -> None:
|
|
ctx.register_hook("pre_api_request", on_pre_llm_request)
|
|
ctx.register_hook("post_api_request", on_post_llm_call)
|
|
...
|
|
```
|
|
|
|
**Capture scope: full payload always.** `full_request` and `full_response` JSON are stored for every API call. No conditional toggle — if storage is a concern, lower the retention period.
|
|
|
|
### Storage layer: Local SQLite, two tables
|
|
|
|
```sql
|
|
CREATE TABLE turns (
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
session_id TEXT NOT NULL,
|
|
|
|
-- Call metadata
|
|
model TEXT,
|
|
turn_type TEXT DEFAULT 'main', -- reserved for future classification
|
|
api_call_count INTEGER, -- which call in this turn (0, 1, 2...)
|
|
|
|
-- Previews (for fast listing)
|
|
request_preview TEXT, -- first 300 chars of user/request
|
|
response_preview TEXT, -- first 300 chars of assistant response
|
|
reasoning TEXT, -- model's reasoning/thinking text
|
|
finish_reason TEXT, -- "stop", "tool_calls", "length", ...
|
|
|
|
-- Token usage (from post_api_request `usage` dict)
|
|
input_tokens INTEGER,
|
|
output_tokens INTEGER,
|
|
total_tokens INTEGER,
|
|
cache_read_tokens INTEGER,
|
|
cache_write_tokens INTEGER,
|
|
reasoning_tokens INTEGER,
|
|
|
|
-- Performance
|
|
elapsed_ms INTEGER,
|
|
tool_calls_count INTEGER DEFAULT 0,
|
|
|
|
-- Full payload (always stored)
|
|
full_request TEXT, -- JSON: full messages sent to the LLM
|
|
full_response TEXT, -- JSON: full response from the LLM
|
|
|
|
-- Timestamp
|
|
created_at TEXT DEFAULT (datetime('now'))
|
|
);
|
|
|
|
CREATE TABLE tool_calls (
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
turn_id INTEGER NOT NULL REFERENCES turns(id) ON DELETE CASCADE,
|
|
call_order INTEGER, -- order within the turn
|
|
name TEXT NOT NULL, -- tool name
|
|
args TEXT, -- full arguments JSON
|
|
result TEXT, -- full result JSON
|
|
duration_ms INTEGER,
|
|
error TEXT, -- non-null if tool raised an error
|
|
created_at TEXT DEFAULT (datetime('now'))
|
|
);
|
|
|
|
CREATE INDEX idx_turns_session_created ON turns(session_id, created_at DESC, id DESC);
|
|
CREATE INDEX idx_toolcalls_turn ON tool_calls(turn_id);
|
|
```
|
|
|
|
No FTS5 index for MVP — full-text search is deferred.
|
|
|
|
### Retention & cleanup
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| Retention | **N days**, configurable via `HERMES_TRACE_RETENTION_DAYS` env var |
|
|
| Default | **7 days** |
|
|
| Size limit | None (retention is the only guard) |
|
|
| Cleanup mechanism | **Non-blocking daemon thread**, spawned in `register()` |
|
|
| Cleanup interval | Every hour; deletes `WHERE created_at < datetime('now', '-N days')` |
|
|
|
|
### Frontend: Granian auto-spawned by plugin
|
|
|
|
The plugin spawns a **Granian daemon thread** inside `register()`. No manual start needed.
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| Server | Granian (RSGI — `granian --interface asgi`) |
|
|
| Framework | FastAPI + Jinja2 templates |
|
|
| Frontend | HTMX (server round-trips) + Alpine.js (client UI state) |
|
|
| Port | **7788**, configurable via `HERMES_TRACE_PORT` env var |
|
|
| Auto-start | ✅ Yes — daemon thread in `register()` |
|
|
| Zero build step | ✅ Pure server-rendered HTML |
|
|
|
|
### MVP pages (6 routes)
|
|
|
|
| Route | Type | Content |
|
|
|-------|------|---------|
|
|
| `GET /` | Full page | Session list, sorted by recency; each row shows session_id, turn count, last activity, model |
|
|
| `GET /session/{id}` | Full page | Flat turn timeline table; clicking a row opens the right-side detail overlay |
|
|
| `GET /session/{id}/waterfall/{turn_id}` | HTMX fragment | Legacy waterfall tree fragment with no live navigation path in the UI |
|
|
| `GET /turn/{id}` | HTMX fragment | Full turn detail with metadata badges, user input/response, and technical details in the overlay |
|
|
| `GET /turn/{id}?focus_tc=N` | HTMX fragment | Same as above, auto-scrolls to tool call #N (for multi-tool-click in waterfall) |
|
|
| `GET /turn/{id}/payload?type=request\|response` | HTMX fragment | Lazy-loaded full JSON content in `<textarea readonly>` (crash-safe for large payloads) |
|
|
|
|
Deferred: full-text search, statistics/analytics, auto-refresh, turn type filtering.
|
|
|
|
## Design Decisions (grill, 2026-06-18)
|
|
|
|
| # | Decision | Conclusion |
|
|
|---|----------|-----------|
|
|
| 1 | Capture scope | **Full payload always** — full_request, full_response, tool call args + result all stored unconditionally |
|
|
| 2 | Retention | **Configurable N days** (`retention_days`), default **7 days**; no size limit |
|
|
| 3 | Cleanup timing | **Async daemon thread** — spawned in `register()`, runs every hour, deletes expired rows |
|
|
| 4 | Turn type classification | **Deferred** — not needed for MVP, all turns stored indiscriminately |
|
|
| 5 | Frontend startup | **Plugin auto-spawn** Granian in a daemon thread at register time |
|
|
| 6 | Frontend port | **7788**, configurable via `plugins_config` |
|
|
| 7 | DB schema | **Two tables**: `turns` (per-API-call) + `tool_calls` (per-tool-invocation), FK cascade |
|
|
| 8 | MVP scope | 3 pages: session list → turn timeline → turn detail (HTMX expandable) |
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌────────────────────────────────────────────┐
|
|
│ Hermes Plugin (hermes-trace) │
|
|
│ pre/post_api_request hooks │
|
|
│ post_tool_call hook │
|
|
│ (writes every LLM API call to telemetry) │
|
|
│ + cleanup daemon thread (every hour) │
|
|
│ + Granian daemon thread (:7788) │
|
|
└────────────┬───────────────────────────────┘
|
|
│ writes & reads
|
|
┌────────────▼───────────────────────────────┐
|
|
│ $HERMES_HOME/telemetry/telemetry.db │
|
|
│ Tables: turns, tool_calls │
|
|
└────────────────────────────────────────────┘
|
|
│ reads
|
|
┌────────────▼───────────────────────────────┐
|
|
│ Granian → FastAPI + Jinja2 templates │
|
|
│ Routes: /, /session/{id}, /session/{id}/waterfall/{turn_id}, /turn/{id}, /turn/{id}/payload │
|
|
└────────────┬───────────────────────────────┘
|
|
│ serves HTML
|
|
┌────────────▼───────────────────────────────┐
|
|
│ Browser (HTMX + Alpine.js) │
|
|
│ - HTMX: server round-trips (list, nav) │
|
|
│ - Alpine.js: client UI state (expand) │
|
|
└────────────────────────────────────────────┘
|
|
```
|
|
|
|
## Configuration
|
|
|
|
Configuration is read from **environment variables** (not `plugins_config`):
|
|
|
|
| Env var | Default | Description |
|
|
|---------|---------|-------------|
|
|
| `HERMES_TRACE_DB_PATH` | `$HERMES_HOME/telemetry/telemetry.db` | SQLite database path |
|
|
| `HERMES_TRACE_HOST` | `127.0.0.1` | Granian bind address |
|
|
| `HERMES_TRACE_PORT` | `7788` | Frontend server port |
|
|
| `HERMES_TRACE_RETENTION_DAYS` | `7` | Days to keep turn data |
|
|
|
|
In `~/.hermes/config.yaml`, the plugin can be enabled/disabled but runtime config uses env vars:
|
|
|
|
```yaml
|
|
plugins:
|
|
enabled:
|
|
- hermes-trace
|
|
```
|
|
|
|
## Reference: bundled Langfuse plugin
|
|
|
|
The canonical reference for this approach is Hermes Agent's bundled Langfuse observability plugin:
|
|
|
|
**Location:** `~/.hermes/hermes-agent/plugins/observability/langfuse/__init__.py`
|
|
|
|
| Lines | What to look at |
|
|
|-------|-----------------|
|
|
| 752-821 | `on_pre_llm_request` — per-API-call pre-hook handler |
|
|
| 824-949 | `on_post_llm_call` — per-API-call post-hook handler (note: confusing naming) |
|
|
| 952-976 | `on_pre_tool_call` — bundled Langfuse pre-hook; not implemented in hermes-trace because `post_tool_call` includes the arguments |
|
|
| 978-1023 | `on_post_tool_call` — tool result tracking |
|
|
| 1026-1035 | `register()` — how hooks are wired |
|
|
| 43-56 | `TraceState` — state management across hooks |
|
|
| 503-562 | `_usage_and_cost` — token/cost extraction |
|
|
| 432-444 | `_coerce_request_messages` — message normalization |
|
|
|
|
Also useful: `plugins/observability/nemo_relay/__init__.py` — another observability plugin using the same hooks.
|
|
|
|
## Files
|
|
|
|
```
|
|
hermes-trace/
|
|
├── .gitignore # Python project ignores
|
|
├── plugin.yaml # Hermes plugin manifest
|
|
├── PLAN.md # this file
|
|
├── HERMES_PLUGIN_RESEARCH.md # Detailed research notes
|
|
├── __init__.py # register(ctx): wire hooks + spawn daemons
|
|
├── db.py # SQLite schema + write/read abstraction
|
|
├── app.py # Granian + FastAPI + Jinja2 frontend
|
|
├── templates/
|
|
│ ├── layout.html # Base layout + all CSS + HTMX/Alpine CDN
|
|
│ ├── index.html # Session list page
|
|
│ ├── session.html # Flat timeline table + right-side overlay
|
|
│ ├── waterfall.html # Legacy unlinked waterfall tree fragment
|
|
│ ├── turn_detail.html # Turn detail fragment (overlay panel)
|
|
│ └── payload.html # Lazy-loaded JSON textarea fragment
|
|
```
|
|
|
|
## Implementation Order
|
|
|
|
1. **`db.py`** — SQLite schema (`turns` + `tool_calls`), write/read helpers, cleanup query
|
|
2. **`__init__.py`** — `register()`: wire 3 hooks + spawn cleanup daemon + spawn Granian subprocess
|
|
3. **`app.py`** — FastAPI app, Jinja2 templates, 6 routes
|
|
4. **`templates/`** — layout, index, session, waterfall, turn_detail, payload
|
|
5. **Test** — with real fork scenarios
|
|
|
|
## Status
|
|
|
|
- [x] Investigate background review fork persistence gap
|
|
- [x] Research correct capture mechanism: `pre_api_request` / `post_api_request` hooks
|
|
- [x] Read Langfuse bundled plugin as reference implementation
|
|
- [x] Decide frontend: HTMX + Alpine.js + Jinja2
|
|
- [x] Design decisions locked (grill, 2026-06-18)
|
|
- [x] Document all findings in HERMES_PLUGIN_RESEARCH.md
|
|
- [x] Git init + .gitignore
|
|
- [x] Write plugin DB schema (`db.py`)
|
|
- [x] Write plugin hooks (`__init__.py`)
|
|
- [x] Write Granian frontend (`app.py` + templates: layout, index, session, waterfall, turn_detail, payload)
|
|
- [x] Flat-table + overlay trace viewer: timeline table with a right-side resizable detail panel
|
|
- [x] Crash-safe large JSON display (`<textarea readonly>` instead of `<pre>`)
|
|
- [x] Token usage breakdown tooltip (Langfuse-style)
|
|
- [x] Table row → right-side overlay detail navigation
|
|
- [x] JSON pretty/raw toggle on tool call args/result and Full Request/Response
|
|
- [ ] Full-text search across turns
|
|
- [ ] Statistics / analytics dashboard
|
|
- [ ] Turn type filtering (main / background_review / subagent)
|