133 lines
5.2 KiB
Markdown
133 lines
5.2 KiB
Markdown
# hermes-trace Hook Implementation
|
|
|
|
## Capture Mechanism
|
|
|
|
Uses **3 official Hermes plugin hooks** — the same mechanism as the bundled Langfuse plugin.
|
|
|
|
### Hook Registration (in `__init__.py`)
|
|
|
|
```python
|
|
# __init__.py
|
|
import threading
|
|
|
|
from .hooks import on_post_api_request, on_post_tool_call, on_pre_api_request
|
|
from .launcher import _cleanup_loop, _run_frontend
|
|
|
|
|
|
def register(ctx):
|
|
# Configuration and database initialization happen before this wiring.
|
|
ctx.register_hook("pre_api_request", on_pre_api_request)
|
|
ctx.register_hook("post_api_request", on_post_api_request)
|
|
ctx.register_hook("post_tool_call", on_post_tool_call)
|
|
|
|
threading.Thread(
|
|
target=_cleanup_loop, daemon=True, name="hermes-trace-cleanup"
|
|
).start()
|
|
threading.Thread(
|
|
target=_run_frontend, daemon=True, name="hermes-trace-frontend"
|
|
).start()
|
|
```
|
|
|
|
The handlers are implemented in `hooks.py`; the daemon targets are provided by
|
|
`launcher.py`. `__init__.py` keeps `register()` and re-exports the package
|
|
surface used by the plugin and its tests.
|
|
|
|
### Why hooks, not middleware
|
|
|
|
| Aspect | Hooks (`pre/post_api_request`) | Middleware (`llm_execution`) |
|
|
|--------|-------------------------------|------------------------------|
|
|
| Documented | ✅ Yes (official plugin docs) | ❌ No (internal, undocumented) |
|
|
| Used by | ✅ Langfuse plugin | ❌ No official plugin uses it |
|
|
| Signature | Stable, typed kwargs | May change without notice |
|
|
|
|
## Hook Callbacks
|
|
|
|
### `pre_api_request`
|
|
|
|
Fires **before** every LLM API call. Used to capture:
|
|
- Full request messages (conversation history + current prompt)
|
|
- Request preview (first 300 chars of the last non-system message)
|
|
|
|
> **Note:** Background review fork detection (scanning for `_SKILL_REVIEW_PROMPT`) is **not yet implemented** — all turns are currently stored without turn_type classification. This is tracked in PLAN.md as deferred.
|
|
|
|
### `post_api_request`
|
|
|
|
Fires **after** every LLM API call. Used to capture:
|
|
- Response content and tool calls
|
|
- Token usage (input, output, total, cache_read, cache_write, reasoning)
|
|
- Latency (api_duration)
|
|
- Finish reason
|
|
- Full response payload
|
|
|
|
### `post_tool_call`
|
|
|
|
Fires after each tool returns. Captures tool name, arguments, result, and duration.
|
|
|
|
> **⚠️ Error parameter limitation:** The Hermes hook does **not** provide an `error` parameter — tool failure information is encoded in the `result` field. The `tool_calls.error` column in the database schema exists but is **always `None`** because there is no error data available from the hook payload. Inspect the `result` field to detect tool failures.
|
|
|
|
## Daemon Threads / Subprocess
|
|
|
|
Spawning pattern (in `register()`):
|
|
```python
|
|
threading.Thread(target=_cleanup_loop, daemon=True, name="hermes-trace-cleanup").start()
|
|
threading.Thread(
|
|
target=_run_frontend, daemon=True, name="hermes-trace-frontend"
|
|
).start()
|
|
```
|
|
|
|
Both threads are daemon — they live as long as the Hermes process.
|
|
|
|
### Cleanup Thread
|
|
- Runs every 60 minutes
|
|
- Deletes turns older than `retention_days` (default: 7)
|
|
- Tool calls deleted via `ON DELETE CASCADE`
|
|
### Frontend Subprocess
|
|
|
|
`_run_frontend()` is a daemon thread that:
|
|
|
|
1. **Acquires a singleton lock** via `fcntl.flock(LOCK_EX | LOCK_NB)` — only one
|
|
Hermes process across all profiles runs the frontend.
|
|
2. **Resolves the runtime** — prefers the plugin's own `.venv/bin/python`
|
|
(verified via a subprocess preflight that checks `granian`, `fastapi`, and
|
|
`jinja2` are importable). If unavailable, logs an error and exits
|
|
without retrying.
|
|
3. **Kills orphan processes** — reads a PID file + `/proc/<pid>/cmdline` to
|
|
identify and terminate any lingering Granian from a previous crash.
|
|
4. **Launches Granian** via `subprocess.Popen()` as a **separate OS process**.
|
|
5. **Monitors and respawns** — if Granian exits, sleeps 3 seconds and retries.
|
|
The loop is bounded by the `finally` block which releases the lock,
|
|
closes the file descriptor, and cleans up PID and lock files on exit.
|
|
|
|
Reason for subprocess: Granian's `.serve()` calls `signal.signal()` which only
|
|
works in the main thread of the main interpreter. Running it inside the Hermes
|
|
process would require blocking the Hermes event loop.
|
|
|
|
## Plugin Configuration
|
|
|
|
Configuration is read from **environment variables** (not `plugins_config`):
|
|
|
|
| Env var | Default | Description |
|
|
|---------|---------|-------------|
|
|
| `HERMES_TRACE_DB_PATH` | `$HERMES_HOME/telemetry/telemetry.db`¹ | SQLite database path |
|
|
| `HERMES_TRACE_HOST` | `127.0.0.1` | Granian bind address |
|
|
| `HERMES_TRACE_PORT` | `7788` | Frontend server port |
|
|
| `HERMES_TRACE_RETENTION_DAYS` | `7` | Days to keep turn data |
|
|
|
|
> ¹ `$HERMES_HOME` 預設為 `~/.hermes`(可透過環境變數覆蓋)。使用 `HERMES_HOME` 讓 database 自動跟隨 Hermes profile 切換。
|
|
|
|
In `~/.hermes/config.yaml`, the plugin can be enabled/disabled but runtime config uses env vars:
|
|
|
|
```yaml
|
|
plugins:
|
|
enabled:
|
|
- hermes-trace
|
|
```
|
|
|
|
## Full Payload Storage
|
|
|
|
All fields stored unconditionally:
|
|
- `full_request`: Normalized conversation transcript assembled from provider-bound request data (JSON)
|
|
- `full_response`: Sanitized response snapshot exposed by the Hermes observability hook (JSON)
|
|
- Tool call args and result (JSON)
|
|
|
|
No conditional capture toggle. If storage space is a concern, lower `retention_days`.
|