hermes-agent/website/docs/user-guide/checkpoints-and-rollback.md
unsupportedpastels 9ec9028e50 fix(checkpoints): label checkpoints that did not capture a nested repo
A nested repository is stored as a gitlink, so its files are not in the
checkpoint. Say so when the checkpoint is taken: its message gains
"[nested git repos not captured: <paths>]", which every /rollback listing
(CLI, gateway, TUI, Desktop) already shows. The gitlink parse is shared
with the rollback guard.

rollback.list sent the TUI an always-empty message (it read a "message"
key the manager calls "reason"), so TUI and Desktop listings never showed
what a checkpoint was; map it from reason.

Co-authored-by: FroRaut <[email protected]>
2026-09-29 14:12:41 -04:00

307 lines
14 KiB
Markdown

---
sidebar_position: 8
sidebar_label: "Checkpoints & Rollback"
title: "Checkpoints and /rollback"
description: "Filesystem safety nets for destructive operations using shadow git repos and automatic snapshots"
---
# Checkpoints and `/rollback`
Hermes Agent can automatically snapshot your project before **destructive operations** and restore it with a single command. Checkpoints are **opt-in** as of v2 — most users never use `/rollback`, and the shadow-store storage is non-trivial over time, so the default is off.
Enable checkpoints per-session with `--checkpoints`:
```bash
hermes chat --checkpoints
```
Or enable globally in `~/.hermes/config.yaml`:
```yaml
checkpoints:
enabled: true
```
This safety net is powered by an internal **Checkpoint Manager** that keeps a single shared shadow git repository under `~/.hermes/checkpoints/store/` — your real project `.git` is never touched. Every project the agent works in shares the same store, so git's content-addressable object DB deduplicates across projects and across turns.
## What Triggers a Checkpoint
Checkpoints are taken automatically before:
- **File tools** — `write_file` and `patch`
- **Destructive terminal commands** — `rm`, `rmdir`, `cp`, `install`, `mv`, `sed -i`, `truncate`, `dd`, `shred`, output redirects (`>`), and `git reset`/`clean`/`checkout`
The agent creates **at most one checkpoint per directory per turn**, so long-running sessions don't spam snapshots.
## Quick Reference
In-session slash commands:
| Command | Description |
|---------|-------------|
| `/rollback` | List all checkpoints with change stats |
| `/rollback <N>` | Restore to checkpoint N, keeping your hand-edits (also undoes last chat turn) |
| `/rollback <N> --all` | Full restore — overwrites your hand-edits too |
| `/rollback diff <N>` | Preview diff between checkpoint N and current state |
| `/rollback <N> <file>` | Restore a single file from checkpoint N |
CLI for inspecting and managing the store outside a session:
| Command | Description |
|---------|-------------|
| `hermes checkpoints` | Show total size, project count, per-project breakdown |
| `hermes checkpoints status` | Same as bare `checkpoints` |
| `hermes checkpoints list` | Alias for `status` |
| `hermes checkpoints prune` | Force a sweep: delete orphans/stale, GC, enforce size cap |
| `hermes checkpoints clear` | Nuke the entire checkpoint base (asks first) |
| `hermes checkpoints clear-legacy` | Delete only the `legacy-*` archives from v1 migration |
## How Checkpoints Work
At a high level:
- Hermes detects when tools are about to **modify files** in your working tree.
- Once per conversation turn (per directory), it:
- Resolves a reasonable project root for the file.
- Initialises or reuses the **single shared shadow store** at `~/.hermes/checkpoints/store/`.
- Stages into a per-project index, builds a tree, and commits to a per-project ref (`refs/hermes/<project-hash>`).
- These per-project refs form a checkpoint history that you can inspect and restore via `/rollback`.
```mermaid
flowchart LR
user["User command\n(hermes, gateway)"]
agent["AIAgent\n(run_agent.py)"]
tools["File & terminal tools"]
cpMgr["CheckpointManager"]
store["Shared shadow store\n~/.hermes/checkpoints/store/"]
user --> agent
agent -->|"tool call"| tools
tools -->|"before mutate\nensure_checkpoint()"| cpMgr
cpMgr -->|"git add/commit-tree/update-ref"| store
cpMgr -->|"OK / skipped"| tools
tools -->|"apply changes"| agent
```
## Configuration
Configure in `~/.hermes/config.yaml`:
```yaml
checkpoints:
enabled: false # master switch (default: false — opt-in)
max_snapshots: 20 # max checkpoints per project (enforced via ref rewrite + gc)
max_total_size_mb: 500 # hard cap on total store size; oldest commits dropped
max_file_size_mb: 10 # skip any single file larger than this
# Auto-maintenance (on by default): sweep ~/.hermes/checkpoints/ in the
# background — the CLI on a helper thread right after launch, the gateway
# on its housekeeping tick — and delete project entries whose last_touch is
# older than retention_days. Runs at most once per min_interval_hours,
# tracked via a .last_prune marker. It never blocks the prompt or gateway
# startup: the `git gc` that reclaims space can take tens of seconds on a
# large store. This sweep never deletes "orphan" entries (working directory
# not found) — a missing workdir is ambiguous (deleted project vs. an
# unmounted external volume / network share / VPN not yet up), so orphan
# cleanup is only ever done via the explicit `hermes checkpoints prune`
# command below, with a confirmation prompt.
auto_prune: true
retention_days: 7
min_interval_hours: 24
```
To disable everything:
```yaml
checkpoints:
enabled: false
auto_prune: false
```
When `enabled: false`, the Checkpoint Manager is a no-op and never attempts git operations. When `auto_prune: false`, the store grows until you run `hermes checkpoints prune` manually.
## Listing Checkpoints
From a CLI session:
```
/rollback
```
Hermes responds with a formatted list showing change statistics:
```text
📸 Checkpoints for /path/to/project:
1. 4270a8c 2026-03-16 04:36 before patch (1 file, +1/-0)
2. eaf4c1f 2026-03-16 04:35 before write_file
3. b3f9d2e 2026-03-16 04:34 before terminal: sed -i s/old/new/ config.py (1 file, +1/-1)
/rollback <N> restore to checkpoint N (keeps your hand-edits)
/rollback <N> --all full restore, overwriting your hand-edits too
/rollback diff <N> preview changes since checkpoint N
/rollback <N> <file> restore a single file from checkpoint N
```
## Inspecting the Store from the Shell
```bash
hermes checkpoints
```
Sample output:
```text
Checkpoint base: /home/you/.hermes/checkpoints
Total size: 142.3 MB
store/ 138.1 MB
legacy-* 4.2 MB
Projects: 12
WORKDIR COMMITS LAST TOUCH STATE
/home/you/code/hermes-agent 20 2h ago live
/home/you/code/experiments/rl-runner 8 1d ago live
/home/you/code/old-prototype 3 9d ago orphan
...
Legacy archives (1):
legacy-20260506-050616 4.2 MB
Clear with: hermes checkpoints clear-legacy
```
Force a full sweep (ignores the 24h idempotency marker):
```bash
hermes checkpoints prune --retention-days 3 --max-size-mb 200
```
## Previewing Changes with `/rollback diff`
Before committing to a restore, preview what has changed since a checkpoint:
```
/rollback diff 1
```
This shows a git diff stat summary followed by the actual diff.
## Restoring with `/rollback`
```
/rollback 1
```
Behind the scenes, Hermes:
1. Verifies the target commit exists in the shadow store.
2. Takes a **pre-rollback snapshot** of the current state so you can "undo the undo" later.
3. Restores tracked files in your working directory — **preserving your hand-edits** (see below).
4. **Undoes the last conversation turn** so the agent's context matches the restored filesystem state.
### User hand-edits are preserved by default
`/rollback <N>` restores only the files Hermes itself changed. Every successful
`write_file` / `patch` records the file's content hash in an **agent-write
ledger**; at restore time, any file whose current contents no longer match what
Hermes last wrote (you edited it afterwards, or Hermes never touched it) is
**skipped** instead of overwritten, and listed in the output:
```
✅ Restored to checkpoint a1b2c3d4: before write_file
↷ Kept your hand-edits: src/config.py, notes.md
Use /rollback <N> --all to restore those too.
```
To force the classic full restore that reverts everything — including your own
edits — add `--all`:
```
/rollback 1 --all
```
If the ledger is empty (a store created before this feature, or Hermes hasn't
written any files in the project yet), `/rollback` falls back to the full
restore automatically.
## Single-File Restore
Restore just one file from a checkpoint without affecting the rest of the directory:
```
/rollback 1 src/broken_file.py
```
## Safety and Performance Guards
### Nested Git repositories
A checkpoint of a parent directory can store a nested repository as a Git
**gitlink** (a commit reference), not a copy of its files. Recursive capture of
nested repositories is not supported: their uncommitted edits and untracked
files are not recoverable from that parent checkpoint. Checkpoints taken by this
version or later are labelled in `/rollback` listings, for example
`before write_file: app.py [nested git repos not captured: tool]`.
If the selected checkpoint contains gitlinks, a full rollback (including
`--all`) is refused before changing files or creating a pre-rollback snapshot.
Selecting a nested repository, a file below it, or a Git pathspec matching it
also refuses the restore rather than reporting success for uncaptured files.
You can still restore unrelated captured files, for example
`/rollback 1 notes.txt`. Keep separate backups or checkpoints taken directly
from the nested repository's own working directory.
### Container Backends
With a container terminal backend (`docker`, `singularity`, `modal`, `daytona`, `vercel_sandbox`, or a container plugin), file paths belong to the sandbox rather than the host. Hermes therefore does not take checkpoints or record the agent-write ledger for those paths, and `/rollback` explains the limitation: it still lists existing host checkpoints but refuses diff and restore, on the CLI and in messaging-gateway chats alike; `/diff session` answers with the same reason. The TUI and Desktop behave the same: `/rollback list` still works while `/rollback diff` and `/rollback <N>` are refused with that reason. Local and SSH backends are unaffected. To point `terminal.cwd` at the container-side view of a mounted directory see [`terminal.docker_mount_cwd_to_workspace`](./configuration.md).
- **Git availability** — if `git` is not found on `PATH`, checkpoints are transparently disabled.
- **Directory scope** — Hermes skips overly broad directories (root `/`, home `$HOME`).
- **Repository size** — directories with more than 50,000 files are skipped.
- **Per-file size cap** — files larger than `max_file_size_mb` (default 10 MB) are excluded from the snapshot. Prevents accidentally swallowing datasets, model weights, or generated media.
- **Total store size cap** — when the store exceeds `max_total_size_mb` (default 500 MB), the oldest commit per project is dropped round-robin. Each drop is reclaimed before deciding whether another is necessary. Every project keeps at least one snapshot. A failed Git operation stops pruning and is reported; maintenance never discards more history to compensate for failed reclamation.
- **Real pruning, off the hot path** — `max_snapshots` and the size cap are enforced by rewriting the per-project ref at checkpoint time (cheap); the store is then marked `.gc-pending` and the periodic prune runs `git gc --prune=now` once, so loose objects don't accumulate and a tool call never waits on a full repack.
- **Concurrent operations** — snapshots, restores, diffs and maintenance use one process-shared store lock. An operation reports a busy store rather than running GC over another process's unpublished objects. A restore applies its selected tree before pruning the safety snapshot's history.
- **No-change snapshots** — if there are no changes since the last snapshot, the checkpoint is skipped.
- **Non-fatal errors** — snapshot failures do not block your tools. Pruning failures are logged as warnings; explicit maintenance reports an error count.
## Where Checkpoints Live
```text
~/.hermes/checkpoints/
├── store/ # single shared bare git repo
│ ├── HEAD, objects/ # git internals (shared across projects)
│ ├── refs/hermes/<hash> # per-project branch tip
│ ├── indexes/<hash> # per-project git index
│ ├── projects/<hash>.json # workdir + created_at + last_touch
│ ├── .gc-pending # refs rewritten since the last gc; cleared by the next prune
│ └── info/exclude
├── .last_prune # auto-prune idempotency marker
└── legacy-<ts>/ # archived pre-v2 per-project shadow repos
```
Each `<hash>` is derived from the absolute path of the working directory. You normally never need to touch these manually — use `hermes checkpoints status` / `prune` / `clear` instead.
The sibling `.checkpoints.lock` coordinates processes and survives a store clear. Do not remove it while Hermes is running.
### Migration from v1
Before the v2 rewrite, each working directory got its own complete shadow git repo directly under `~/.hermes/checkpoints/<hash>/`. That layout couldn't dedup objects across projects and had a documented no-op pruner — the store would grow without bound.
On first v2 run, any pre-v2 shadow repos are moved into `~/.hermes/checkpoints/legacy-<timestamp>/` so the new single-store layout starts clean. Old `/rollback` history is still reachable by manually inspecting the legacy archive with `git`; once you're confident you don't need it, run:
```bash
hermes checkpoints clear-legacy
```
to reclaim the space. Legacy archives are also swept by `auto_prune` after `retention_days`.
## Best Practices
- **Enable checkpoints only when you need them** — `hermes chat --checkpoints` or per-profile `enabled: true`.
- **Use `/rollback diff` before restoring** — preview what will change to pick the right checkpoint.
- **Use `/rollback` instead of `git reset`** when you want to undo agent-driven changes only.
- **Check `hermes checkpoints status` occasionally** if you use checkpoints regularly — shows which projects are active and what the store costs you.
- **Combine with Git worktrees** for maximum safety — keep each Hermes session in its own worktree/branch, with checkpoints as an extra layer.
For running multiple agents in parallel on the same repo, see the guide on [Git worktrees](./git-worktrees.md).