mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-11-17 04:11:35 +00:00
Two Windows agent-loop friction fixes: 1. tools/mcp_tool.py (#56536): shutil.which(cmd, path=env_path) reads executable extensions from the PARENT process PATHEXT, not the MCP subprocess env — a stdio MCP config supplying both PATH and PATHEXT could fail to resolve a command its own env can locate, and startup then got a bare command name. On Windows, when the first which() call misses and the config env carries PATHEXT (any key casing), retry the resolution with the config's PATHEXT temporarily applied. 2. skills/ + optional-skills/ (#50606): 42 SKILL.md files that declare platforms: [.., windows] used python3 in their command examples. python3 does not exist on native Windows (the toolchain probe in the system prompt reports python3=missing), so every copy-pasted example burned a failed agent turn before self-correction. Replaced the command word python3 -> python (python3-config / python3.x version strings untouched). python is the spelling that exists in every Hermes-managed environment (Windows native, uv-managed venvs on all three OSes); agents on POSIX hosts additionally see the probed toolchain line and adapt either way.
278 lines
15 KiB
Markdown
278 lines
15 KiB
Markdown
---
|
|
name: tldraw-offline
|
|
description: Drive and script tldraw offline canvases with an agent.
|
|
version: 1.0.0
|
|
author: Teknium + Hermes Agent
|
|
license: MIT
|
|
platforms: [linux, macos, windows]
|
|
metadata:
|
|
hermes:
|
|
tags: [tldraw, canvas, whiteboard, document-script, diagramming]
|
|
category: creative
|
|
related_skills: []
|
|
---
|
|
|
|
# tldraw offline Skill
|
|
|
|
Work with the tldraw offline desktop app (offline.tldraw.com): read the open
|
|
canvas, make edits, and write **document scripts** — JavaScript embedded in a
|
|
`.tldraw` file that runs on load and gives the file durable behavior. The app
|
|
runs a **local HTTP API** (default `localhost:7236`) that a coding agent drives
|
|
with plain `curl` from its terminal — this is exactly how the app's own homepage
|
|
demo (Codex editing a canvas live) works. The agent does NOT use computer-use /
|
|
GUI clicking, and does NOT hand-edit the `.tldraw` file directly. Keep tldraw
|
|
offline open while you work.
|
|
|
|
## When to Use
|
|
|
|
- The user has tldraw offline open and asks you to build or modify a canvas
|
|
(diagrams, wireframes, layouts).
|
|
- You want to add durable behavior to a drawing (reactive shapes, interactive
|
|
buttons, animation, connection logic) via an embedded document script.
|
|
|
|
Do NOT hand-place shapes to imitate a drawing — write the code that generates
|
|
them. Agents are far better at scripting the canvas than at drawing on it.
|
|
|
|
## Prerequisites
|
|
|
|
- **tldraw offline installed and running**, with a document open. Releases:
|
|
https://github.com/tldraw/tldraw-offline/releases/latest (macOS DMG, Windows
|
|
x64/Arm64, Linux `x86_64`/`arm64` AppImage or amd64/arm64 `.deb`).
|
|
- **Agent skills installed in the app**: `Develop → Install Agent Skills`. The
|
|
app writes its own tldraw skill into `~/.codex/skills/`, `~/.claude/skills/`,
|
|
`~/.cursor/skills/`, and `~/.gemini/skills/` — teaching that agent the `curl`
|
|
recipes below. (This Hermes skill mirrors that guidance for Hermes.)
|
|
- **The local control API.** On launch the app writes `server.json` to its config
|
|
dir (Linux `~/.config/tldraw/`, macOS `~/Library/Application Support/tldraw/`,
|
|
Windows `%APPDATA%\tldraw\`) with `port` (default `7236`), a bearer `token`,
|
|
`pid`, and `startedAt`. Every request except `GET /` needs
|
|
`Authorization: Bearer <token>`. A clean quit removes `server.json`; if it's
|
|
present but the port doesn't answer, the app quit uncleanly — treat as not
|
|
running.
|
|
- **Re-read port + token on EVERY shell call.** Each terminal call is a fresh
|
|
shell, so an `export`ed token does not persist — "export once and reuse" sends
|
|
an empty token and 401s. Read both inline at the top of each call:
|
|
`PORT=$(jq -r .port <server.json>); TOKEN=$(jq -r .token <server.json>)`.
|
|
- No account or network needed for local editing.
|
|
|
|
## How to Run
|
|
|
|
Two distinct workflows. Pick by whether the change must survive a reload.
|
|
|
|
**A. One-off canvas edits (`/exec`)** — layout, generating shapes, cleanup. This
|
|
is a live edit, not saved script:
|
|
|
|
```bash
|
|
BASE=http://localhost:7236
|
|
TOKEN=$(python -c "import json;print(json.load(open('$HOME/.config/tldraw/server.json'))['token'])")
|
|
# find the focused document id
|
|
DOC=$(curl -s "$BASE/api/search" -X POST -H 'content-type: application/json' \
|
|
-H "Authorization: Bearer $TOKEN" \
|
|
-d '{"code":"return (await api.getFocusedDoc()).id"}' | python -c "import sys,json;print(json.load(sys.stdin)['result'])")
|
|
# run code with the live `editor` + `helpers` in scope
|
|
curl -s "$BASE/api/doc/$DOC/exec" -X POST -H 'content-type: application/json' \
|
|
-H "Authorization: Bearer $TOKEN" \
|
|
-d '{"code":"const {createShapeId,toRichText}=await import(\"tldraw\"); editor.createShape({id:createShapeId(),type:\"geo\",x:0,y:0,props:{geo:\"rectangle\",w:200,h:100,color:\"blue\",fill:\"solid\",richText:toRichText(\"hello\")}}); return editor.getCurrentPageShapes().length"}'
|
|
```
|
|
|
|
**B. Durable behavior (`script/main.js`)** — reactive/interactive logic that must
|
|
survive reload. Edit the file on disk; the app's watcher applies it:
|
|
|
|
```bash
|
|
# get the live script file path for the doc
|
|
curl -s "$BASE/api/doc/$DOC/script-workspace" -X POST \
|
|
-H "Authorization: Bearer $TOKEN" # -> result.mainJsPath, result.isDefaultScript
|
|
# edit result.mainJsPath with read_file / patch / write_file (see scripts/main.js)
|
|
# then confirm the watcher applied it:
|
|
curl -s "$BASE/api/doc/$DOC/script-status" -H "Authorization: Bearer $TOKEN"
|
|
```
|
|
|
|
The ready-to-adapt document script is `scripts/main.js`.
|
|
|
|
## Quick Reference
|
|
|
|
The document-script contract (verified against the app's bundled
|
|
`script-context.d.ts`):
|
|
|
|
```js
|
|
import { createShapeId, toRichText } from 'tldraw' // primitives: import, not globals
|
|
|
|
export default function ({ editor, helpers, signal }) {
|
|
editor.run(() => { // batch = one undo step
|
|
helpers.createShapeIfMissing({ // idempotent furniture
|
|
id: createShapeId('node-1'), type: 'geo', x: 0, y: 0,
|
|
props: { geo: 'rectangle', w: 200, h: 100, richText: toRichText('hi') },
|
|
})
|
|
})
|
|
|
|
const stop = editor.store.listen(() => { /* react */ }) // fires the tick AFTER a commit
|
|
signal.addEventListener('abort', () => stop()) // REQUIRED cleanup on rerun/close
|
|
}
|
|
```
|
|
|
|
- `ctx.editor` — the live `Editor` (`createShape`, `updateShape`, `deleteShapes`,
|
|
`getCurrentPageShapes`, `getShape`, `getBindingsFromShape`, `zoomToFit`,
|
|
`on('tick'|'event', fn)`, `run(fn, { history: 'ignore' })`).
|
|
- `ctx.helpers` — `createShapeIfMissing`, `createShapesIfMissing`,
|
|
`createArrowBetweenShapes(from, to, { arrowheadEnd })`, `translateShapes`,
|
|
`onShapeTranslate(id, fn, { signal })`, `richTextToPlainText`, `boxShapes`,
|
|
`getLints`.
|
|
- `ctx.signal` — `AbortSignal`; attach every listener/interval teardown to it.
|
|
- `config.js` (separate file) registers custom shape/tool/component utils and
|
|
runs before mount; `main.js` runs against the mounted editor and reruns on save.
|
|
|
|
## Interactive UI (clickable buttons that drive state)
|
|
|
|
Drawn shapes can behave like a real app — the thing a static whiteboard can't do.
|
|
Full example: `scripts/counter.js` (a number display + MINUS/RESET/PLUS buttons).
|
|
|
|
Verification boundary — read this before claiming interaction works or doesn't.
|
|
The app's OWN agent playbook says to verify a clickable-UI script with "one
|
|
simulated click and one state read" via `/exec` (`editor.dispatch` a pointer
|
|
event, await a tick, read the shape's state) — NOT by driving a real mouse. By
|
|
that standard the counter is verified: dispatched clicks stepped it
|
|
`0 → 1 → 2 → 1 → 0`. Two caveats worth writing down:
|
|
- **The script only runs once the app's file-watcher applies it.** On Linux that
|
|
watcher uses inotify; a host with an exhausted `fs.inotify.max_user_instances`
|
|
logs `inotify_add_watch ... No space left on device`, `script-status` shows
|
|
`state: "not-watching"` / `hasEntry: false`, and the script never executes.
|
|
This is a host limit, not a script bug — a normal desktop is unaffected.
|
|
- **Driving the canvas with computer-use needs FOREGROUND delivery, not
|
|
background.** cua-driver's default `background` delivery returns
|
|
`background_unavailable` against Electron's occluded renderer — but that is the
|
|
first rung, not a wall. cua-driver returns an `escalation: "foreground"` hint;
|
|
set `delivery_mode: "foreground"` (pair with `bring_to_front`) and its X11 XTest
|
|
path (`x11_xtest_fg`) clicks land — this dismisses the "Run Script" consent
|
|
dialog and clicks the canvas. Verified by the Cua team on this exact v1.11.0
|
|
AppImage (Linux/X11). Do NOT conclude "Electron rejects synthetic clicks" and
|
|
give up in background mode — climb to foreground. (The real product path is
|
|
still `/exec`, not clicks; this note is for computer-use-driven testing.)
|
|
|
|
The pattern:
|
|
|
|
```js
|
|
export default function ({ editor, helpers, signal }) {
|
|
// 1. Build buttons idempotently; tag each with meta so the handler finds them.
|
|
// Give buttons a visible label AND a meta.action.
|
|
// 2. Hit-test pointer_down in PAGE coordinates against the button bounds:
|
|
const inside = (b, p) => p.x >= b.x && p.x <= b.x + b.w && p.y >= b.y && p.y <= b.y + b.h
|
|
function onEvent(info) {
|
|
if (!info || info.name !== 'pointer_down') return
|
|
let p = null
|
|
try { if (info.point && editor.screenToPage) p = editor.screenToPage(info.point) } catch {}
|
|
p = p ?? editor.inputs?.currentPagePoint
|
|
if (!p) return
|
|
const hit = editor.getCurrentPageShapes().find(
|
|
(s) => s.meta?.ui === 'button' &&
|
|
inside({ x: s.x, y: s.y, w: s.props.w, h: s.props.h }, p)
|
|
)
|
|
if (hit) runAction(hit.meta.action) // mutate state; store it in a shape's meta
|
|
}
|
|
editor.on('event', onEvent)
|
|
signal.addEventListener('abort', () => editor.off('event', onEvent)) // REQUIRED
|
|
}
|
|
```
|
|
|
|
- Find buttons by `meta` (or visible label via `helpers.richTextToPlainText`),
|
|
not by hard-coded coordinates.
|
|
- **One script owns both build and read.** If the shapes are created by one code
|
|
path (with `meta.action: 'inc'`) and the handler reads another convention
|
|
(`meta.action === 'PLUS'`), clicks silently do nothing. Ship the buttons built
|
|
by the same script that handles them, or ship an empty canvas so the script
|
|
builds them fresh — never pre-bake mismatched shapes into the file's db.
|
|
- Keep app state in a shape's `meta` (e.g. `meta.count`) and render it as that
|
|
shape's `richText` label, so it survives save and is readable for verification.
|
|
- **Detach the listener on `signal` abort.** Skipping this is not cosmetic: on
|
|
the next save the old `onEvent` stays attached alongside the new one, so every
|
|
click fires twice and a counter jumps by 2 instead of 1.
|
|
- For continuous motion use `editor.on('tick', fn)`; for a moving anchor with
|
|
attached pieces use `helpers.onShapeTranslate(id, fn, { signal })`.
|
|
|
|
### Shipping a self-running scripted `.tldraw`
|
|
|
|
A `.tldraw` is a zip of `metadata.json` + `session.json` + `db.sqlite` + `assets/`
|
|
+ `script/` (only those entries are packable). For the script to auto-run without
|
|
the "This document contains a script → Run Script" consent dialog:
|
|
|
|
- `metadata.json` must carry a `script` manifest: `{ "sha256": "<digest>" }`, where
|
|
the digest is `sha256` over each sorted `script/` path as `` `${path}\0${sha256hex(bytes)}\n` ``.
|
|
A mismatch is rejected as tampered.
|
|
- Pre-trust the digest by adding it to `~/.tldraw/script-trust.json`
|
|
(`{ "trusted": ["<digest>"] }`, or `$TLDRAW_SCRIPT_TRUST`). The app skips consent
|
|
when `isScriptTrusted(digest)` is true.
|
|
|
|
## Procedure
|
|
|
|
1. Read the current token/port from `server.json`. Find the target doc with
|
|
`api.getFocusedDoc()` (or `api.getDocs()`); name it explicitly if several are
|
|
open.
|
|
2. For layout/generation, use `/exec`. For durable behavior, edit
|
|
`script/main.js` via `/script-workspace`.
|
|
3. Make scripts idempotent: create durable shapes with `helpers.createShapeIfMissing`
|
|
and stable `createShapeId('name')` ids. Scripts rerun on every load.
|
|
4. Keep script-owned writes out of the user's undo stack:
|
|
`editor.run(fn, { history: 'ignore' })` (or `helpers.translateShapes`, which
|
|
already does).
|
|
5. For reactivity, `editor.store.listen(cb)` and tear it down on `signal` abort.
|
|
For interaction, `editor.on('event', h)` (hit-test `pointer_down` in page
|
|
coords); for animation, `editor.on('tick', h)`.
|
|
6. For a single moving anchor + attached internals, prefer
|
|
`helpers.onShapeTranslate(anchorId, fn, { signal })` over a broad store
|
|
listener — a broad listener can turn your own writes into feedback loops.
|
|
|
|
## Shape props (validated against tldraw SDK v5 schema)
|
|
|
|
`editor.createShape` / `createShapeIfMissing` accept partial props (shape utils
|
|
fill defaults). When building **raw records** for a file snapshot, every prop
|
|
below is required (run `scripts/validate_shapes.mjs`):
|
|
|
|
| Shape | Required props |
|
|
|-------|----------------|
|
|
| `note` | `richText`, `color`, `labelColor`, `size`, `font`, `align`, `verticalAlign`, `growY`, `fontSizeAdjustment`, `url`, `scale`, `textLastEditedBy` |
|
|
| `text` | `richText`, `color`, `size`, `font`, `textAlign`, `w`, `scale`, `autoSize` |
|
|
| `frame` | `w`, `h`, `name`, `color` |
|
|
| `geo` | `geo`, `w`, `h`, `color`, `fill`, `richText` (+ dash/size/etc. defaulted) |
|
|
|
|
`richText` must be `toRichText('...')` — a bare string is rejected. `color` enum:
|
|
`black grey light-violet violet blue light-blue yellow orange green light-green
|
|
light-red red white`. `font` enum: `draw sans serif mono`.
|
|
|
|
## Pitfalls
|
|
|
|
- **`store.listen` fires on the tick AFTER a commit, not synchronously.** If you
|
|
write a shape and immediately read state expecting the listener to have run, it
|
|
hasn't. Verified live: an in-turn read shows 0 fires; after one `setTimeout`
|
|
tick it shows 1. Same reason the app notes `editor.dispatch` is async — await a
|
|
tick before verifying.
|
|
- **`ctx`, not globals.** The entry is `export default function ({ editor,
|
|
helpers, signal })`. There is no bare `editor` global in a document script.
|
|
`createShapeId` / `toRichText` / `Vec` come from `import ... from 'tldraw'`.
|
|
- **`richText`, not `text`.** Text/note/geo labels use `richText: toRichText(s)`.
|
|
- **Raw records need every prop; `createShape` does not.** In-app pass only the
|
|
props you care about; a hand-built `.tldraw` snapshot needs the full set (table).
|
|
- **Scripts rerun on every load — be idempotent.** Use `createShapeIfMissing`
|
|
with stable ids or you duplicate content and clobber user edits.
|
|
- **Clean up on `signal`.** `signal.addEventListener('abort', () => stop())` for
|
|
every `store.listen` / `editor.on` / `setInterval`; the signal fires before
|
|
rerun and on close.
|
|
- **Keep script writes out of undo:** `editor.run(fn, { history: 'ignore' })`.
|
|
- **`editor.on('tick')` pauses when the window is hidden** (it is a RAF loop);
|
|
`setInterval` keeps firing but Electron throttles it to ~1/s in the background.
|
|
- **The API needs the bearer token** from `server.json`; the port can be non-default
|
|
(`server.listen(0)` picks one) — always read the file, don't hardcode `7236`.
|
|
- **Only `tldraw` / `react` / `react-dom` import** — not a Node project.
|
|
|
|
## Verification
|
|
|
|
- **Shape schema (offline, no app):** `node scripts/validate_shapes.mjs` — builds
|
|
the real tldraw schema and validates note/text/frame. Passing prints `3/3`.
|
|
- **Live canvas edits:** after `/exec`, read back with `/api/search` →
|
|
`api.getShapes(docId)` (returns `{ page, viewport, shapes }`) and
|
|
`api.getBindings(docId)` (array). Confirm expected shapes/bindings exist. Grab
|
|
`api.getScreenshot(docId)` (returns `{ filePath, ... }`) and inspect the PNG/JPEG
|
|
with `vision_analyze`.
|
|
- **Durable script applied:** `GET /api/doc/:id/script-status`. Success is
|
|
`state: "applied"` (`currentDiskDigest === lastAppliedDigest === manifestSha256`,
|
|
`pendingApply === false`, `lastApplyError === null`). If it stays `"pending"`
|
|
after a short retry, report that instead of claiming success; `"error"` means
|
|
the apply failed — read `errorLogPath`.
|