hermes-agent/cron/AGENTS.md
kshitijk4poor 744f05e220 fix(cron): stop the external worker when PM refuses its dependency boot
worker_bootstrap() logged a failed activate_dependencies() and let the worker
carry on through the inherited PYTHONPATH pin. That path is not leased: once
the gateway releases its generation the collector deletes it under the live
worker and the next dependency import dies mid-job (#122936 review, reproduced
with a committed generation that has no site-packages).

The failure now propagates, so the worker exits before its ownership ack and
_launch_external_cron_worker reports the dispatch failure with the worker's
stderr. PM's own no-op cases (venv, wheel, Nix, nothing committed under a venv)
still return normally.

Tests drive the real PM boot in a marked child: it imports a sentinel that
exists only in the committed generation, and when an update commits a newer
one mid-job the real collector keeps the worker's generation until the worker
exits. A damaged selection exits before cron.jobs is imported.

123456hyu made the same prelude move and fail-closed change on #122290
(dff16e12); this carries them without the argv gate and source-order test.

Co-authored-by: 123456hyu <[email protected]>
2026-09-29 03:32:36 +05:30

150 lines
12 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# cron/ (+ kanban) — scheduled jobs and the multi-agent work queue
Applies on top of the root `AGENTS.md`. Long-form: `website/docs/developer-guide/cron-internals.md`;
user docs `website/docs/user-guide/features/cron.md`, `kanban.md`.
## Cron
`cron/jobs.py` (job store) + `cron/scheduler.py` (tick loop; `scheduler_*.py` siblings). Agents
schedule via the `cronjob` tool; users via `hermes cron list|add|edit|pause|resume|run|remove` or
`/cron`. Schedules: duration (`"30m"`, `"2h"`, `"1d"`), "every" phrase (`"every 2h"`, `"every monday
9am"`), 5-field cron (`"0 9 * * *"`), ISO one-shot (`"2026-06-01T09:00:00Z"`). Per-job fields:
`skills`, `model`/`provider` overrides, `script` (pre-run data-collection script whose stdout is
injected into the prompt; `no_agent=True` makes the script the whole job), `context_from` (chain job
A's last output into job B's prompt), `workdir` (run with that directory's `AGENTS.md`/`CLAUDE.md`
loaded), multi-platform delivery.
Hardening invariants — each guards a real failure; don't weaken without answering for it:
- **Inactivity watchdog** on cron agent sessions (`_cron_inactivity_seconds()`): default 600s idle,
`HERMES_CRON_TIMEOUT` overrides, `0` = unlimited. It is idle time, not wall-clock — a stalled
session is hard-interrupted so it cannot monopolise the scheduler, while a long-but-active job
is never cut off. Attached scripts (pre-run or `no_agent`) are bounded separately by the script
timeout (`_DEFAULT_SCRIPT_TIMEOUT`, 3600s).
- Catch-up window = half the period, clamped to 120s–2h; 120s grace for missed one-shots.
- Every recurring occurrence is accounted for: `tick()` advances `next_run_at` BEFORE dispatch
(at-most-once across a mid-run crash) and stamps `pending_slot` in the same save; a scan that
finds the stamp with a dead owner restores the instant ONCE (`cron/occurrences.py`), the
executions ledger's `scheduled_instant` blocks a second fire, `cron.catch_up_missed: false`
skips past-grace misses with a logged reason. Never drop a slot silently (#107485).
- Per-home tick lock `<home>/cron/.tick.lock` prevents duplicate ticks across processes for
that profile's store; never a `~/.hermes/...` literal.
- **The ticker binds each served profile's scope for the whole tick, including pre-loop code.**
`scheduler_provider.py::_start_multiplex` is ONE ticker iterating `profiles_to_serve()`
sequentially under `_profile_cron_scope(home)` (home + secret scope + terminal scope) — never N
threads (module globals race). Everything a tick touches lives inside that guard: store open,
lock path, backoff/failure counters (`_note_tick_failure`), job env construction, and the
`on_session_end` flush of a finished job. Supervision (`scheduler_thread.py::
SupervisedTickerThread`; start on gateway boot, stand down for homes another gateway already
serves, re-enumerate when a profile dir appears or is tombstoned) is per served home, not per
process. Why: a store opened before the scope was entered wrote a secondary profile's run
records into the launch profile's `jobs.json`.
- **Cron ownership is not gated on `gateway.multiplex_profiles`.** That flag gates ADAPTERS; one
host gateway process ticks EVERY profile's store either way (`run.py::_cron_tick_profile_homes`).
Gating the tick set on it left every non-launch profile's jobs in a store no ticker visited.
- **Per-profile process assumptions are the bug class.** One process ticks N homes, so anything
keyed on "this process's profile" is wrong: in-flight state (`_running_job_ids`,
`_running_since`, `_running_futures`, `_running_worker_pids`, `_running_fire_owners`,
`_interrupted_job_ids`) is keyed by `_inflight_key(job_id)` = `(home key, job id)` — two
profiles legitimately carry a `daily-brief`; the parallel pool is keyed by home
(`cron.max_parallel_jobs` is per profile); and the stale-code yield gate asks
`scheduler_ownership.owns_cron_tick_for(home)` / `live_gateway_ticking(home)` instead of the
process-global runtime-lock boolean. Public accessors (`get_running_job_ids`,
`get_running_job_details`, `get_wedged_job_ids`) still report the host-wide union of bare job
ids for the shutdown drain, but LIVENESS consumers (`jobs.py::_job_running_in_this_process`,
`tools/cronjob_tools`) ask `is_job_running(job_id, home=...)` — the union made profile A's
running `daily-brief` answer for profile B's idle one.
- **A claim is released under the key it was registered with.** The cron scope is a ContextVar:
`try_register_running_job` runs on the ticker thread inside `_profile_cron_scope`, while the
pool worker's `finally` sits OUTSIDE `ctx.run` and resolves the LAUNCH home. Pass the
registering home (`release_running_job(job_id, home=...)`), or every secondary profile's claim
leaks — the job skips a fire window until the force-release backstop sweeps it, and the drain
sees phantom work. Never rebuild a home from a key half (`Path(key[0])`): `hermes_home_key`
normcases, so use `_inflight_home_path`.
- **Ticked-home state is reclaimed when a home leaves the set.** `register_ticked_homes` is
republished every cycle and reaps the departed homes' parallel pools; pools used to live until
`atexit`, so each home ever ticked kept a ThreadPoolExecutor and its worker threads forever.
- **The host gateway stands down for a profile that runs its OWN gateway.** `run.py::
_cron_profile_gate` (the same gate `hermes_cli/web_server.py` passes) keeps the launch process
and a per-profile gateway off one store: the tick lock stops a simultaneous double-run but not
the race, and when the launch process wins, delivery goes through `SharedRouteAdapters`/
fail-closed instead of that profile's live adapters. The gate compares the liveness PID against
`os.getpid()` — this process holds the launch `gateway.pid` AND publishes every served profile
in `served_profiles`, so a bare liveness answer would stand cron down host-wide.
- **The restart-safe external worker boots itself.** `_launch_external_cron_worker` pins the
checkout — and, on a PM install, the committed generation's `site-packages` — on the child's
`PYTHONPATH` and marks it `_HERMES_CRON_WORKER_BOOT`; the child's package entry
(`cron/__init__.py` → `cron/worker_bootstrap.py`, ahead of the `cron.jobs` import — `-m
cron.scheduler` runs the package first) then runs PM's `activate_dependencies`, which
leases the committed generation for the worker's lifetime, runs its `.pth` files and
activates it before the first third-party import. A failed activation is fatal: the worker
exits before its ownership ack (a reported dispatch failure) rather than run on an unleased
generation the collector may delete. The gateway never re-runs the boot — `hermes_bootstrap`
already did at its own launch (#122222).
- Cron sessions pass `skip_memory=True`; memory providers intentionally do not run during cron.
- Cron execution has its own session. Eligible continuable deliveries may mirror or seed the
reply-facing conversation: origin, origin-less home fallback, user-written bare-platform home,
or opted-in explicit targets. `all` expansions do not gain home mirror eligibility. Mirrored
briefs are labelled user turns appended at a turn boundary, preserving role alternation.
- The cron ticker runs in the desktop-spawned backend when `HERMES_DESKTOP=1` — that env var means
"spawned by the app", not "a GUI is watching" (root: capability is a property of the session).
- Background `delegate_task` is process-local; work that must survive restarts is a cron job or a
`terminal(background=True, notify_on_complete=True)` process.
## Kanban (multi-agent work queue)
Durable SQLite-backed board letting multiple profiles/workers collaborate. Users: `hermes kanban
<verb>`; dispatcher-spawned workers use a dedicated `kanban_*` toolset so their schema footprint is
zero outside a kanban task (footprint ladder rung 3).
- **CLI:** `hermes_cli/kanban.py` facade + 14 `kanban_*.py` siblings (`boards`, `db`, `db_connect`,
`db_dispatch`, `db_notify`, `db_graph` (task initialization and decomposition), `workspace`, ...). Verbs: `init, create, list (ls), show, assign, link,
unlink, comment, attach, attachments, attach-rm, complete, request-review, request-changes,
reopen-review, block, unblock, archive, tail`, plus `watch, stats, runs, log, assignees, heartbeat,
notify-*, dispatch, daemon, gc`. Argparse alias dispatch must accept both `list` and `ls` (root).
- **Toolset:** `tools/kanban_tools.py` — `kanban_show, kanban_complete, kanban_request_review,
kanban_request_changes, kanban_block, kanban_heartbeat, kanban_comment, kanban_create, kanban_link,
kanban_attach, kanban_attach_url, kanban_attachments`; platforms whose saved selection enables
`kanban` (`hermes tools enable kanban --platform <p>`; default-off, in `CONFIGURABLE_TOOLSETS`) get
the full set plus `kanban_list`/`kanban_unblock` for board routing. The check_fn reads the schema
build's own selection (`tools/kanban_toolset_context.py`), never the legacy top-level `toolsets`
key alone.
- **Dispatcher:** long-lived loop (default 60s) that reclaims stale claims, promotes ready tasks,
atomically claims, and spawns assigned profiles. Runs **inside the gateway** by default
(`kanban.dispatch_in_gateway: true`). Standalone: `plugins/kanban/systemd/hermes-kanban-dispatcher.service`.
- **Plugin assets:** `plugins/kanban/dashboard/` (web UI) + systemd unit. `kanban_db.connect` is its
own connection helper — do not alias it to `projects_db.connect` (a path-proximity generator did).
Isolation: **board** is the hard boundary — workers get `HERMES_KANBAN_BOARD` pinned in their env and
cannot see other boards; **tenant** is a soft namespace within a board (workspace-path + memory-key
isolation, one fleet serving several businesses). After `kanban.failure_limit` consecutive
non-success attempts on a task (default 2) the dispatcher auto-blocks it to stop spin loops; a
worker exit of `KANBAN_TERMINAL_PROVIDER_EXIT_CODE` (78 — credential revoked, model gone; the
worker's own `failure_reason` classification via `cli._TERMINAL_PROVIDER_REASONS`) trips it on
the first attempt, sticky, because no retry can heal it (#114587).
Process-identity note: `kanban --preserve-cache` contains "serve" — never classify processes by argv
substring (root). Worker liveness is `(worker_pid, worker_started_at)` — the start-time fingerprint
(`gateway.status.get_process_start_time`) recorded at claim time — never bare PID existence, or a
recycled PID gets killed on reclaim.
- **Notifications leave through the task's owning profile.** `hermes_cli/kanban_db_notify.py`
subscriptions carry the profile; `gateway/kanban_watchers_notifier.py` delivers via THAT
profile's adapter under its scope (`_notify_profile_filter`), never the multiplexer's launch
adapter; a fail-closed skip logs once at WARNING with the remedy, never a bare `continue`.
Dispatched workers get `HERMES_KANBAN_BOARD` and the assignee's `HERMES_HOME` pinned in a
scrubbed child env (`build_subprocess_env` + `strip_launch_profile_env`); they never inherit the
default profile's `.env`.
- **Prompt injection sites gate on ownership, not tool access.** Tool access (`kanban_show` visible
via a profile's toolset) and an inherited `HERMES_KANBAN_TASK` (delegate children, cron runs beside
a worker) are not ownership. The kanban guidance (`agent_init`, `system_prompt` fallback) and the
stop nudge resolve the task via `agent/delegation_context.py::owned_kanban_task()`; other readers
pair their env read with `is_dispatcher_owned_worker_context()`.
- **Descendant fence is a path, not a flag.** A delegated child's Kanban marker
(`agent/delegation_context.py::DELEGATED_CHILD_ENV_MARKER`) carries the fenced board ROOT;
`kanban_path_is_fenced(path)` denies mutations only on the dispatcher-pinned `HERMES_KANBAN_DB`
or under that root, so a child working against a scratch `HERMES_HOME` keeps a writable board.
## Tests
`tests/cron/`, `tests/hermes_cli/test_kanban*.py`, `tests/tools/test_kanban*.py`. Schedule parsing
and catch-up windows are pure functions — test them as data. Never assert on the verb list or
toolset size (root: no change-detectors). Time-based tests use loose bounds (≥ 2s) and event sync.