mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-11-17 04:11:35 +00:00
The upstream ahujasid/blender-mcp and ahujasid/ableton-mcp GitHub repos were hijacked on 2026-08-08: the maintainer (@sidahuj) publicly reported his account was compromised and ownership stripped, and both repos now redirect to an attacker-controlled org (MCPBlender, created the same day, pushing new commits since). Although our catalog pinned blender-mcp==1.6.4 from PyPI (pre-compromise, sha256 verified unchanged), the server is only half the bridge: the manifest's post-install instructions and the optional skill directed users to download addon.py — arbitrary Python executed inside Blender — from the now-compromised GitHub repo (the raw URL currently 404s, and the addon ships in no PyPI artifact). There is no trustworthy source for the addon half, so the entry cannot be installed safely end-to-end. Removing the catalog entry and skill entirely until the maintainer confirms account recovery; re-adding is a follow-up PR once upstream is verified clean. - optional-mcps/blender/: removed - optional-skills/creative/blender-mcp/: removed - docs: catalog rows, sidebar entry, skill pages (en + zh-Hans) removed - cross-references in unreal-mcp and kanban-video-orchestrator cleaned
364 lines
19 KiB
Markdown
364 lines
19 KiB
Markdown
# Unreal MCP — Tool Surface Reference
|
|
|
|
How Epic's editor-embedded MCP server organizes, advertises, and executes
|
|
tools, and how to extend the surface when the shipped tools run out.
|
|
Everything here is against UE 5.8's experimental plugin (id
|
|
`ModelContextProtocol`); expect drift between engine versions — the live
|
|
`describe_toolset` schema always outranks this file.
|
|
|
|
## Architecture in one paragraph
|
|
|
|
The **Unreal MCP** plugin hosts the HTTP server inside the editor process
|
|
(default `http://127.0.0.1:8000/mcp`, loopback-only, no auth, HTTP + SSE
|
|
only — no stdio/WebSocket). It implements the protocol but ships no tools of
|
|
its own. Tools come from **Toolsets** — classes deriving from
|
|
`UToolsetDefinition` (C++) or `unreal.ToolsetDefinition` (Python) — collected
|
|
at startup by the **Toolset Registry** subsystem (sibling plugin,
|
|
auto-enabled; the registry itself ships no toolsets either). The shipped
|
|
tools live in per-domain plugins under `Engine/Plugins/Experimental/
|
|
Toolsets/` — the workhorse is **EditorToolset** (core editor toolsets,
|
|
Python + C++) — and **AllToolsets** is a one-checkbox aggregator plugin
|
|
that depends on ~21 of them (verified from `AllToolsets.uplugin`, 5.8):
|
|
AIModule, AnimationAssistant, AutomationTest, ConfigSettings, Conversation,
|
|
DataRegistry, DataflowAgent, Editor, GameFeatures, GameplayTags, GAS,
|
|
MCPClient, Niagara, PCG, Physics, Plugin, SemanticSearch, SlateInspector,
|
|
StateTree, UMG, WorldConditions. Project plugins and Game Feature Plugins
|
|
can contribute more. Unreal MCP wraps every registered tool call as an MCP
|
|
Tool. Execution is **serialized onto the game thread** — one tool call at a
|
|
time, editor UI blocked while each runs.
|
|
|
|
## Tool-search mode (the default contract)
|
|
|
|
With `Enable Tool Search` on (default), `tools/list` advertises exactly three
|
|
meta-tools:
|
|
|
|
| Meta-tool | Args | Returns |
|
|
|---|---|---|
|
|
| `list_toolsets` | — | Registered toolset names + descriptions |
|
|
| `describe_toolset` | toolset name | JSON Schemas for every tool in that toolset |
|
|
| `call_tool` | toolset/tool name + arguments object | The tool's result, same turn |
|
|
|
|
Discipline:
|
|
|
|
- `list_toolsets` once per session; re-run only after `RefreshTools`, plugin
|
|
changes, or reconnect.
|
|
- `describe_toolset` before first use of any toolset. Parameter names, types,
|
|
and required fields come from the schema — never from memory or this file.
|
|
- Results: primitive results arrive wrapped as `{"result": ...}` (CVar
|
|
`ModelContextProtocol.WrapPODToolResultsInObject`, default true).
|
|
Structured results serialize with field-level schema.
|
|
- Errors come back as tool-call errors with the engine-side message — read
|
|
them; they usually name the offending parameter or missing asset.
|
|
|
|
Eager mode (`Enable Tool Search` off) advertises every tool individually.
|
|
Under Hermes that means each tool becomes `mcp_unreal_engine_<tool_name>` at
|
|
session start, and `hermes mcp configure unreal-engine` can prune the list.
|
|
Schema payload grows with every registered toolset, and tool authors are told
|
|
NOT to rely on eager advertising — stay in tool-search mode unless a very
|
|
small fixed surface is wanted.
|
|
|
|
## call_tool dispatch semantics (live-verified, 5.8)
|
|
|
|
Verified against a running 5.8 server; these details are where naive
|
|
clients die:
|
|
|
|
- `list_toolsets` returns **fully-qualified** toolset names — Python:
|
|
`editor_toolset.toolsets.scene.SceneTools`; C++:
|
|
`EditorToolset.EditorAppToolset`. Epic's prose says "SceneTools"; the
|
|
registry speaks qualified names. Use them verbatim in `describe_toolset`
|
|
and `call_tool`'s `toolset_name`.
|
|
- `tool_name` must be the **short** name (`get_current_level`,
|
|
`CaptureViewport`). Passing the fully-qualified tool name fails with
|
|
"Unknown tool" even though `describe_toolset` displays qualified names.
|
|
- `call_tool` args: `{"toolset_name": ..., "tool_name": ..., "arguments":
|
|
{...}}`; result returns on the same turn (the HTTP response blocks until
|
|
the game thread finishes the call).
|
|
- **`TOptional` parameters must be passed explicitly as `null`** — omitting
|
|
them errors with `input param "X" needs a default value`. E.g.
|
|
`CaptureViewport` minimal call is `{"captureTransform": null,
|
|
"annotations": null, "bShowUI": false}`.
|
|
- **Schema `required` is literal.** `find_actors` marks `name`, `tag`,
|
|
`collision_channels` required even though they're semantically optional —
|
|
pass `""` / `[]` to mean "any".
|
|
- **Property names are camelCase with UE's `b` prefix intact** at this
|
|
reflection layer: `bUseTemperature`, `bAtmosphereSunLight`, `fogDensity`,
|
|
`bRealTimeCapture`, `mobility`. Writing `useTemperature` does NOT error
|
|
the whole call — the response names each property that could not be set
|
|
(schema-in-error style; READ error text, it lists the exact failures and
|
|
often the full input schema).
|
|
- **Object references travel as `{"refPath": "<soft object path>"}`**
|
|
everywhere (actors, classes, components). Class refs use
|
|
`/Script/Module.Class` (e.g. `/Script/Engine.PointLight`); actor refs are
|
|
the full path (`/Temp/Untitled_1.Untitled_1:PersistentLevel.DirectionalLight_UAID_...`).
|
|
Spawn/find tools RETURN refPaths — capture and reuse them.
|
|
- **`ObjectTools.set_properties` takes `values` as a JSON *string***, not
|
|
an object: `{"instance": {"refPath": ...}, "values":
|
|
"{\"intensity\": 10.0}"}`. `get_properties` likewise returns a JSON
|
|
string inside `returnValue`. Double-encode/decode accordingly.
|
|
- Primitive results arrive wrapped as `{"returnValue": ...}` inside the
|
|
text content block.
|
|
|
|
### HTTP wire behavior (for raw clients / debugging)
|
|
|
|
- `initialize` → plain JSON response + `Mcp-Session-Id` header you must
|
|
echo on every subsequent request; `notifications/initialized` → 202
|
|
empty; `tools/call` → **`text/event-stream`**: the result arrives as an
|
|
`event: message` + `data: <jsonrpc>` frame only when the game thread
|
|
finishes. A client that treats the response as plain JSON reads an empty
|
|
body. Send `Accept: application/json, text/event-stream` always.
|
|
|
|
## Shipped toolsets
|
|
|
|
The registry is project-dependent; `describe_toolset` on the live server is
|
|
the only source of truth for schemas. The core surface below is verified
|
|
against EditorToolset's source in the 5.8 install (Python:
|
|
`.../EditorToolset/Content/Python/editor_toolset/toolsets/`; C++:
|
|
`EditorAppToolset.h`).
|
|
|
|
**EditorToolset plugin (the core), Python toolsets** (live-verified on 5.8;
|
|
qualified prefix `editor_toolset.toolsets.<module>.<Class>`):
|
|
|
|
| Toolset | Verified tools (subset) |
|
|
|---|---|
|
|
| `scene.SceneTools` | `load_level`, `get_current_level`, `find_actors` (by name/type/tag/bounds), `add_to_scene_from_class`, `add_to_scene_from_asset`, `remove_from_scene`, `save_actor`, `create_level_instance`, folders |
|
|
| `actor.ActorTools` | `get_label`/`set_label`, tags, `get_actor_transform`/`set_actor_transform` (`xform` fields optional = "don't change"), parenting, components |
|
|
| `primitive.PrimitiveTools` | `add_cube` (dimensions), `add_sphere` (radius), `add_cylinder`/`add_cone` (radius+height) — adds StaticMeshComponents with `local_transform` to a host actor: spawn `/Script/Engine.Actor`, then compose. The fastest blocking path, zero asset dependencies |
|
|
| `object.ObjectTools` | `list_properties` (returns full JSON schema of every property), `get_properties`/`set_properties` (JSON-string `values`), `reset_properties` (restore defaults — also your rollback), `get_class`, `search_subclasses` |
|
|
| `material_instance.MaterialInstanceTools` | `create`, `list_parameters`, `get/set_scalar_parameter`, `get/set_vector_parameter` |
|
|
| `asset.AssetTools` | `find_assets`, `load_asset`, `exists`, `save_assets`, `is_dirty`, `get_dependencies`/`get_referencers` (check before delete!), `delete`, `move`, `duplicate`, folders, `read_file`/`write_file` (project-scoped) |
|
|
| `blueprint.BlueprintTools` (+ dsl/layout/node) | Blueprint authoring |
|
|
| `material.MaterialTools`, `static_mesh.StaticMeshTools`, `texture.TextureTools`, `data_table.DataTableTools`, … | per-asset-type operations |
|
|
| `programmatic.ProgrammaticToolset` | **the batching escape hatch** — see below |
|
|
|
|
**`EditorToolset.EditorAppToolset` (C++, same plugin) — the agent's eyes
|
|
(full live list):** `CaptureViewport`, `CaptureEditorImage`,
|
|
`CaptureAssetImage`, `GetCameraTransform`/`SetCameraTransform`,
|
|
`GetSelectedActors`/`SelectActors`/`FocusOnActors`/`GetVisibleActors`,
|
|
`WorldPosToScreenCoords`/`ScreenCoordsToWorld`,
|
|
`GetSelectedAssets`/`SelectAssets`,
|
|
`GetContentBrowserPath`/`SetContentBrowserPath`, `OpenEditorForAsset`,
|
|
`GetOpenAssets`, `SearchCVars`, `StartPIE`/`StopPIE`/`IsPIERunning`.
|
|
|
|
`CaptureViewport` specifics (live-verified): args `{"captureTransform":
|
|
<transform-or-null>, "annotations": <config-or-null>, "bShowUI": false}`.
|
|
Returns base64 PNG (decode + save it yourself) plus camera
|
|
location/rotation/FOV. `captureTransform` captures from any pose WITHOUT
|
|
moving the user's viewport — use it as a virtual camera. Annotation config
|
|
`{"gridSpacingCm": 500, "gridExtentCm": 3000, "gridHeight": <ground Z>,
|
|
"labelActors": true}` overlays a projected ground grid and actor callouts;
|
|
**grid coordinate labels are in METERS** (world cm ÷ 100). Use annotated
|
|
captures for placement work, clean ones for beauty checks.
|
|
|
|
Also confirmed live: `ToolsetRegistry.AgentSkillToolset`,
|
|
`EditorToolset.LogsToolset` (read Output Log + set verbosity — useful for
|
|
self-debugging), `SemanticSearchToolset` (hybrid vector+BM25 asset search),
|
|
five `NiagaraToolsets.NiagaraToolset_*` groups, `PCGToolset` (+Spatial),
|
|
`UMGToolSet`, three `GASToolsets.*`, `AutomationTestToolset`,
|
|
`ConfigSettingsToolset` (read/write Project Settings & Editor Preferences
|
|
sections by schema — the remote path to exposure defaults, rendering
|
|
settings, etc.), `SlateInspectorToolset`, `PluginToolset`,
|
|
`animation_toolset.toolsets.sequencer.SequencerTools` + keyframing/
|
|
controlrig/outliner siblings, `aimodule_toolset` BehaviorTreeTools,
|
|
`state_tree_toolset` StateTreeTools, and more — 67 toolsets on a blank
|
|
project with AllToolsets enabled.
|
|
|
|
Known gap: no mesh-modelling tools — spawn/place/instance existing meshes,
|
|
yes; author new geometry, no. The supported route to parametric geometry is
|
|
a custom Python toolset wrapping **Geometry Script** (`UDynamicMesh`:
|
|
append box/cylinder/sphere, booleans, then `Create New Static Mesh Asset
|
|
from Mesh` to bake an `SM_` asset). For organic/sculpted meshes, model in
|
|
Blender and import.
|
|
|
|
First-session move: `list_toolsets`, then `describe_toolset` each group you
|
|
plan to use, and keep those schemas in working memory for the session.
|
|
|
|
## ProgrammaticToolset — sanctioned batching
|
|
|
|
The serial-call rule makes N-step edits slow over the wire. The shipped
|
|
answer is `ProgrammaticToolset` (verified in `programmatic.py`):
|
|
|
|
1. `get_execution_environment` — **mandatory first call** (the tool's own
|
|
docstring requires it); returns the allowed modules, script constraints,
|
|
and usage instructions.
|
|
2. `execute_tool_script(script)` — runs a **sandboxed** Python script that
|
|
defines `run() -> dict`. Inside, you call other registered tools
|
|
programmatically and glue them with logic — one MCP round-trip for a
|
|
whole loop (e.g. spawn 20 actors with computed transforms).
|
|
|
|
Sandbox facts (from source): allowed imports are `json`, `math`,
|
|
`datetime`, `copy`, `re`, `time` only; `open()` is restricted to
|
|
project-contained paths; scripts run inside an editor **transaction scope**
|
|
(undo-friendly); it is tool orchestration, NOT general Python — arbitrary
|
|
`unreal.*` calls are not the contract. Data returns via `run()`'s dict.
|
|
|
|
Use it whenever a recipe loop exceeds ~5 homogeneous calls; keep one-off
|
|
edits as plain `call_tool`.
|
|
|
|
## Project Agent Skills (AgentSkillToolset)
|
|
|
|
Projects and plugins can register **Agent Skills** — named instruction
|
|
bundles for project-specific conventions and workflows (naming schemes,
|
|
folder layout, canonical multi-step sequences). They are NOT listed by
|
|
`list_toolsets`; reach them through `call_tool`:
|
|
|
|
1. `AgentSkillToolset.ListSkills` → names + descriptions of registered
|
|
skills.
|
|
2. If one matches the task, `AgentSkillToolset.GetSkills` on it → full
|
|
instructions, then FOLLOW THEM — a project skill exists precisely
|
|
because the project's way differs from the obvious way, and it takes
|
|
precedence over this skill's generic defaults.
|
|
|
|
Check at the start of unfamiliar work in any project, not just once ever.
|
|
|
|
## Seeing your work: screenshots and captures
|
|
|
|
An agent that can't see the viewport is flying blind. In order of preference:
|
|
|
|
1. **`EditorAppToolset.CaptureViewport`** (confirmed shipped) — returns the
|
|
image through MCP as base64 PNG with camera metadata; supports capturing
|
|
from an arbitrary transform without disturbing the user's viewport, and
|
|
an optional annotation overlay (world-space meter grid + actor callouts)
|
|
for spatial-placement work. This is the default verification tool.
|
|
2. **Console `HighResShot` via any console/exec tool** when you need
|
|
resolutions beyond the viewport: `HighResShot 3840x2160` writes to
|
|
`<Project>/Saved/Screenshots/<Platform>/` on the EDITOR host's
|
|
filesystem; read the file back (same machine) with `vision_analyze`.
|
|
3. **Custom toolset escape hatch** for anything else (e.g. camera-actor
|
|
framed captures with MRQ-quality settings).
|
|
|
|
Always `vision_analyze` the capture and art-direct against the brief before
|
|
declaring a milestone done.
|
|
|
|
## Plugin configuration reference
|
|
|
|
Editor Preferences > General > Model Context Protocol:
|
|
|
|
| Property | Default | Notes |
|
|
|---|---|---|
|
|
| Auto Start Server | `false` | Turn on for frictionless sessions |
|
|
| Server Port Number | `8000` | Change on conflict; mirror in Hermes config url |
|
|
| Server URL Path | `/mcp` | Same |
|
|
| Enable Tool Search | `true` | Keep on (see above) |
|
|
|
|
Console commands (editor console, backtick):
|
|
|
|
| Command | Effect |
|
|
|---|---|
|
|
| `ModelContextProtocol.StartServer [port]` | Start server (optional port override) |
|
|
| `ModelContextProtocol.StopServer` | Stop server, close all sessions |
|
|
| `ModelContextProtocol.RefreshTools` | Re-poll toolset providers — run after authoring/hot-reload/Game-Feature activation |
|
|
| `ModelContextProtocol.GenerateClientConfig <Client\|All>` | Write client config files (ClaudeCode/Cursor/VSCode/Gemini/Codex) — NOT used for Hermes |
|
|
|
|
Command-line flags for launching the editor pre-configured:
|
|
`-ModelContextProtocolStartServer` (force start regardless of preference),
|
|
`-ModelContextProtocolPort=N`.
|
|
|
|
Console variables:
|
|
|
|
| CVar | Default | Notes |
|
|
|---|---|---|
|
|
| `ModelContextProtocol.WrapPODToolResultsInObject` | `true` | Primitive results wrapped as `{"result": ...}` |
|
|
| `ModelContextProtocol.AudioResultOggFormat` | `false` | OGG instead of WAV for audio results |
|
|
| `ModelContextProtocol.ProgressIntervalSeconds` | `1.0` | Min interval between progress notifications |
|
|
| `ModelContextProtocol.PaginationPageSize` | `0` | 0 = no pagination of list results |
|
|
| `ModelContextProtocol.EnableAnalytics` | `true` | Epic telemetry gate |
|
|
|
|
## Debugging the connection
|
|
|
|
- **Output Log** at editor startup logs bind address/port/path — first stop
|
|
when the server seems absent. Port-in-use and missing-dependency failures
|
|
surface here.
|
|
- **Log verbosity:** `Log LogModelContextProtocol Verbose` in the editor
|
|
console.
|
|
- **MCP Inspector** (`npx @modelcontextprotocol/inspector`, point at
|
|
`http://127.0.0.1:8000/mcp`, transport "Streamable HTTP") lists every
|
|
advertised tool with schemas and offers form-based invocation — isolates
|
|
"server broken" from "agent calling it wrong".
|
|
- **After Live Coding / authoring:** connected clients can hold stale
|
|
schemas. `ModelContextProtocol.RefreshTools`, then reconnect (new Hermes
|
|
session) if schemas still look stale.
|
|
|
|
## Extending the surface: custom toolsets
|
|
|
|
When shipped tools don't cover an operation, the supported path is authoring
|
|
a project toolset — NOT trying to smuggle arbitrary code through unrelated
|
|
tools. Python toolsets are first-class and hot-loadable, so prefer them.
|
|
|
|
### Python toolset (recommended)
|
|
|
|
Any enabled plugin's `Content/Python/` directory (or the project's) can hold
|
|
toolset modules; the registry discovers them at startup. Shape (mirrors
|
|
Epic's shipped `ActorTools`):
|
|
|
|
```python
|
|
import unreal
|
|
import toolset_registry
|
|
|
|
@unreal.uclass()
|
|
class MySceneTools(unreal.ToolsetDefinition):
|
|
"""One-line toolset description — surfaces to the agent in list_toolsets."""
|
|
|
|
@toolset_registry.tool_call
|
|
@staticmethod
|
|
def take_viewport_screenshot(filename: str, width: int, height: int) -> str:
|
|
"""Capture the active viewport to Saved/Screenshots.
|
|
|
|
Args:
|
|
filename: Base filename without extension.
|
|
width: Output width in pixels.
|
|
height: Output height in pixels.
|
|
|
|
Returns:
|
|
Absolute path the screenshot will be written to.
|
|
"""
|
|
...
|
|
```
|
|
|
|
Conventions that matter (they generate the schema the agent sees):
|
|
|
|
- `@unreal.uclass()` on the class; inherit `unreal.ToolsetDefinition`.
|
|
- Class docstring = toolset description; write it for an agent audience.
|
|
- Each advertised function: `@toolset_registry.tool_call` + `@staticmethod`.
|
|
Functions without the decorator stay private.
|
|
- Type hints (`str`, `bool`, `list[str]`, `unreal.Actor`, dataclasses) drive
|
|
the JSON Schema; Google-style docstrings (`Args:`/`Returns:`) become the
|
|
parameter descriptions. Write them with API-surface care.
|
|
- Small, single-responsibility tools with structured return types beat
|
|
mega-tools returning prose. Data leaves the tool via its RETURN VALUE —
|
|
`print()`/stdout go to the UE log, not back over MCP.
|
|
|
|
After authoring: `ModelContextProtocol.RefreshTools` in the editor console,
|
|
then re-`list_toolsets` from Hermes. Users on Claude Code can scaffold with
|
|
the `create-toolset` skill from Epic's `unreal-mcp` plugin pack; the
|
|
conventions above still apply.
|
|
|
|
### C++ toolset
|
|
|
|
Derive from `UToolsetDefinition`, mark the class `UCLASS(BlueprintType,
|
|
Hidden)`, expose static `UFUNCTION(meta = (AICallable))` methods; doc
|
|
comments reflect into schemas. Use only when Python can't reach the API,
|
|
when reflected `USTRUCT` signatures are needed, or when the Python boundary
|
|
cost matters. Exclude a function with `meta = (AIIgnore)`. Live Coding
|
|
propagates edited function bodies, but NEW `UFUNCTION`s require a full
|
|
editor restart. There is also a direct-registration path
|
|
(`IModelContextProtocolTool` + `IModelContextProtocolModule::AddTool()`) for
|
|
runtime-shaped tools; caller owns deregistration.
|
|
|
|
## Runtime and cooked builds
|
|
|
|
The server is editor-hosted by default but not editor-only: runtime modules
|
|
can host it in cooked builds via `IModelContextProtocolModule::StartServer()`.
|
|
The Toolset Registry adapter (and the three tool-search meta-tools) are
|
|
editor-only, though — cooked-build tools must be registered explicitly
|
|
through `AddTool()` and are advertised eagerly. MCP Resources and Prompts are
|
|
not advertised by any shipping toolset.
|
|
|
|
## Known limitations (5.8, experimental)
|
|
|
|
- HTTP + SSE transports only; loopback-only listener; non-loopback `Origin`
|
|
headers rejected; no auth layer. Not safe beyond the local machine.
|
|
- Serial game-thread execution: overlapping calls unsupported; editor UI
|
|
blocks during each call.
|
|
- Feature-incomplete by Epic's own labeling; APIs and data formats subject
|
|
to change without notice.
|
|
- Live Coding does not propagate new `UFUNCTION` declarations.
|