hermes-agent/apps/desktop/electron/gateway-ws-probe.ts
brooklyn! b4bad6fca1 fix(desktop): gate the boot WS probe extension on child liveness, not output
Follow-up to the progress-aware probe (#96177):

- Wait while the spawned child is alive instead of "alive and wrote output
  in the last 30s". The stall this fixes is silent: the backend answers
  HTTP, then holds the GIL importing gateway platform modules and prints
  nothing until the loop recovers (the web_server heartbeat only logs
  "event loop stalled" afterwards). Output recency counted down during
  exactly that window, so a long stall still failed. A live loopback child
  whose listener accepted the TCP connect is busy, not refusing; refusals
  and auth rejections are immediate error/close events, not timeouts.
  Drops the now-unused BackendOutputTail.lastActivityAt/backendMakingProgress.
- One policy home: spawnedBackendProbeOptions() in gateway-ws-probe.ts
  (base 10s, cap 90s = the port-announcement cold-start budget); both the
  primary and pool spawn paths use it. Remote gateways, "Test remote" and
  host-backend attach keep the fixed budget.
- Without keepWaitingWhile the probe is again a single one-shot timer (no
  1s polling during the base budget). Timeout reasons name what ended the
  wait: base budget, backend stopped after the budget, or the cap.
- Tests drive every deadline with fake timers (no wall-clock races) and
  cover the production policy: a 30s stall succeeds while alive, a child
  that exits mid-wait fails at the next check, a live backend that never
  answers fails at the cap.
2026-09-23 18:28:20 -05:00

331 lines
11 KiB
TypeScript

/**
* Live WebSocket validation for the remote-gateway "Test remote" button.
*
* Background: the desktop boot does two independent things to a remote gateway:
*
* 1. The MAIN process hits ``GET /api/status`` over HTTP (token in a header)
* to confirm the backend is up. This is what "Test remote" historically
* checked, and what the boot logs print as "Remote Hermes backend is
* ready".
* 2. The RENDERER then opens a live WebSocket to ``/api/ws`` (credential in a
* query param) via ``gateway.connect()``. The chat surface only works once
* THIS succeeds.
*
* Those two paths use different processes, transports, and credentials, and the
* server applies extra guards to the WS upgrade that the HTTP status route never
* sees (Host/Origin checks, ws-ticket/token auth, peer-IP checks). So a gateway
* can pass the HTTP status check yet reject the WebSocket — which surfaces to
* the user as a green "Test remote" followed by an opaque "Could not connect to
* Hermes gateway" on the boot overlay.
*
* This module performs the second half of the check: it actually opens the WS
* URL and confirms the upgrade is accepted (and isn't immediately torn down by
* a post-upgrade auth rejection). The ``WebSocketImpl`` is injectable so the
* unit tests can drive the handshake without a real socket; in production the
* caller passes the Node/Electron global ``WebSocket``.
*
* The boot-time probe of a backend child this app spawned also uses it, with
* ``keepWaitingWhile``: a Windows cold start can stall the backend's event
* loop for 12-28s after HTTP is up, well past the fixed 10s budget (#96177).
* See ``spawnedBackendProbeOptions``.
*/
const DEFAULT_CONNECT_TIMEOUT_MS = 10_000
// After the upgrade is accepted, a gateway that rejects the credential
// post-handshake closes the socket almost immediately. Wait a short grace
// window: a frame (gateway.ready) or a still-open socket means success; an
// early close means the upgrade was accepted but the session was refused.
const DEFAULT_READY_GRACE_MS = 750
// Past the base budget, a `keepWaitingWhile` probe re-checks on this cadence.
const DEFAULT_PROGRESS_CHECK_INTERVAL_MS = 1_000
/**
* Attempt a live WebSocket connection and classify the outcome.
*
* @param {string} wsUrl - Fully-formed ws(s):// URL including the credential.
* @returns {Promise<{ ok: boolean, reason?: string }>}
*/
function probeGatewayWebSocket<T>(
wsUrl: string,
options: {
WebSocketImpl?: any
connectTimeoutMs?: number
readyGraceMs?: number
/**
* Consulted once `connectTimeoutMs` passes, then every
* `progressCheckIntervalMs`: while it returns true the probe keeps
* waiting instead of failing, never past `maxConnectWaitMs`. Omitted, the
* probe is a single fixed `connectTimeoutMs` deadline. A throwing
* callback counts as false (fail closed).
*/
keepWaitingWhile?: () => boolean
/** Cadence at which the connect deadline is re-evaluated (default 1s). */
progressCheckIntervalMs?: number
/** Hard cap on the total wait; defaults to `connectTimeoutMs` (no extension). */
maxConnectWaitMs?: number
/** Extra upgrade-request headers (access-proxy credentials such as
* Cloudflare Access service tokens). Passed as the non-standard second
* constructor argument `{ headers }` that Node's (undici) WebSocket —
* the impl the Electron main process supplies — understands. Without
* this the probe would dial the bare upgrade and fail against a gateway
* the renderer (whose upgrade gets headers injected via
* webRequest.onBeforeSendHeaders) can actually reach — the exact
* false-negative the probe exists to prevent. */
headers?: Record<string, string>
} = {}
) {
const WebSocketImpl = options.WebSocketImpl
const connectTimeoutMs = options.connectTimeoutMs ?? DEFAULT_CONNECT_TIMEOUT_MS
const readyGraceMs = options.readyGraceMs ?? DEFAULT_READY_GRACE_MS
const progressCheckIntervalMs = options.progressCheckIntervalMs ?? DEFAULT_PROGRESS_CHECK_INTERVAL_MS
const keepWaitingWhile = options.keepWaitingWhile
const maxConnectWaitMs = options.maxConnectWaitMs ?? connectTimeoutMs
const headers = options.headers && Object.keys(options.headers).length > 0 ? options.headers : null
if (typeof WebSocketImpl !== 'function') {
return Promise.resolve({
ok: false,
reason: 'WebSocket is not available in this runtime.'
})
}
return new Promise<any>(resolve => {
let settled = false
let opened = false
let connectTimer = null
let graceTimer = null
let socket
const clearTimers = () => {
if (connectTimer !== null) {
clearTimeout(connectTimer)
connectTimer = null
}
if (graceTimer !== null) {
clearTimeout(graceTimer)
graceTimer = null
}
}
const finish = result => {
if (settled) {
return
}
settled = true
clearTimers()
try {
socket?.close?.()
} catch {
// ignore — best effort teardown
}
resolve(result)
}
try {
socket = headers ? new WebSocketImpl(wsUrl, { headers }) : new WebSocketImpl(wsUrl)
} catch (error) {
finish({
ok: false,
reason: error instanceof Error ? error.message : String(error)
})
return
}
const onOpen = () => {
if (settled) {
return
}
opened = true
// Upgrade accepted. Give the server a brief window to reject the
// credential post-handshake (early close) before declaring success.
graceTimer = setTimeout(() => {
finish({ ok: true })
}, readyGraceMs)
}
const onMessage = () => {
// Any frame means the gateway accepted us and is talking — unambiguous
// success, no need to wait out the grace window.
finish({ ok: true })
}
const onError = event => {
finish({
ok: false,
reason: extractErrorReason(event) || 'WebSocket connection failed.'
})
}
const onClose = event => {
if (settled) {
return
}
if (opened) {
// Opened, then closed inside the grace window: the upgrade was accepted
// but the session was refused (e.g. ws-ticket/token rejected, or a
// server-side Host/Origin guard tripped after accept).
finish({
ok: false,
reason: closeReason(event, 'The gateway accepted the connection then closed it (credential rejected?).')
})
return
}
finish({
ok: false,
reason: closeReason(event, 'The gateway closed the WebSocket before it opened.')
})
}
addListener(socket, 'open', onOpen)
addListener(socket, 'message', onMessage)
addListener(socket, 'error', onError)
addListener(socket, 'close', onClose)
if (connectTimeoutMs > 0) {
// A healthy gateway upgrades in well under a second, so the base budget
// stays short. With `keepWaitingWhile` (a locally spawned backend,
// #96177) the base deadline becomes a checkpoint: while the callback
// says the backend is still there, re-check on a cadence, never past
// `maxConnectWaitMs`. Without it this is the original one-shot timer.
const startedAt = Date.now()
const hardCapMs = Math.max(connectTimeoutMs, maxConnectWaitMs)
const shouldKeepWaiting = () => {
try {
return Boolean(keepWaitingWhile?.())
} catch {
// A throwing check must never hold the probe open: fail closed.
return false
}
}
const check = () => {
if (settled) {
return
}
const elapsedMs = Date.now() - startedAt
if (!shouldKeepWaiting()) {
finish({
ok: false,
reason:
elapsedMs > connectTimeoutMs
? `Timed out after ${elapsedMs}ms waiting for the WebSocket to open (backend stopped after the ${connectTimeoutMs}ms budget).`
: `Timed out after ${connectTimeoutMs}ms waiting for the WebSocket to open.`
})
return
}
if (elapsedMs >= hardCapMs) {
finish({
ok: false,
reason: `Timed out after ${elapsedMs}ms waiting for the WebSocket to open (backend still running at the ${hardCapMs}ms cap).`
})
return
}
connectTimer = setTimeout(check, Math.min(progressCheckIntervalMs, hardCapMs - elapsedMs))
}
connectTimer = setTimeout(check, connectTimeoutMs)
}
})
}
function addListener(socket, type, handler) {
if (typeof socket.addEventListener === 'function') {
socket.addEventListener(type, handler)
return
}
// Node's global WebSocket implements addEventListener; this fallback keeps the
// helper usable with the `ws` package's EventEmitter shape too.
if (typeof socket.on === 'function') {
socket.on(type, handler)
}
}
function extractErrorReason(event) {
if (!event) {
return ''
}
if (event instanceof Error) {
return event.message
}
const err = event.error || event.message
if (err instanceof Error) {
return err.message
}
if (typeof err === 'string') {
return err
}
return ''
}
function closeReason(event, fallback) {
const code = event && typeof event.code === 'number' ? event.code : null
const reason = event && typeof event.reason === 'string' ? event.reason.trim() : ''
if (code && reason) {
return `${fallback} (code ${code}: ${reason})`
}
if (code) {
return `${fallback} (code ${code})`
}
if (reason) {
return `${fallback} (${reason})`
}
return fallback
}
// Boot-time probe policy for a backend child THIS app spawned (#96177). On a
// Windows cold start the backend answers HTTP, then holds the GIL for 12-28s
// importing gateway platform modules; the event loop (and so the WS upgrade)
// is stalled and the process prints nothing until it recovers — the
// web_server loop heartbeat only logs "event loop stalled" afterwards. So the
// honest signal is liveness: a live loopback child whose listener accepted
// the TCP connect but has not answered the upgrade is busy, not refusing (a
// refusal or auth rejection is an immediate error/close, not a timeout). A
// dead child still fails at the base budget; a live-but-wedged one fails at
// the cap, which reuses the port-announcement cold-start budget
// (DEFAULT_PORT_ANNOUNCE_TIMEOUT_MS, backend-ready.ts). Remote gateways, the
// "Test remote" button and host-backend attach keep the fixed budget.
const SPAWNED_BACKEND_MAX_CONNECT_WAIT_MS = 90_000
function spawnedBackendProbeOptions(isChildAlive: () => boolean) {
return {
connectTimeoutMs: DEFAULT_CONNECT_TIMEOUT_MS,
maxConnectWaitMs: SPAWNED_BACKEND_MAX_CONNECT_WAIT_MS,
keepWaitingWhile: isChildAlive
}
}
export {
DEFAULT_CONNECT_TIMEOUT_MS,
DEFAULT_PROGRESS_CHECK_INTERVAL_MS,
DEFAULT_READY_GRACE_MS,
probeGatewayWebSocket,
SPAWNED_BACKEND_MAX_CONNECT_WAIT_MS,
spawnedBackendProbeOptions
}