mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-11-17 04:11:35 +00:00
Follow-up to the progress-aware probe (#96177): - Wait while the spawned child is alive instead of "alive and wrote output in the last 30s". The stall this fixes is silent: the backend answers HTTP, then holds the GIL importing gateway platform modules and prints nothing until the loop recovers (the web_server heartbeat only logs "event loop stalled" afterwards). Output recency counted down during exactly that window, so a long stall still failed. A live loopback child whose listener accepted the TCP connect is busy, not refusing; refusals and auth rejections are immediate error/close events, not timeouts. Drops the now-unused BackendOutputTail.lastActivityAt/backendMakingProgress. - One policy home: spawnedBackendProbeOptions() in gateway-ws-probe.ts (base 10s, cap 90s = the port-announcement cold-start budget); both the primary and pool spawn paths use it. Remote gateways, "Test remote" and host-backend attach keep the fixed budget. - Without keepWaitingWhile the probe is again a single one-shot timer (no 1s polling during the base budget). Timeout reasons name what ended the wait: base budget, backend stopped after the budget, or the cap. - Tests drive every deadline with fake timers (no wall-clock races) and cover the production policy: a 30s stall succeeds while alive, a child that exits mid-wait fails at the next check, a live backend that never answers fails at the cap.
331 lines
11 KiB
TypeScript
331 lines
11 KiB
TypeScript
/**
|
|
* Live WebSocket validation for the remote-gateway "Test remote" button.
|
|
*
|
|
* Background: the desktop boot does two independent things to a remote gateway:
|
|
*
|
|
* 1. The MAIN process hits ``GET /api/status`` over HTTP (token in a header)
|
|
* to confirm the backend is up. This is what "Test remote" historically
|
|
* checked, and what the boot logs print as "Remote Hermes backend is
|
|
* ready".
|
|
* 2. The RENDERER then opens a live WebSocket to ``/api/ws`` (credential in a
|
|
* query param) via ``gateway.connect()``. The chat surface only works once
|
|
* THIS succeeds.
|
|
*
|
|
* Those two paths use different processes, transports, and credentials, and the
|
|
* server applies extra guards to the WS upgrade that the HTTP status route never
|
|
* sees (Host/Origin checks, ws-ticket/token auth, peer-IP checks). So a gateway
|
|
* can pass the HTTP status check yet reject the WebSocket — which surfaces to
|
|
* the user as a green "Test remote" followed by an opaque "Could not connect to
|
|
* Hermes gateway" on the boot overlay.
|
|
*
|
|
* This module performs the second half of the check: it actually opens the WS
|
|
* URL and confirms the upgrade is accepted (and isn't immediately torn down by
|
|
* a post-upgrade auth rejection). The ``WebSocketImpl`` is injectable so the
|
|
* unit tests can drive the handshake without a real socket; in production the
|
|
* caller passes the Node/Electron global ``WebSocket``.
|
|
*
|
|
* The boot-time probe of a backend child this app spawned also uses it, with
|
|
* ``keepWaitingWhile``: a Windows cold start can stall the backend's event
|
|
* loop for 12-28s after HTTP is up, well past the fixed 10s budget (#96177).
|
|
* See ``spawnedBackendProbeOptions``.
|
|
*/
|
|
|
|
const DEFAULT_CONNECT_TIMEOUT_MS = 10_000
|
|
// After the upgrade is accepted, a gateway that rejects the credential
|
|
// post-handshake closes the socket almost immediately. Wait a short grace
|
|
// window: a frame (gateway.ready) or a still-open socket means success; an
|
|
// early close means the upgrade was accepted but the session was refused.
|
|
const DEFAULT_READY_GRACE_MS = 750
|
|
// Past the base budget, a `keepWaitingWhile` probe re-checks on this cadence.
|
|
const DEFAULT_PROGRESS_CHECK_INTERVAL_MS = 1_000
|
|
|
|
/**
|
|
* Attempt a live WebSocket connection and classify the outcome.
|
|
*
|
|
* @param {string} wsUrl - Fully-formed ws(s):// URL including the credential.
|
|
* @returns {Promise<{ ok: boolean, reason?: string }>}
|
|
*/
|
|
function probeGatewayWebSocket<T>(
|
|
wsUrl: string,
|
|
options: {
|
|
WebSocketImpl?: any
|
|
connectTimeoutMs?: number
|
|
readyGraceMs?: number
|
|
/**
|
|
* Consulted once `connectTimeoutMs` passes, then every
|
|
* `progressCheckIntervalMs`: while it returns true the probe keeps
|
|
* waiting instead of failing, never past `maxConnectWaitMs`. Omitted, the
|
|
* probe is a single fixed `connectTimeoutMs` deadline. A throwing
|
|
* callback counts as false (fail closed).
|
|
*/
|
|
keepWaitingWhile?: () => boolean
|
|
/** Cadence at which the connect deadline is re-evaluated (default 1s). */
|
|
progressCheckIntervalMs?: number
|
|
/** Hard cap on the total wait; defaults to `connectTimeoutMs` (no extension). */
|
|
maxConnectWaitMs?: number
|
|
/** Extra upgrade-request headers (access-proxy credentials such as
|
|
* Cloudflare Access service tokens). Passed as the non-standard second
|
|
* constructor argument `{ headers }` that Node's (undici) WebSocket —
|
|
* the impl the Electron main process supplies — understands. Without
|
|
* this the probe would dial the bare upgrade and fail against a gateway
|
|
* the renderer (whose upgrade gets headers injected via
|
|
* webRequest.onBeforeSendHeaders) can actually reach — the exact
|
|
* false-negative the probe exists to prevent. */
|
|
headers?: Record<string, string>
|
|
} = {}
|
|
) {
|
|
const WebSocketImpl = options.WebSocketImpl
|
|
const connectTimeoutMs = options.connectTimeoutMs ?? DEFAULT_CONNECT_TIMEOUT_MS
|
|
const readyGraceMs = options.readyGraceMs ?? DEFAULT_READY_GRACE_MS
|
|
const progressCheckIntervalMs = options.progressCheckIntervalMs ?? DEFAULT_PROGRESS_CHECK_INTERVAL_MS
|
|
const keepWaitingWhile = options.keepWaitingWhile
|
|
const maxConnectWaitMs = options.maxConnectWaitMs ?? connectTimeoutMs
|
|
const headers = options.headers && Object.keys(options.headers).length > 0 ? options.headers : null
|
|
|
|
if (typeof WebSocketImpl !== 'function') {
|
|
return Promise.resolve({
|
|
ok: false,
|
|
reason: 'WebSocket is not available in this runtime.'
|
|
})
|
|
}
|
|
|
|
return new Promise<any>(resolve => {
|
|
let settled = false
|
|
let opened = false
|
|
let connectTimer = null
|
|
let graceTimer = null
|
|
let socket
|
|
|
|
const clearTimers = () => {
|
|
if (connectTimer !== null) {
|
|
clearTimeout(connectTimer)
|
|
connectTimer = null
|
|
}
|
|
|
|
if (graceTimer !== null) {
|
|
clearTimeout(graceTimer)
|
|
graceTimer = null
|
|
}
|
|
}
|
|
|
|
const finish = result => {
|
|
if (settled) {
|
|
return
|
|
}
|
|
|
|
settled = true
|
|
clearTimers()
|
|
|
|
try {
|
|
socket?.close?.()
|
|
} catch {
|
|
// ignore — best effort teardown
|
|
}
|
|
|
|
resolve(result)
|
|
}
|
|
|
|
try {
|
|
socket = headers ? new WebSocketImpl(wsUrl, { headers }) : new WebSocketImpl(wsUrl)
|
|
} catch (error) {
|
|
finish({
|
|
ok: false,
|
|
reason: error instanceof Error ? error.message : String(error)
|
|
})
|
|
|
|
return
|
|
}
|
|
|
|
const onOpen = () => {
|
|
if (settled) {
|
|
return
|
|
}
|
|
|
|
opened = true
|
|
// Upgrade accepted. Give the server a brief window to reject the
|
|
// credential post-handshake (early close) before declaring success.
|
|
graceTimer = setTimeout(() => {
|
|
finish({ ok: true })
|
|
}, readyGraceMs)
|
|
}
|
|
|
|
const onMessage = () => {
|
|
// Any frame means the gateway accepted us and is talking — unambiguous
|
|
// success, no need to wait out the grace window.
|
|
finish({ ok: true })
|
|
}
|
|
|
|
const onError = event => {
|
|
finish({
|
|
ok: false,
|
|
reason: extractErrorReason(event) || 'WebSocket connection failed.'
|
|
})
|
|
}
|
|
|
|
const onClose = event => {
|
|
if (settled) {
|
|
return
|
|
}
|
|
|
|
if (opened) {
|
|
// Opened, then closed inside the grace window: the upgrade was accepted
|
|
// but the session was refused (e.g. ws-ticket/token rejected, or a
|
|
// server-side Host/Origin guard tripped after accept).
|
|
finish({
|
|
ok: false,
|
|
reason: closeReason(event, 'The gateway accepted the connection then closed it (credential rejected?).')
|
|
})
|
|
|
|
return
|
|
}
|
|
|
|
finish({
|
|
ok: false,
|
|
reason: closeReason(event, 'The gateway closed the WebSocket before it opened.')
|
|
})
|
|
}
|
|
|
|
addListener(socket, 'open', onOpen)
|
|
addListener(socket, 'message', onMessage)
|
|
addListener(socket, 'error', onError)
|
|
addListener(socket, 'close', onClose)
|
|
|
|
if (connectTimeoutMs > 0) {
|
|
// A healthy gateway upgrades in well under a second, so the base budget
|
|
// stays short. With `keepWaitingWhile` (a locally spawned backend,
|
|
// #96177) the base deadline becomes a checkpoint: while the callback
|
|
// says the backend is still there, re-check on a cadence, never past
|
|
// `maxConnectWaitMs`. Without it this is the original one-shot timer.
|
|
const startedAt = Date.now()
|
|
const hardCapMs = Math.max(connectTimeoutMs, maxConnectWaitMs)
|
|
|
|
const shouldKeepWaiting = () => {
|
|
try {
|
|
return Boolean(keepWaitingWhile?.())
|
|
} catch {
|
|
// A throwing check must never hold the probe open: fail closed.
|
|
return false
|
|
}
|
|
}
|
|
|
|
const check = () => {
|
|
if (settled) {
|
|
return
|
|
}
|
|
|
|
const elapsedMs = Date.now() - startedAt
|
|
|
|
if (!shouldKeepWaiting()) {
|
|
finish({
|
|
ok: false,
|
|
reason:
|
|
elapsedMs > connectTimeoutMs
|
|
? `Timed out after ${elapsedMs}ms waiting for the WebSocket to open (backend stopped after the ${connectTimeoutMs}ms budget).`
|
|
: `Timed out after ${connectTimeoutMs}ms waiting for the WebSocket to open.`
|
|
})
|
|
|
|
return
|
|
}
|
|
|
|
if (elapsedMs >= hardCapMs) {
|
|
finish({
|
|
ok: false,
|
|
reason: `Timed out after ${elapsedMs}ms waiting for the WebSocket to open (backend still running at the ${hardCapMs}ms cap).`
|
|
})
|
|
|
|
return
|
|
}
|
|
|
|
connectTimer = setTimeout(check, Math.min(progressCheckIntervalMs, hardCapMs - elapsedMs))
|
|
}
|
|
|
|
connectTimer = setTimeout(check, connectTimeoutMs)
|
|
}
|
|
})
|
|
}
|
|
|
|
function addListener(socket, type, handler) {
|
|
if (typeof socket.addEventListener === 'function') {
|
|
socket.addEventListener(type, handler)
|
|
|
|
return
|
|
}
|
|
|
|
// Node's global WebSocket implements addEventListener; this fallback keeps the
|
|
// helper usable with the `ws` package's EventEmitter shape too.
|
|
if (typeof socket.on === 'function') {
|
|
socket.on(type, handler)
|
|
}
|
|
}
|
|
|
|
function extractErrorReason(event) {
|
|
if (!event) {
|
|
return ''
|
|
}
|
|
|
|
if (event instanceof Error) {
|
|
return event.message
|
|
}
|
|
|
|
const err = event.error || event.message
|
|
|
|
if (err instanceof Error) {
|
|
return err.message
|
|
}
|
|
|
|
if (typeof err === 'string') {
|
|
return err
|
|
}
|
|
|
|
return ''
|
|
}
|
|
|
|
function closeReason(event, fallback) {
|
|
const code = event && typeof event.code === 'number' ? event.code : null
|
|
const reason = event && typeof event.reason === 'string' ? event.reason.trim() : ''
|
|
|
|
if (code && reason) {
|
|
return `${fallback} (code ${code}: ${reason})`
|
|
}
|
|
|
|
if (code) {
|
|
return `${fallback} (code ${code})`
|
|
}
|
|
|
|
if (reason) {
|
|
return `${fallback} (${reason})`
|
|
}
|
|
|
|
return fallback
|
|
}
|
|
|
|
// Boot-time probe policy for a backend child THIS app spawned (#96177). On a
|
|
// Windows cold start the backend answers HTTP, then holds the GIL for 12-28s
|
|
// importing gateway platform modules; the event loop (and so the WS upgrade)
|
|
// is stalled and the process prints nothing until it recovers — the
|
|
// web_server loop heartbeat only logs "event loop stalled" afterwards. So the
|
|
// honest signal is liveness: a live loopback child whose listener accepted
|
|
// the TCP connect but has not answered the upgrade is busy, not refusing (a
|
|
// refusal or auth rejection is an immediate error/close, not a timeout). A
|
|
// dead child still fails at the base budget; a live-but-wedged one fails at
|
|
// the cap, which reuses the port-announcement cold-start budget
|
|
// (DEFAULT_PORT_ANNOUNCE_TIMEOUT_MS, backend-ready.ts). Remote gateways, the
|
|
// "Test remote" button and host-backend attach keep the fixed budget.
|
|
const SPAWNED_BACKEND_MAX_CONNECT_WAIT_MS = 90_000
|
|
|
|
function spawnedBackendProbeOptions(isChildAlive: () => boolean) {
|
|
return {
|
|
connectTimeoutMs: DEFAULT_CONNECT_TIMEOUT_MS,
|
|
maxConnectWaitMs: SPAWNED_BACKEND_MAX_CONNECT_WAIT_MS,
|
|
keepWaitingWhile: isChildAlive
|
|
}
|
|
}
|
|
|
|
export {
|
|
DEFAULT_CONNECT_TIMEOUT_MS,
|
|
DEFAULT_PROGRESS_CHECK_INTERVAL_MS,
|
|
DEFAULT_READY_GRACE_MS,
|
|
probeGatewayWebSocket,
|
|
SPAWNED_BACKEND_MAX_CONNECT_WAIT_MS,
|
|
spawnedBackendProbeOptions
|
|
}
|