mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-11-17 04:11:35 +00:00
* fix(relay): re-dial once with a fresh token before treating a 4401 as revocation A 4401 close after a successful handshake was read unconditionally as the connector having revoked this gateway's per-gateway secret (opt-out), so the transport latched auth_revoked, the adapter went relay_disabled, and all messaging stopped until a manual restart. But the connector sends the same plain 4401 'unauthorized' for an EXPIRED upgrade token (make_upgrade_token TTL is 300s). Scale-to-zero makes that routine: the instance is suspended while a re-dial is in flight, the token was minted before the freeze, the dial completes on resume with a token past its TTL, the connector refuses it with 4401, and the gateway misreads an expired token as a revoked credential. Production incident 2026-09-02 - the connector DB secret was never revoked. Fix, in gateway/relay/ws_transport.py: - The first post-handshake 4401 is provisional. The reader schedules ONE immediate re-dial (_redial_with_fresh_token) that bypasses the reconnect backoff; _dial_and_start mints a fresh token on every call. The backoff supervisor design is untouched. - Only a 4401 against that fresh token (either refused at the upgrade or closed after a descriptor on that connection, tracked by dial generation) latches auth_revoked. Terminal behaviour is otherwise unchanged. - If the fresh dial fails for a non-auth reason, hand off to the normal backoff supervisor. - Read the Close frame reason (_close_reason_of, sibling of _close_code_of). A 4401 whose reason is exactly 'expired' never latches revocation and takes the normal reconnect path - forward-compatible hook for the connector change landing separately. - disconnect() cancels the one-shot retry task alongside the supervisor. Tests (tests/gateway/relay/test_ws_transport.py, real websockets server): - 4401 once, next dial accepted -> reconnected, auth_revoked False, exactly 2 dials. - 4401 on every dial -> auth_revoked True after exactly one retry (2 dials), no supervisor, no further dials. - 4401 reason 'expired' (once and repeated) -> never latched, reconnects via the normal supervisor. - Existing 7d-B tests keep passing (4401 before any handshake stays retryable; the revoking-every-dial stub still latches). * fix(relay): tolerate a partially built transport in disconnect() teardown Three teardown tests construct WebSocketRelayTransport via object.__new__ without __init__, so the new _auth_retry attribute was absent and disconnect() raised AttributeError. Read it with getattr like the other optional teardown handles. * fix(relay): one live dialer — the fresh-token retry is a flag on the next dial, not a second dialer Review (round 1) found a race: a reader that dies with a provisional 4401 while the backoff supervisor is already mid-dial started _redial_with_fresh_token as a SECOND concurrent dialer; both installed sockets/readers and the supervisor could overwrite the accepted retry socket (3 dials, wrong socket). Now a provisional 4401 sets _auth_retry_pending; _dial_and_start consumes it and stamps _auth_retry_generation on whichever dialer performs the next dial. The reader arms a dialer only when none is live (_dialer_running), and an upgrade-time 4401 on that generation latches revocation from either dialer (_latch_if_fresh_token_refused). The reader also iterates its captured ws handle, not self._ws. Regression test reproduces the race with a deterministic fake connect; it fails with the _dialer_running guard removed. * fix(relay): a dial whose reader died mid-hello is a failed dial; the retry marker survives network failures Review round 2: - BLOCKER: with one-live-dialer, a reader that dies while its own dialer is still inside _dial_and_start (hello in flight) arms nothing — that is the dialer's job — but the dialer then returned 'connected', leaving no socket, no reader, no dialer. _dial_and_start now raises ConnectionError when the reader it installed has already finished, so both dialers take their normal failure path (retry -> supervisor; supervisor -> backoff). - MAJOR: _auth_retry_pending was consumed before websockets.connect, so a connect-time network failure un-marked the retry and the NEXT dial's real fresh-token 4401 read as another first strike (revocation never latched). The marker is now consumed only when the token reaches an auth outcome: upgrade accepted (stamp the generation) or upgrade 4401'd (judge it). Two regression tests, each mutation-checked red against its own guard.
565 lines
21 KiB
Python
565 lines
21 KiB
Python
"""WebSocketRelayTransport against a real in-process WebSocket server.
|
|
|
|
Exercises the production transport over an actual ``websockets`` server (no
|
|
mock socket): handshake (hello -> descriptor), inbound frame -> handler,
|
|
outbound request/response correlation, and follow_up routing. Proves the wire
|
|
framing (newline-delimited JSON) and the request/response future plumbing work
|
|
end to end on a live socket.
|
|
|
|
Skipped cleanly if the optional ``websockets`` dependency is absent.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
import json
|
|
|
|
import pytest
|
|
import pytest_asyncio
|
|
|
|
from gateway.relay.ws_transport import WebSocketRelayTransport, WEBSOCKETS_AVAILABLE
|
|
|
|
pytestmark = pytest.mark.skipif(not WEBSOCKETS_AVAILABLE, reason="websockets not installed")
|
|
|
|
if WEBSOCKETS_AVAILABLE:
|
|
import websockets
|
|
|
|
|
|
DESCRIPTOR = {
|
|
"contract_version": 1,
|
|
"platform": "discord",
|
|
"label": "Discord",
|
|
"max_message_length": 2000,
|
|
"supports_draft_streaming": False,
|
|
"supports_edit": True,
|
|
"supports_threads": True,
|
|
"markdown_dialect": "discord",
|
|
"len_unit": "chars",
|
|
}
|
|
|
|
|
|
class _StubConnectorServer:
|
|
"""Minimal connector: answers hello with a descriptor, echoes outbound."""
|
|
|
|
def __init__(self):
|
|
self.received: list[dict] = []
|
|
self._server = None
|
|
self.url = ""
|
|
# Push channel: tests set this to a frame dict to deliver inbound.
|
|
self._to_push: list[dict] = []
|
|
|
|
async def start(self):
|
|
self._server = await websockets.serve(self._handle, "127.0.0.1", 0)
|
|
sock = next(iter(self._server.sockets))
|
|
port = sock.getsockname()[1]
|
|
self.url = f"ws://127.0.0.1:{port}"
|
|
|
|
async def stop(self):
|
|
if self._server is not None:
|
|
self._server.close()
|
|
await self._server.wait_closed()
|
|
|
|
async def _handle(self, ws):
|
|
async for raw in ws:
|
|
for line in str(raw).split("\n"):
|
|
if not line.strip():
|
|
continue
|
|
frame = json.loads(line)
|
|
self.received.append(frame)
|
|
await self._on_frame(ws, frame)
|
|
|
|
async def _on_frame(self, ws, frame):
|
|
ftype = frame.get("type")
|
|
if ftype == "hello":
|
|
await ws.send(json.dumps({"type": "descriptor", "descriptor": DESCRIPTOR}) + "\n")
|
|
# Deliver any queued inbound frames right after handshake.
|
|
for f in self._to_push:
|
|
await ws.send(json.dumps(f) + "\n")
|
|
elif ftype == "outbound":
|
|
action = frame.get("action", {})
|
|
# Echo a successful result correlated by requestId.
|
|
result = {"success": True, "message_id": f"srv-{action.get('op')}"}
|
|
await ws.send(
|
|
json.dumps({"type": "outbound_result", "requestId": frame["requestId"], "result": result})
|
|
+ "\n"
|
|
)
|
|
|
|
|
|
@pytest_asyncio.fixture
|
|
async def server():
|
|
srv = _StubConnectorServer()
|
|
await srv.start()
|
|
yield srv
|
|
await srv.stop()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_handshake_negotiates_descriptor(server):
|
|
t = WebSocketRelayTransport(server.url, "discord", "appShared")
|
|
await t.connect()
|
|
try:
|
|
desc = await t.handshake()
|
|
assert desc.platform == "discord"
|
|
assert desc.max_message_length == 2000
|
|
# The hello carried the platform + botId.
|
|
hello = next(f for f in server.received if f["type"] == "hello")
|
|
assert hello["platform"] == "discord"
|
|
assert hello["botId"] == "appShared"
|
|
finally:
|
|
await t.disconnect()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_inbound_frame_reaches_handler(server):
|
|
server._to_push = [
|
|
{
|
|
"type": "inbound",
|
|
"event": {
|
|
"text": "hello from connector",
|
|
"message_type": "text",
|
|
"source": {"platform": "discord", "chat_id": "chan1", "chat_type": "group", "scope_id": "guildA"},
|
|
},
|
|
"bufferId": "buf-1",
|
|
}
|
|
]
|
|
received = []
|
|
t = WebSocketRelayTransport(server.url, "discord", "appShared")
|
|
t.set_inbound_handler(lambda ev: received.append(ev) or asyncio.sleep(0))
|
|
await t.connect()
|
|
try:
|
|
await t.handshake()
|
|
# Give the reader a tick to deliver the pushed inbound frame.
|
|
await asyncio.sleep(0.05)
|
|
assert len(received) == 1
|
|
assert received[0].text == "hello from connector"
|
|
assert received[0].source.scope_id == "guildA"
|
|
finally:
|
|
await t.disconnect()
|
|
|
|
|
|
# ── Phase 7 Unit 7d-B: terminal 4401 (opt-out revocation) ────────────────────
|
|
|
|
|
|
class _Revoking4401Server:
|
|
"""Connector stub that, on hello, optionally sends a descriptor and then
|
|
closes the socket with application code 4401 (unauthorized) — the shape of a
|
|
connector that has revoked this gateway's per-gateway secret (opt-out)."""
|
|
|
|
def __init__(self, *, send_descriptor_first: bool):
|
|
self._server = None
|
|
self.url = ""
|
|
self._send_descriptor_first = send_descriptor_first
|
|
|
|
async def start(self):
|
|
self._server = await websockets.serve(self._handle, "127.0.0.1", 0)
|
|
port = next(iter(self._server.sockets)).getsockname()[1]
|
|
self.url = f"ws://127.0.0.1:{port}"
|
|
|
|
async def stop(self):
|
|
if self._server is not None:
|
|
self._server.close()
|
|
await self._server.wait_closed()
|
|
|
|
async def _handle(self, ws):
|
|
async for raw in ws:
|
|
for line in str(raw).split("\n"):
|
|
if not line.strip():
|
|
continue
|
|
frame = json.loads(line)
|
|
if frame.get("type") == "hello":
|
|
if self._send_descriptor_first:
|
|
await ws.send(
|
|
json.dumps({"type": "descriptor", "descriptor": DESCRIPTOR}) + "\n"
|
|
)
|
|
# Let the descriptor flush + be processed before the close.
|
|
await asyncio.sleep(0.05)
|
|
# Close with 4401 (the connector's "unauthorized" close).
|
|
await ws.close(code=4401, reason="unauthorized")
|
|
return
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_4401_after_handshake_is_terminal_no_reconnect():
|
|
"""A 4401 close AFTER a successful handshake = a revoked credential (opt-out):
|
|
the transport latches auth_revoked and does NOT spin the reconnect supervisor.
|
|
|
|
Since the expired-token fix, the transport first re-dials ONCE with a fresh
|
|
token; this stub 4401s every dial, so that retry is also refused and the
|
|
latch still lands — the terminal contract is unchanged."""
|
|
srv = _Revoking4401Server(send_descriptor_first=True)
|
|
await srv.start()
|
|
try:
|
|
t = WebSocketRelayTransport(
|
|
srv.url, "discord", "appShared",
|
|
gateway_id="gw-x", upgrade_secret="secret-x",
|
|
reconnect=True, reconnect_backoff_s=0.05,
|
|
)
|
|
await t.connect()
|
|
await t.handshake() # records _handshake_succeeded
|
|
# Wait for the server's 4401 close to propagate through the read loop.
|
|
for _ in range(100):
|
|
if t.auth_revoked:
|
|
break
|
|
await asyncio.sleep(0.02)
|
|
assert t.auth_revoked is True
|
|
# Terminal: no reconnect supervisor was spawned.
|
|
assert t._supervisor is None
|
|
# Give a reconnect (if it were going to happen) time to NOT happen.
|
|
await asyncio.sleep(0.2)
|
|
assert t._supervisor is None
|
|
finally:
|
|
await t.disconnect()
|
|
await srv.stop()
|
|
|
|
|
|
# ── Expired upgrade token vs revoked secret (incident 2026-09-02) ─────────────
|
|
|
|
|
|
class _ExpiredToken4401Server:
|
|
"""Connector stub for the expired-vs-revoked ambiguity. Every dial gets a
|
|
descriptor on hello (the upgrade itself is never refused). After the
|
|
descriptor the stub closes 4401 on the dials whose 1-based index is in
|
|
``close_4401_on`` (``None`` = every dial) with the given reason; other dials
|
|
stay open and serve as a normal connector. ``dials`` counts connections."""
|
|
|
|
def __init__(self, *, close_4401_on: set[int] | None, reason: str = "unauthorized"):
|
|
self._server = None
|
|
self.url = ""
|
|
self.dials = 0
|
|
self._close_on = close_4401_on
|
|
self._reason = reason
|
|
|
|
async def start(self):
|
|
self._server = await websockets.serve(self._handle, "127.0.0.1", 0)
|
|
port = next(iter(self._server.sockets)).getsockname()[1]
|
|
self.url = f"ws://127.0.0.1:{port}"
|
|
|
|
async def stop(self):
|
|
if self._server is not None:
|
|
self._server.close()
|
|
await self._server.wait_closed()
|
|
|
|
async def _handle(self, ws):
|
|
self.dials += 1
|
|
dial = self.dials
|
|
async for raw in ws:
|
|
for line in str(raw).split("\n"):
|
|
if not line.strip():
|
|
continue
|
|
frame = json.loads(line)
|
|
if frame.get("type") == "hello":
|
|
await ws.send(json.dumps({"type": "descriptor", "descriptor": DESCRIPTOR}) + "\n")
|
|
if self._close_on is None or dial in self._close_on:
|
|
await asyncio.sleep(0.05) # let the descriptor land first
|
|
await ws.close(code=4401, reason=self._reason)
|
|
return
|
|
|
|
|
|
def _transport(url: str) -> WebSocketRelayTransport:
|
|
return WebSocketRelayTransport(
|
|
url, "discord", "appShared",
|
|
gateway_id="gw-x", upgrade_secret="secret-x",
|
|
reconnect=True, reconnect_backoff_s=5.0, # slow: proves the retry bypasses backoff
|
|
)
|
|
|
|
|
|
async def _wait_until(pred, *, timeout_s: float = 2.0) -> bool:
|
|
deadline = asyncio.get_running_loop().time() + timeout_s
|
|
while asyncio.get_running_loop().time() < deadline:
|
|
if pred():
|
|
return True
|
|
await asyncio.sleep(0.02)
|
|
return pred()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_4401_after_handshake_redials_once_with_fresh_token_and_recovers():
|
|
"""A single post-handshake 4401 (an expired upgrade token after a scale-to-zero
|
|
suspend) must NOT latch revocation: the transport re-dials immediately with a
|
|
fresh token, the connector accepts it, and the relay stays live."""
|
|
srv = _ExpiredToken4401Server(close_4401_on={1})
|
|
await srv.start()
|
|
t = _transport(srv.url)
|
|
try:
|
|
await t.connect()
|
|
await t.handshake()
|
|
assert srv.dials == 1
|
|
# The 4401 close lands, the fresh-token re-dial connects (well inside the
|
|
# 5s reconnect backoff, so this is the immediate retry, not the supervisor).
|
|
assert await _wait_until(lambda: srv.dials == 2 and t._descriptor is not None)
|
|
assert t.auth_revoked is False
|
|
# The new reader is running against a live socket; no fatal state.
|
|
assert t._reader is not None and not t._reader.done()
|
|
assert t._ws is not None
|
|
assert t._supervisor is None
|
|
await asyncio.sleep(0.2)
|
|
assert srv.dials == 2 # exactly one re-dial, no extra spinning
|
|
assert t.auth_revoked is False
|
|
finally:
|
|
await t.disconnect()
|
|
await srv.stop()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_4401_on_fresh_token_redial_latches_revocation_after_exactly_one_retry():
|
|
"""If the connector 4401s the FRESH token too, the secret really is gone:
|
|
latch auth_revoked after exactly one retry (2 dials total) and stop."""
|
|
srv = _ExpiredToken4401Server(close_4401_on=None)
|
|
await srv.start()
|
|
t = _transport(srv.url)
|
|
try:
|
|
await t.connect()
|
|
await t.handshake()
|
|
assert await _wait_until(lambda: t.auth_revoked)
|
|
assert srv.dials == 2
|
|
assert t._supervisor is None
|
|
await asyncio.sleep(0.2)
|
|
assert srv.dials == 2 # terminal: no further dials
|
|
assert t._supervisor is None
|
|
finally:
|
|
await t.disconnect()
|
|
await srv.stop()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_4401_with_reason_expired_never_latches_and_reconnects_normally():
|
|
"""A connector that labels the close reason 'expired' is explicitly saying the
|
|
token aged out: never a revocation, even repeatedly — take the normal
|
|
reconnect path (backoff supervisor) instead."""
|
|
srv = _ExpiredToken4401Server(close_4401_on={1}, reason="expired")
|
|
await srv.start()
|
|
t = _transport(srv.url)
|
|
t._reconnect_backoff_s = 0.05 # normal path: should reconnect via the supervisor
|
|
try:
|
|
await t.connect()
|
|
await t.handshake()
|
|
assert await _wait_until(lambda: t._supervisor is not None)
|
|
assert t.auth_revoked is False
|
|
assert await _wait_until(lambda: srv.dials == 2 and t._descriptor is not None)
|
|
assert t.auth_revoked is False
|
|
assert t._reader is not None and not t._reader.done()
|
|
finally:
|
|
await t.disconnect()
|
|
await srv.stop()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_4401_with_reason_expired_repeated_still_never_latches():
|
|
"""Even a second 'expired' 4401 in a row is not a revocation."""
|
|
srv = _ExpiredToken4401Server(close_4401_on={1, 2}, reason="expired")
|
|
await srv.start()
|
|
t = _transport(srv.url)
|
|
t._reconnect_backoff_s = 0.05
|
|
try:
|
|
await t.connect()
|
|
await t.handshake()
|
|
assert await _wait_until(lambda: srv.dials == 3 and t._descriptor is not None)
|
|
assert t.auth_revoked is False
|
|
finally:
|
|
await t.disconnect()
|
|
await srv.stop()
|
|
|
|
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_provisional_4401_never_starts_a_second_dialer_while_the_supervisor_is_mid_dial(monkeypatch):
|
|
"""Only ONE dialer may be live. A reader that dies with a provisional 4401
|
|
while the backoff supervisor is already mid-dial (socket installed, hello in
|
|
flight) must not start the fresh-token retry as a second concurrent dialer
|
|
— two dialers replace each other's socket/reader and can leave the
|
|
transport attached to the wrong socket. The supervisor's own re-dial carries
|
|
the fresh token instead, so exactly two dials happen in total."""
|
|
import gateway.relay.ws_transport as mod
|
|
|
|
class _Close4401(Exception):
|
|
code = 4401
|
|
reason = "unauthorized"
|
|
|
|
class _FirstWS:
|
|
"""The supervisor's first dial: its reader 4401s while hello is in flight,
|
|
then the hello itself fails (the connector dropped this connection)."""
|
|
|
|
def __init__(self, second_dial_done: asyncio.Event):
|
|
self.release_reader = asyncio.Event()
|
|
self.second_dial_done = second_dial_done
|
|
|
|
async def send(self, _data):
|
|
self.release_reader.set()
|
|
await asyncio.wait_for(self.second_dial_done.wait(), 1)
|
|
raise RuntimeError("dial lost during hello")
|
|
|
|
def __aiter__(self):
|
|
return self
|
|
|
|
async def __anext__(self):
|
|
await self.release_reader.wait()
|
|
raise _Close4401()
|
|
|
|
async def close(self):
|
|
pass
|
|
|
|
class _IdleWS:
|
|
def __init__(self):
|
|
self.stop = asyncio.Event()
|
|
|
|
async def send(self, _data):
|
|
pass
|
|
|
|
def __aiter__(self):
|
|
return self
|
|
|
|
async def __anext__(self):
|
|
await self.stop.wait()
|
|
raise StopAsyncIteration
|
|
|
|
async def close(self):
|
|
self.stop.set()
|
|
|
|
second_dial_done = asyncio.Event()
|
|
sockets = [_FirstWS(second_dial_done), _IdleWS(), _IdleWS()]
|
|
dials: list = []
|
|
|
|
async def fake_connect(*_args, **_kwargs):
|
|
ws = sockets[len(dials)]
|
|
dials.append(ws)
|
|
if len(dials) == 2:
|
|
second_dial_done.set()
|
|
return ws
|
|
|
|
monkeypatch.setattr(mod.websockets, "connect", fake_connect)
|
|
t = WebSocketRelayTransport(
|
|
"ws://unused", "discord", "bot", reconnect=True,
|
|
reconnect_backoff_s=0, reconnect_max_backoff_s=0,
|
|
)
|
|
t._handshake_succeeded = True
|
|
t._supervisor = asyncio.create_task(t._reconnect_loop())
|
|
try:
|
|
assert await _wait_until(lambda: len(dials) >= 2)
|
|
await asyncio.sleep(0.1) # give a racing second dialer time to show itself
|
|
assert len(dials) == 2, "a second dialer ran alongside the supervisor"
|
|
assert t._ws is sockets[1], "the transport must be attached to the live socket"
|
|
assert t._reader is not None and not t._reader.done()
|
|
assert not t.auth_revoked
|
|
# The supervisor's re-dial carried the fresh token: a 4401 on it is terminal.
|
|
assert t._auth_retry_generation == t._dial_generation
|
|
finally:
|
|
await t.disconnect(budget_s=0)
|
|
|
|
|
|
class _FakeCloseError(Exception):
|
|
"""Stand-in for websockets' ConnectionClosed: carries a close code + reason."""
|
|
|
|
def __init__(self, code: int, reason: str = ""):
|
|
super().__init__(f"closed {code} {reason}")
|
|
self.code = code
|
|
self.reason = reason
|
|
|
|
|
|
class _FakeWS:
|
|
"""Scriptable socket for dialer-ownership tests. ``hello`` decides what
|
|
happens when the transport sends its hello: a callable is awaited (and may
|
|
raise); ``reader`` decides what the reader sees: an exception to raise once
|
|
released, or None to idle until close()."""
|
|
|
|
def __init__(self, *, reader: Exception | None = None, hello=None):
|
|
self._reader_exc = reader
|
|
self._hello = hello
|
|
self.released = asyncio.Event()
|
|
self.stopped = asyncio.Event()
|
|
|
|
async def send(self, _data):
|
|
self.released.set()
|
|
if self._hello is not None:
|
|
await self._hello(self)
|
|
|
|
def __aiter__(self):
|
|
return self
|
|
|
|
async def __anext__(self):
|
|
await self.released.wait()
|
|
if self._reader_exc is not None:
|
|
raise self._reader_exc
|
|
await self.stopped.wait()
|
|
raise StopAsyncIteration
|
|
|
|
async def close(self):
|
|
self.released.set()
|
|
self.stopped.set()
|
|
|
|
|
|
def _scripted_connect(monkeypatch, outcomes: list):
|
|
"""Patch websockets.connect to pop ``outcomes`` in order: an Exception is
|
|
raised (connect-time failure), anything else is returned as the socket.
|
|
Returns the dial log."""
|
|
import gateway.relay.ws_transport as mod
|
|
|
|
dials: list = []
|
|
|
|
async def fake_connect(*_args, **_kwargs):
|
|
item = outcomes[len(dials)]
|
|
dials.append(item)
|
|
if isinstance(item, Exception):
|
|
raise item
|
|
return item
|
|
|
|
monkeypatch.setattr(mod.websockets, "connect", fake_connect)
|
|
return dials
|
|
|
|
|
|
def _fake_transport() -> WebSocketRelayTransport:
|
|
t = WebSocketRelayTransport(
|
|
"ws://unused", "discord", "bot", reconnect=True,
|
|
reconnect_backoff_s=0, reconnect_max_backoff_s=0,
|
|
)
|
|
t._handshake_succeeded = True
|
|
return t
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_fresh_token_retry_socket_dropping_during_hello_still_reconnects(monkeypatch):
|
|
"""Mirror of the supervisor race: the auth-retry is the only live dialer and
|
|
its fresh socket's reader dies (non-4401) after the hello was sent but before
|
|
the dial returns. The reader must not arm a dialer (one is live) — so the
|
|
DIAL itself must notice its reader is gone, report failure, and hand off to
|
|
the supervisor. Otherwise the transport ends with no socket, no reader and
|
|
no dialer: silently dead."""
|
|
first = _FakeWS(reader=_FakeCloseError(1006, "lost"))
|
|
|
|
async def hello_then_let_reader_die(ws):
|
|
# hello send succeeds; the reader (released by send) raises and finishes
|
|
# BEFORE _dial_and_start returns, so _ws is already None on return.
|
|
await asyncio.sleep(0.02)
|
|
|
|
first._hello = hello_then_let_reader_die
|
|
second = _FakeWS()
|
|
dials = _scripted_connect(monkeypatch, [first, second])
|
|
t = _fake_transport()
|
|
t._auth_retry_pending = True
|
|
t._auth_retry = asyncio.create_task(t._redial_with_fresh_token())
|
|
try:
|
|
assert await _wait_until(lambda: len(dials) == 2 and t._ws is second)
|
|
assert t._reader is not None and not t._reader.done()
|
|
assert not t.auth_revoked
|
|
finally:
|
|
await t.disconnect(budget_s=0)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_network_failure_on_the_retry_dial_does_not_unmark_it(monkeypatch):
|
|
"""The fresh-token marker must survive connect-time failures that never
|
|
reached authentication (network, timeout). If a blip consumed it, the NEXT
|
|
dial — the one whose fresh token actually gets judged — would be unmarked
|
|
and its 4401 read as another first strike, so a real revocation would never
|
|
latch (unbounded provisional retries)."""
|
|
refused = _FakeWS(reader=_FakeCloseError(4401, "unauthorized"))
|
|
dials = _scripted_connect(monkeypatch, [RuntimeError("network blip"), refused, _FakeWS(), _FakeWS()])
|
|
t = _fake_transport()
|
|
t._auth_retry_pending = True
|
|
t._auth_retry = asyncio.create_task(t._redial_with_fresh_token())
|
|
try:
|
|
assert await _wait_until(lambda: t.auth_revoked)
|
|
assert len(dials) == 2, "the 4401 on the first judged fresh token must be terminal"
|
|
assert not t._dialer_running()
|
|
finally:
|
|
await t.disconnect(budget_s=0)
|