searxng-mcp-go/docs/agents/e2e-tests.md
XLion 1afa9612c3
docs: align documentation with code behavior (issues #547-#551) (#554)
* docs: align documentation with code behavior (issues #547-#551)

- INSTALL.md: drop SEARXNG_MAX_RETRIES=-1 from the unparsable-value
  example list; cross-reference the hard-error semantic path
- PROMPT_INJECTION_SAFETY.md: mark Option A (warning header + warning
  field) as implemented, referencing #448 and #50
- MCP_TESTING.md: E2E binary is built once into a package-level cache,
  not per test; e2eMCPEnv lives in e2e_helper_test.go; goroutine-leak
  guard is suite-wide, not stress-file-specific
- e2e-tests.md: fix workflow line citations (:87, :104)
- verification.md: drop misleading '(except E2E)' qualifier; narrow the
  E2E-workflow coverage claim for exit-code tests
- test-server.md: add missing test-server-pid-helper recipe row
- .env.example: document SEARXNG_ALLOW_GET_FALLBACK opt-in
- MCP_TOOLS.md: include 'failed to execute search request' segment in
  the network-failure error format
- OUTPUT_FORMAT.md: sync Example 1 count line with the (showing N) format
- adr/014: suppression was relocated to searchWithRetries, not removed
- CONTEXT.md: note empty-text answers are dropped before the engine gate

* docs: address review findings on doc accuracy (P2 items)

- INSTALL.md: SEARXNG_MAX_RETRIES format is an integer in the range
  0-20, not 'non-negative integer' — -1 parses and is rejected by
  SetMaxRetries, so the old wording conflated syntax and semantics
- MCP_TESTING.md: example and E2E_MCP_BINARY section no longer claim a
  per-test build; list e2e_mockserver_test.go among the ungated CLI
  tests that build per-test via buildTestBinary
- CONTEXT.md: dedup description notes the early return when no
  non-empty infobox text exists (original answers returned unchanged)

* docs: fix remaining dedup and E2E_MCP_BINARY wording (P2)

- CONTEXT.md: example dialogue no longer claims non-DuckDuckGo
  answers are never touched; empty answers are dropped before the
  engine check when dedup runs, matching the Search Behavior entry
- test-server.md, verification.md: E2E_MCP_BINARY skips the MCP E2E
  package-level build, not the ungated CLI per-test builds

* docs: scope dedup pass-through claim to non-empty answers

SEARXNG_ANSWER_DEDUP.md and the DeduplicateAnswers doc comment both
claimed non-DuckDuckGo answers always pass through. In fact, when dedup
runs (non-empty infobox content exists), empty-text answers are dropped
before the engine gate; only non-empty non-DuckDuckGo answers pass
through. Aligns both with the implementation in deduplicate.go.

* docs: fix dedup algorithm order in SEARXNG_ANSWER_DEDUP.md

Document the empty-text filter running before the engine gate (matching
deduplicate.go) and add the early-return precondition; sync the Edge
Cases table so retain behavior is scoped to non-empty answers.

* docs: qualify empty-answer drop with early-return precondition

SEARXNG_ANSWER_DEDUP.md edge-case row and the deduplicateAnswers
wrapper comment both stated empty-text answers are always filtered.
When no non-empty infobox exists the implementation early-returns the
original answers unchanged; scope the drop to dedup runs only.
2026-08-11 05:20:47 +08:00

3.2 KiB
Raw Permalink Blame History

E2E Tests

  • Core functional tests assert non-zero results. If the live server returns only an infobox with 0 web results, tests log a warning instead of failing, and warnings are collected in a WARNING SUMMARY block at the end of the test group (TestMCPFunctional, TestMCPStdioE2E, TestCLISmoke).
  • Exceptions are owner-approved and documented. Known limitations of the CI SearXNG instance — for example, the files category being excluded from TestMCPFunctional's category loop, or the time_range="" subtest downgrading zero-result outcomes to a WARNING SUMMARY entry — must not be broadened without approval. (The time_range="" subtest is named "all" in TestMCPFunctional's "all time ranges" subtest.) The time_range="day", "month", and "year" subtests are expected to potentially return zero results and log the outcome as (persistent, expected); they are not downgraded to warnings and are not part of the exception list.
  • Adding strict assertions is preferred when a test can meaningfully assert non-zero results with a better query or parameter combination.
  • Live-server assertions belong in Go E2E tests (TestMCPFunctional, TestMCPStdioE2E, TestCLISmoke) so they can use the shared warning-summary path. Shell smoke in .github/workflows/e2e.yml must stay limited to deterministic exit-code or structural checks and must not use || echo, grep -Eq "A|B", swallowed python3 -c assertions, or similar patterns that mask missing live-server evidence.
  • For local server setup, use just test-server-start; never run searxng-server-test/01-start-fg.sh from agents or CI. See docs/agents/test-server.md.

CI Retry Wrapper

.github/workflows/e2e.yml wraps all E2E test suites — those matching -run 'TestMCP' (which includes TestMCPFunctional, TestMCPStdioE2E, TestMCPErrors_InvalidInputs, TestMCPLifecycle_SIGINTGracefulShutdown, TestMCPLifecycle_SIGTERMGracefulShutdown, and TestMCPLifecycle_InvalidJSONAfterInitialize) and -run 'TestCLISmoke' — in Wandalen/wretry.action with attempt_limit=3 using -v -tags=e2e -run 'TestMCP' -race -count=1 -timeout=600s . (see .github/workflows/e2e.yml:87 and :104 for the exact commands). This is because upstream search engines probabilistically rate-limit or ban the CI SearXNG instance during automated runs — a characteristic inherent to meta-search against live third-party engines, not a code or SearXNG bug.

Library-level retry (SEARXNG_MAX_RETRIES=2) covers single-request flakes; the CI retry wrapper recovers from whole-instance throttling by retrying the entire test run to land on a clean window. See the inline comment in e2e.yml for the worst-case budget.

Race Detector

E2E workflows (e2e.yml, e2e-stress.yml) run with -race. The race detector adds non-trivial wall-clock overhead (typically 2–5× for a given test binary), but this is the expected cost of catching data races at the live-server layer — the only layer that exercises the real subprocess stdio boundary, real SearXNG integration, and real MCP CommandTransport lifecycle. The existing -timeout values include sufficient headroom for this overhead; if runs become timeout-limited, increase them as needed.