searxng-mcp-go/docs/agents/e2e-tests.md
XLion 1afa9612c3
docs: align documentation with code behavior (issues #547-#551) (#554)
* docs: align documentation with code behavior (issues #547-#551)

- INSTALL.md: drop SEARXNG_MAX_RETRIES=-1 from the unparsable-value
  example list; cross-reference the hard-error semantic path
- PROMPT_INJECTION_SAFETY.md: mark Option A (warning header + warning
  field) as implemented, referencing #448 and #50
- MCP_TESTING.md: E2E binary is built once into a package-level cache,
  not per test; e2eMCPEnv lives in e2e_helper_test.go; goroutine-leak
  guard is suite-wide, not stress-file-specific
- e2e-tests.md: fix workflow line citations (:87, :104)
- verification.md: drop misleading '(except E2E)' qualifier; narrow the
  E2E-workflow coverage claim for exit-code tests
- test-server.md: add missing test-server-pid-helper recipe row
- .env.example: document SEARXNG_ALLOW_GET_FALLBACK opt-in
- MCP_TOOLS.md: include 'failed to execute search request' segment in
  the network-failure error format
- OUTPUT_FORMAT.md: sync Example 1 count line with the (showing N) format
- adr/014: suppression was relocated to searchWithRetries, not removed
- CONTEXT.md: note empty-text answers are dropped before the engine gate

* docs: address review findings on doc accuracy (P2 items)

- INSTALL.md: SEARXNG_MAX_RETRIES format is an integer in the range
  0-20, not 'non-negative integer' — -1 parses and is rejected by
  SetMaxRetries, so the old wording conflated syntax and semantics
- MCP_TESTING.md: example and E2E_MCP_BINARY section no longer claim a
  per-test build; list e2e_mockserver_test.go among the ungated CLI
  tests that build per-test via buildTestBinary
- CONTEXT.md: dedup description notes the early return when no
  non-empty infobox text exists (original answers returned unchanged)

* docs: fix remaining dedup and E2E_MCP_BINARY wording (P2)

- CONTEXT.md: example dialogue no longer claims non-DuckDuckGo
  answers are never touched; empty answers are dropped before the
  engine check when dedup runs, matching the Search Behavior entry
- test-server.md, verification.md: E2E_MCP_BINARY skips the MCP E2E
  package-level build, not the ungated CLI per-test builds

* docs: scope dedup pass-through claim to non-empty answers

SEARXNG_ANSWER_DEDUP.md and the DeduplicateAnswers doc comment both
claimed non-DuckDuckGo answers always pass through. In fact, when dedup
runs (non-empty infobox content exists), empty-text answers are dropped
before the engine gate; only non-empty non-DuckDuckGo answers pass
through. Aligns both with the implementation in deduplicate.go.

* docs: fix dedup algorithm order in SEARXNG_ANSWER_DEDUP.md

Document the empty-text filter running before the engine gate (matching
deduplicate.go) and add the early-return precondition; sync the Edge
Cases table so retain behavior is scoped to non-empty answers.

* docs: qualify empty-answer drop with early-return precondition

SEARXNG_ANSWER_DEDUP.md edge-case row and the deduplicateAnswers
wrapper comment both stated empty-text answers are always filtered.
When no non-empty infobox exists the implementation early-returns the
original answers unchanged; scope the drop to dedup runs only.
2026-08-11 05:20:47 +08:00

55 lines
3.2 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# E2E Tests
- Core functional tests assert non-zero results. If the live server returns only
an infobox with 0 web results, tests log a warning instead of failing, and
warnings are collected in a `WARNING SUMMARY` block at the end of the test
group (`TestMCPFunctional`, `TestMCPStdioE2E`, `TestCLISmoke`).
- Exceptions are owner-approved and documented. Known limitations of the CI
SearXNG instance — for example, the `files` category being excluded from
`TestMCPFunctional`'s category loop, or the `time_range=""` subtest
downgrading zero-result outcomes to a `WARNING SUMMARY` entry — must not be
broadened without approval. (The `time_range=""` subtest is named `"all"` in
`TestMCPFunctional`'s `"all time ranges"` subtest.) The `time_range="day"`,
`"month"`, and `"year"` subtests are expected to potentially return zero
results and log the outcome as `(persistent, expected)`; they are not
downgraded to warnings and are not part of the exception list.
- Adding strict assertions is preferred when a test can meaningfully assert
non-zero results with a better query or parameter combination.
- Live-server assertions belong in Go E2E tests (`TestMCPFunctional`,
`TestMCPStdioE2E`, `TestCLISmoke`) so they can use the shared warning-summary
path. Shell smoke in `.github/workflows/e2e.yml` must stay limited to
deterministic exit-code or structural checks and must not use `|| echo`,
`grep -Eq "A|B"`, swallowed `python3 -c` assertions, or similar patterns that
mask missing live-server evidence.
- For local server setup, use `just test-server-start`; never run
`searxng-server-test/01-start-fg.sh` from agents or CI. See
`docs/agents/test-server.md`.
## CI Retry Wrapper
`.github/workflows/e2e.yml` wraps all E2E test suites — those matching
`-run 'TestMCP'` (which includes `TestMCPFunctional`, `TestMCPStdioE2E`,
`TestMCPErrors_InvalidInputs`, `TestMCPLifecycle_SIGINTGracefulShutdown`,
`TestMCPLifecycle_SIGTERMGracefulShutdown`, and
`TestMCPLifecycle_InvalidJSONAfterInitialize`) and `-run 'TestCLISmoke'` — in
`Wandalen/wretry.action` with `attempt_limit=3` using
`-v -tags=e2e -run 'TestMCP' -race -count=1 -timeout=600s .` (see
`.github/workflows/e2e.yml:87` and `:104` for the exact commands).
This is because upstream search engines probabilistically rate-limit or ban
the CI SearXNG instance during automated runs — a characteristic inherent to
meta-search against live third-party engines, not a code or SearXNG bug.
Library-level retry (`SEARXNG_MAX_RETRIES=2`) covers single-request flakes;
the CI retry wrapper recovers from whole-instance throttling by retrying the
entire test run to land on a clean window. See the inline comment in
`e2e.yml` for the worst-case budget.
## Race Detector
E2E workflows (`e2e.yml`, `e2e-stress.yml`) run with `-race`. The race
detector adds non-trivial wall-clock overhead (typically 2–5× for a given
test binary), but this is the expected cost of catching data races at the
live-server layer — the only layer that exercises the real subprocess stdio
boundary, real SearXNG integration, and real MCP `CommandTransport` lifecycle.
The existing `-timeout` values include sufficient headroom for this overhead;
if runs become timeout-limited, increase them as needed.