searxng-mcp-go/docs/MCP_TESTING.md
XLion 1afa9612c3
docs: align documentation with code behavior (issues #547-#551) (#554)
* docs: align documentation with code behavior (issues #547-#551)

- INSTALL.md: drop SEARXNG_MAX_RETRIES=-1 from the unparsable-value
  example list; cross-reference the hard-error semantic path
- PROMPT_INJECTION_SAFETY.md: mark Option A (warning header + warning
  field) as implemented, referencing #448 and #50
- MCP_TESTING.md: E2E binary is built once into a package-level cache,
  not per test; e2eMCPEnv lives in e2e_helper_test.go; goroutine-leak
  guard is suite-wide, not stress-file-specific
- e2e-tests.md: fix workflow line citations (:87, :104)
- verification.md: drop misleading '(except E2E)' qualifier; narrow the
  E2E-workflow coverage claim for exit-code tests
- test-server.md: add missing test-server-pid-helper recipe row
- .env.example: document SEARXNG_ALLOW_GET_FALLBACK opt-in
- MCP_TOOLS.md: include 'failed to execute search request' segment in
  the network-failure error format
- OUTPUT_FORMAT.md: sync Example 1 count line with the (showing N) format
- adr/014: suppression was relocated to searchWithRetries, not removed
- CONTEXT.md: note empty-text answers are dropped before the engine gate

* docs: address review findings on doc accuracy (P2 items)

- INSTALL.md: SEARXNG_MAX_RETRIES format is an integer in the range
  0-20, not 'non-negative integer' — -1 parses and is rejected by
  SetMaxRetries, so the old wording conflated syntax and semantics
- MCP_TESTING.md: example and E2E_MCP_BINARY section no longer claim a
  per-test build; list e2e_mockserver_test.go among the ungated CLI
  tests that build per-test via buildTestBinary
- CONTEXT.md: dedup description notes the early return when no
  non-empty infobox text exists (original answers returned unchanged)

* docs: fix remaining dedup and E2E_MCP_BINARY wording (P2)

- CONTEXT.md: example dialogue no longer claims non-DuckDuckGo
  answers are never touched; empty answers are dropped before the
  engine check when dedup runs, matching the Search Behavior entry
- test-server.md, verification.md: E2E_MCP_BINARY skips the MCP E2E
  package-level build, not the ungated CLI per-test builds

* docs: scope dedup pass-through claim to non-empty answers

SEARXNG_ANSWER_DEDUP.md and the DeduplicateAnswers doc comment both
claimed non-DuckDuckGo answers always pass through. In fact, when dedup
runs (non-empty infobox content exists), empty-text answers are dropped
before the engine gate; only non-empty non-DuckDuckGo answers pass
through. Aligns both with the implementation in deduplicate.go.

* docs: fix dedup algorithm order in SEARXNG_ANSWER_DEDUP.md

Document the empty-text filter running before the engine gate (matching
deduplicate.go) and add the early-return precondition; sync the Edge
Cases table so retain behavior is scoped to non-empty answers.

* docs: qualify empty-answer drop with early-return precondition

SEARXNG_ANSWER_DEDUP.md edge-case row and the deduplicateAnswers
wrapper comment both stated empty-text answers are always filtered.
When no non-empty infobox exists the implementation early-returns the
original answers unchanged; scope the drop to dedup runs only.
2026-08-11 05:20:47 +08:00

313 lines
12 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# MCP Testing Guide
How to test the MCP server in searxng-mcp-go at the right layer. This guide
covers three complementary approaches — in-memory transport, subprocess stdio
with `mcp.CommandTransport`, and raw `exec.Command` for process-level checks —
and when to reach for each.
For the layer overview, see [Test Layers](#test-layers) below.
With go-sdk v1.7, a normal `Client.Connect` starts with `server/discover`
negotiation and falls back to the legacy `initialize` handshake when needed.
The SDK owns that negotiation; tests should not assume that every client starts
with the same first message.
## Pipe-Based Testing Has Narrow Uses
Using `echo '{"jsonrpc":...}' | ./searxng-mcp-go` as a general MCP test is
unreliable:
- Stateful or multi-step MCP sessions require a **persistent bidirectional
session** over stdin/stdout.
- `echo` is one-shot: it writes one message and closes stdin, so it cannot
coordinate responses, first-message negotiation, or follow-up messages.
- Each stateful test must establish a fresh session, including the SDK's
`server/discover` negotiation and any legacy `initialize` fallback.
- Multi-step flows, notifications, and session behavior cannot be tested with a
one-shot pipe.
A deliberately synchronized raw subprocess can still test the first-message
gate or one stateless request. Use a dedicated harness that keeps the process
and pipes under explicit control for those cases; `echo | binary` remains
unreliable because it cannot synchronize with the server's response.
## Recommended: InMemoryTransport (from official SDK)
The `modelcontextprotocol/go-sdk` provides `NewInMemoryTransports()` specifically for in-process testing. No subprocess, no pipe, no network — just direct function calls.
### Basic Setup
```go
import (
"testing"
"github.com/modelcontextprotocol/go-sdk/mcp"
)
func setupTest(t *testing.T) (*mcp.ClientSession, *mcp.ServerSession, func()) {
ctx := t.Context()
// 1. Create server
server := mcp.NewServer(&mcp.Implementation{
Name: "searxng-mcp-go",
Version: "1.0.0",
}, nil)
// ... add tools to server ...
// 2. Create paired transports (net.Pipe internally)
serverTransport, clientTransport := mcp.NewInMemoryTransports()
// 3. Server connects FIRST (the SDK client negotiates server/discover and
// may fall back to initialize on Connect)
serverSession, err := server.Connect(ctx, serverTransport, nil)
if err != nil {
t.Fatal(err)
}
// 4. Client connects second
client := mcp.NewClient(&mcp.Implementation{
Name: "test-client",
Version: "1.0",
}, nil)
clientSession, err := client.Connect(ctx, clientTransport, nil)
if err != nil {
t.Fatal(err)
}
cleanup := func() {
clientSession.Close()
serverSession.Wait()
}
return clientSession, serverSession, cleanup
}
```
### Testing a Tool Call
```go
func TestSearchTool(t *testing.T) {
clientSession, _, cleanup := setupTest(t)
defer cleanup()
ctx := t.Context()
// List tools
tools, err := clientSession.ListTools(ctx, nil)
if err != nil {
t.Fatal(err)
}
// Verify "search" tool exists with expected schema
// Call the search tool
result, err := clientSession.CallTool(ctx, &mcp.CallToolParams{
Name: "search",
Arguments: map[string]any{
"query": "golang",
},
})
if err != nil {
t.Fatal(err)
}
// Verify result structure
textContent := result.Content[0].(*mcp.TextContent)
// Parse JSON from textContent.Text and assert
}
```
### Testing Error Handling
```go
func TestSearchValidation(t *testing.T) {
clientSession, _, cleanup := setupTest(t)
defer cleanup()
// Empty query should return error
result, err := clientSession.CallTool(t.Context(), &mcp.CallToolParams{
Name: "search",
Arguments: map[string]any{"query": ""},
})
if err != nil {
t.Fatal(err) // MCP transport error, not tool error
}
if !result.IsError {
t.Error("expected IsError=true for empty query")
}
}
```
## Alternative: IOTransport (custom io streams)
If you need to test with actual io streams instead of in-memory:
```go
// Create paired pipes
serverReader, clientWriter := io.Pipe()
clientReader, serverWriter := io.Pipe()
serverTransport := &mcp.IOTransport{Reader: serverReader, Writer: serverWriter}
clientTransport := &mcp.IOTransport{Reader: clientReader, Writer: clientWriter}
```
But `NewInMemoryTransports()` is preferred — it uses `net.Pipe()` internally and is designed for this exact purpose.
## End-to-End: Subprocess stdio with CommandTransport
In-memory transports prove the protocol surface and tool wiring, but they do
not exercise the real stdio boundary, the binary's startup environment, or
process lifecycle. Most E2E tests in `e2e_*_test.go` use build tag `e2e`; the
stress tests in `e2e_stress_test.go` require `e2e && stress`. They use the
SDK's `mcp.CommandTransport`, which spawns the built binary as a subprocess
and talks JSON-RPC over its stdin/stdout. This is the only layer that catches
issues like wrong env-var parsing on startup, missing stderr flushing on
shutdown, or stdio framing regressions.
Three files are not gated by the `e2e` build tag:
`e2e_exitcode_test.go`, `e2e_cli_test.go`, and `e2e_mockserver_test.go`.
They exercise the built binary as a subprocess with no external dependencies
(no live SearXNG server, no MCP session) and run in under a second each,
so they ship in the default `go test ./...` set alongside the unit tests.
### Pattern
```go
//go:build e2e
import (
"bytes"
"context"
"os"
"os/exec"
"testing"
"time"
"github.com/modelcontextprotocol/go-sdk/mcp"
)
func TestE2E(t *testing.T) {
ctx, cancel := context.WithTimeout(t.Context(), 180*time.Second)
defer cancel()
// 1. Reuse a pre-built binary if E2E_MCP_BINARY is set;
// otherwise fall back to the package-level cached build.
binaryPath := os.Getenv("E2E_MCP_BINARY")
if binaryPath == "" {
binaryPath = buildE2EMCPBinary(ctx, t) // go build -o <cached tempdir>/searxng-mcp-go .
}
var stderr bytes.Buffer
cmd := exec.CommandContext(ctx, binaryPath)
cmd.Env = e2eMCPEnv(os.Getenv("SEARXNG_URL")) // adds SEARXNG_MAX_RETRIES=2
cmd.Stderr = &stderr
client := mcp.NewClient(&mcp.Implementation{Name: "e2e", Version: "0"}, nil)
session, err := client.Connect(ctx, &mcp.CommandTransport{Command: cmd}, nil)
if err != nil {
t.Fatalf("connect failed: %v\nstderr:\n%s", err, stderr.String())
}
t.Cleanup(func() {
_ = session.Close()
if cmd.Process != nil && cmd.ProcessState == nil {
_ = cmd.Process.Kill()
_, _ = cmd.Process.Wait()
}
})
// ... session.ListTools(), session.CallTool(), assert results ...
}
```
### `E2E_MCP_BINARY`
`E2E_MCP_BINARY` is an optional environment variable read by every E2E test
that uses `mcp.CommandTransport` (`e2e_mcp_test.go`, `e2e_functional_test.go`,
`e2e_error_test.go`, `e2e_lifecycle_test.go`, `e2e_stress_test.go`). When set, the test reuses that
binary instead of triggering the package-level cached build. CI sets it to
`./searxng-mcp-go` after a single explicit build step (see
`.github/workflows/e2e.yml`), which makes the in-workflow retry loop cheap.
For local development:
```bash
go build -o ./searxng-mcp-go .
E2E_MCP_BINARY=$PWD/searxng-mcp-go \
SEARXNG_URL=http://127.0.0.1:8888 \
go test -tags=e2e -run TestMCPStdioE2E -count=1 .
```
When `E2E_MCP_BINARY` is unset, the binary is built once into a
package-level cached temp directory via the `e2eMCPBinaryPath` helper and
reused across all E2E tests (the directory is intentionally leaked to avoid
rebuilding the binary for every test). Only the ungated CLI tests
(`e2e_exitcode_test.go`, `e2e_cli_test.go`, `e2e_mockserver_test.go`) build
per-test in `t.TempDir()` via `buildTestBinary`.
`e2e_exitcode_test.go` does not use `mcp.CommandTransport` and is not gated
by the `e2e` build tag; it asserts CLI exit codes via raw `exec.Command` and
builds its own binary in `t.TempDir()` regardless of `E2E_MCP_BINARY`. It
runs as part of the default `go test ./...` set, so CI's regular test job
picks it up via the `-tags=stress`-less default run.
### `SEARXNG_MAX_RETRIES=2`
The shared `e2eMCPEnv` helper (in `e2e_helper_test.go`) injects
`SEARXNG_MAX_RETRIES=2` on top of the inherited environment. The production
default is 5; the E2E tests deliberately lower it to keep wall-clock time
predictable while still exercising the retry path. Do not raise this for new
E2E tests without an explicit reason, and do not lower it to 0 — that would
hide regressions in the retry layer.
## MCP Inspector (manual testing)
```bash
# Interactive web UI
SEARXNG_URL=http://127.0.0.1:8888 npx @modelcontextprotocol/inspector ./searxng-mcp-go
# Or with a built binary
go build -o searxng-mcp-go .
SEARXNG_URL=http://127.0.0.1:8888 npx @modelcontextprotocol/inspector ./searxng-mcp-go
```
Opens a web UI where you can:
- View all registered tools and their schemas
- Call tools interactively
- See all JSON-RPC messages in real time
## Test Layers
The repository tests MCP at five layers, each catching a different class of
defect. The three complementary MCP testing approaches from the intro above
cover rows 2–4 of the table; unit tests and manual / smoke checks bookend
them. Pick the lowest layer that can meaningfully assert what you want to
verify.
| Layer | Method | Files | What it catches |
|-------|--------|-------|-----------------|
| Unit | direct handler call (`NewSearchToolHandler()`) | `mcp_tool_test.go`, `internal/searxng/*_test.go` | Search logic, input validation, JSON response shape |
| MCP integration | `mcp.NewInMemoryTransports()` (`net.Pipe` in-process) | (unit-style tests of the MCP server surface) | MCP protocol wiring, tool registration, schema, session lifecycle |
| CLI subprocess (default `go test ./...`) | raw `exec.Command` on the built binary | `e2e_exitcode_test.go`, `e2e_cli_test.go`, `e2e_mockserver_test.go` | CLI exit code contract, stderr/stdout split, flag parsing |
| MCP E2E (stdio, `-tags=e2e`) | `exec.Command` + `mcp.CommandTransport` (subprocess) | `e2e_mcp_test.go`, `e2e_functional_test.go`, `e2e_error_test.go`, `e2e_lifecycle_test.go` | Real stdio framing, env-var startup, process lifecycle, live SearXNG behavior |
| MCP E2E stress (stdio, `-tags='e2e stress'`) | `exec.Command` + `mcp.CommandTransport` (subprocess) | `e2e_stress_test.go` | Concurrent searches, lifecycle concurrency; suite-wide goroutine-leak guard active during stress runs |
| Manual / smoke | MCP Inspector, CI shell smoke | `.github/workflows/e2e.yml` | Interactive verification, deterministic exit-code smoke |
Rule of thumb:
- Use **InMemoryTransport** for fast, deterministic tests of the MCP protocol
surface (no subprocess, no live network).
- Use **CommandTransport** when the assertion depends on the binary actually
starting, reading env vars, or talking over real stdin/stdout.
- Use **raw `exec.Command`** for process-level assertions (exit code, stderr
text) and carefully synchronized first-message or single-stateless-request
E2E checks; use `CommandTransport` for multi-step MCP protocol behavior.
## Key Gotchas
1. **Server must Connect before Client for SDK negotiation** — go-sdk v1.7 clients send `server/discover` first and may fall back to `initialize`, so the server must be ready to receive either message
2. **Always call `serverSession.Wait()` in cleanup** — ensures the server goroutine exits cleanly
3. **`clientSession.Close()` before `serverSession.Wait()`** — client initiates shutdown
## References
- MCP Go SDK: https://github.com/modelcontextprotocol/go-sdk
- MCP Inspector: https://github.com/modelcontextprotocol/inspector
- SDK test examples: go-sdk/mcp/mcp_test.go (65KB, 25 test files)