searxng-mcp-go/docs/MCP_TESTING.md
XLion 1afa9612c3
docs: align documentation with code behavior (issues #547-#551) (#554)
* docs: align documentation with code behavior (issues #547-#551)

- INSTALL.md: drop SEARXNG_MAX_RETRIES=-1 from the unparsable-value
  example list; cross-reference the hard-error semantic path
- PROMPT_INJECTION_SAFETY.md: mark Option A (warning header + warning
  field) as implemented, referencing #448 and #50
- MCP_TESTING.md: E2E binary is built once into a package-level cache,
  not per test; e2eMCPEnv lives in e2e_helper_test.go; goroutine-leak
  guard is suite-wide, not stress-file-specific
- e2e-tests.md: fix workflow line citations (:87, :104)
- verification.md: drop misleading '(except E2E)' qualifier; narrow the
  E2E-workflow coverage claim for exit-code tests
- test-server.md: add missing test-server-pid-helper recipe row
- .env.example: document SEARXNG_ALLOW_GET_FALLBACK opt-in
- MCP_TOOLS.md: include 'failed to execute search request' segment in
  the network-failure error format
- OUTPUT_FORMAT.md: sync Example 1 count line with the (showing N) format
- adr/014: suppression was relocated to searchWithRetries, not removed
- CONTEXT.md: note empty-text answers are dropped before the engine gate

* docs: address review findings on doc accuracy (P2 items)

- INSTALL.md: SEARXNG_MAX_RETRIES format is an integer in the range
  0-20, not 'non-negative integer' — -1 parses and is rejected by
  SetMaxRetries, so the old wording conflated syntax and semantics
- MCP_TESTING.md: example and E2E_MCP_BINARY section no longer claim a
  per-test build; list e2e_mockserver_test.go among the ungated CLI
  tests that build per-test via buildTestBinary
- CONTEXT.md: dedup description notes the early return when no
  non-empty infobox text exists (original answers returned unchanged)

* docs: fix remaining dedup and E2E_MCP_BINARY wording (P2)

- CONTEXT.md: example dialogue no longer claims non-DuckDuckGo
  answers are never touched; empty answers are dropped before the
  engine check when dedup runs, matching the Search Behavior entry
- test-server.md, verification.md: E2E_MCP_BINARY skips the MCP E2E
  package-level build, not the ungated CLI per-test builds

* docs: scope dedup pass-through claim to non-empty answers

SEARXNG_ANSWER_DEDUP.md and the DeduplicateAnswers doc comment both
claimed non-DuckDuckGo answers always pass through. In fact, when dedup
runs (non-empty infobox content exists), empty-text answers are dropped
before the engine gate; only non-empty non-DuckDuckGo answers pass
through. Aligns both with the implementation in deduplicate.go.

* docs: fix dedup algorithm order in SEARXNG_ANSWER_DEDUP.md

Document the empty-text filter running before the engine gate (matching
deduplicate.go) and add the early-return precondition; sync the Edge
Cases table so retain behavior is scoped to non-empty answers.

* docs: qualify empty-answer drop with early-return precondition

SEARXNG_ANSWER_DEDUP.md edge-case row and the deduplicateAnswers
wrapper comment both stated empty-text answers are always filtered.
When no non-empty infobox exists the implementation early-returns the
original answers unchanged; scope the drop to dedup runs only.
2026-08-11 05:20:47 +08:00

12 KiB
Raw Permalink Blame History

MCP Testing Guide

How to test the MCP server in searxng-mcp-go at the right layer. This guide covers three complementary approaches — in-memory transport, subprocess stdio with mcp.CommandTransport, and raw exec.Command for process-level checks — and when to reach for each.

For the layer overview, see Test Layers below.

With go-sdk v1.7, a normal Client.Connect starts with server/discover negotiation and falls back to the legacy initialize handshake when needed. The SDK owns that negotiation; tests should not assume that every client starts with the same first message.

Pipe-Based Testing Has Narrow Uses

Using echo '{"jsonrpc":...}' | ./searxng-mcp-go as a general MCP test is unreliable:

  • Stateful or multi-step MCP sessions require a persistent bidirectional session over stdin/stdout.
  • echo is one-shot: it writes one message and closes stdin, so it cannot coordinate responses, first-message negotiation, or follow-up messages.
  • Each stateful test must establish a fresh session, including the SDK's server/discover negotiation and any legacy initialize fallback.
  • Multi-step flows, notifications, and session behavior cannot be tested with a one-shot pipe.

A deliberately synchronized raw subprocess can still test the first-message gate or one stateless request. Use a dedicated harness that keeps the process and pipes under explicit control for those cases; echo | binary remains unreliable because it cannot synchronize with the server's response.

The modelcontextprotocol/go-sdk provides NewInMemoryTransports() specifically for in-process testing. No subprocess, no pipe, no network — just direct function calls.

Basic Setup

import (
    "testing"
    "github.com/modelcontextprotocol/go-sdk/mcp"
)

func setupTest(t *testing.T) (*mcp.ClientSession, *mcp.ServerSession, func()) {
    ctx := t.Context()

    // 1. Create server
    server := mcp.NewServer(&mcp.Implementation{
        Name:    "searxng-mcp-go",
        Version: "1.0.0",
    }, nil)
    // ... add tools to server ...

    // 2. Create paired transports (net.Pipe internally)
    serverTransport, clientTransport := mcp.NewInMemoryTransports()

    // 3. Server connects FIRST (the SDK client negotiates server/discover and
    //    may fall back to initialize on Connect)
    serverSession, err := server.Connect(ctx, serverTransport, nil)
    if err != nil {
        t.Fatal(err)
    }

    // 4. Client connects second
    client := mcp.NewClient(&mcp.Implementation{
        Name:    "test-client",
        Version: "1.0",
    }, nil)
    clientSession, err := client.Connect(ctx, clientTransport, nil)
    if err != nil {
        t.Fatal(err)
    }

    cleanup := func() {
        clientSession.Close()
        serverSession.Wait()
    }

    return clientSession, serverSession, cleanup
}

Testing a Tool Call

func TestSearchTool(t *testing.T) {
    clientSession, _, cleanup := setupTest(t)
    defer cleanup()
    ctx := t.Context()

    // List tools
    tools, err := clientSession.ListTools(ctx, nil)
    if err != nil {
        t.Fatal(err)
    }
    // Verify "search" tool exists with expected schema

    // Call the search tool
    result, err := clientSession.CallTool(ctx, &mcp.CallToolParams{
        Name: "search",
        Arguments: map[string]any{
            "query": "golang",
        },
    })
    if err != nil {
        t.Fatal(err)
    }

    // Verify result structure
    textContent := result.Content[0].(*mcp.TextContent)
    // Parse JSON from textContent.Text and assert
}

Testing Error Handling

func TestSearchValidation(t *testing.T) {
    clientSession, _, cleanup := setupTest(t)
    defer cleanup()

    // Empty query should return error
    result, err := clientSession.CallTool(t.Context(), &mcp.CallToolParams{
        Name:      "search",
        Arguments: map[string]any{"query": ""},
    })

    if err != nil {
        t.Fatal(err) // MCP transport error, not tool error
    }

    if !result.IsError {
        t.Error("expected IsError=true for empty query")
    }
}

Alternative: IOTransport (custom io streams)

If you need to test with actual io streams instead of in-memory:

// Create paired pipes
serverReader, clientWriter := io.Pipe()
clientReader, serverWriter := io.Pipe()

serverTransport := &mcp.IOTransport{Reader: serverReader, Writer: serverWriter}
clientTransport := &mcp.IOTransport{Reader: clientReader, Writer: clientWriter}

But NewInMemoryTransports() is preferred — it uses net.Pipe() internally and is designed for this exact purpose.

End-to-End: Subprocess stdio with CommandTransport

In-memory transports prove the protocol surface and tool wiring, but they do not exercise the real stdio boundary, the binary's startup environment, or process lifecycle. Most E2E tests in e2e_*_test.go use build tag e2e; the stress tests in e2e_stress_test.go require e2e && stress. They use the SDK's mcp.CommandTransport, which spawns the built binary as a subprocess and talks JSON-RPC over its stdin/stdout. This is the only layer that catches issues like wrong env-var parsing on startup, missing stderr flushing on shutdown, or stdio framing regressions.

Three files are not gated by the e2e build tag: e2e_exitcode_test.go, e2e_cli_test.go, and e2e_mockserver_test.go. They exercise the built binary as a subprocess with no external dependencies (no live SearXNG server, no MCP session) and run in under a second each, so they ship in the default go test ./... set alongside the unit tests.

Pattern

//go:build e2e

import (
    "bytes"
    "context"
    "os"
    "os/exec"
    "testing"
    "time"

    "github.com/modelcontextprotocol/go-sdk/mcp"
)

func TestE2E(t *testing.T) {
    ctx, cancel := context.WithTimeout(t.Context(), 180*time.Second)
    defer cancel()

    // 1. Reuse a pre-built binary if E2E_MCP_BINARY is set;
    //    otherwise fall back to the package-level cached build.
    binaryPath := os.Getenv("E2E_MCP_BINARY")
    if binaryPath == "" {
        binaryPath = buildE2EMCPBinary(ctx, t) // go build -o <cached tempdir>/searxng-mcp-go .
    }

    var stderr bytes.Buffer
    cmd := exec.CommandContext(ctx, binaryPath)
    cmd.Env = e2eMCPEnv(os.Getenv("SEARXNG_URL")) // adds SEARXNG_MAX_RETRIES=2
    cmd.Stderr = &stderr

    client := mcp.NewClient(&mcp.Implementation{Name: "e2e", Version: "0"}, nil)
    session, err := client.Connect(ctx, &mcp.CommandTransport{Command: cmd}, nil)
    if err != nil {
        t.Fatalf("connect failed: %v\nstderr:\n%s", err, stderr.String())
    }
    t.Cleanup(func() {
        _ = session.Close()
        if cmd.Process != nil && cmd.ProcessState == nil {
            _ = cmd.Process.Kill()
            _, _ = cmd.Process.Wait()
        }
    })

    // ... session.ListTools(), session.CallTool(), assert results ...
}

E2E_MCP_BINARY

E2E_MCP_BINARY is an optional environment variable read by every E2E test that uses mcp.CommandTransport (e2e_mcp_test.go, e2e_functional_test.go, e2e_error_test.go, e2e_lifecycle_test.go, e2e_stress_test.go). When set, the test reuses that binary instead of triggering the package-level cached build. CI sets it to ./searxng-mcp-go after a single explicit build step (see .github/workflows/e2e.yml), which makes the in-workflow retry loop cheap. For local development:

go build -o ./searxng-mcp-go .
E2E_MCP_BINARY=$PWD/searxng-mcp-go \
  SEARXNG_URL=http://127.0.0.1:8888 \
  go test -tags=e2e -run TestMCPStdioE2E -count=1 .

When E2E_MCP_BINARY is unset, the binary is built once into a package-level cached temp directory via the e2eMCPBinaryPath helper and reused across all E2E tests (the directory is intentionally leaked to avoid rebuilding the binary for every test). Only the ungated CLI tests (e2e_exitcode_test.go, e2e_cli_test.go, e2e_mockserver_test.go) build per-test in t.TempDir() via buildTestBinary.

e2e_exitcode_test.go does not use mcp.CommandTransport and is not gated by the e2e build tag; it asserts CLI exit codes via raw exec.Command and builds its own binary in t.TempDir() regardless of E2E_MCP_BINARY. It runs as part of the default go test ./... set, so CI's regular test job picks it up via the -tags=stress-less default run.

SEARXNG_MAX_RETRIES=2

The shared e2eMCPEnv helper (in e2e_helper_test.go) injects SEARXNG_MAX_RETRIES=2 on top of the inherited environment. The production default is 5; the E2E tests deliberately lower it to keep wall-clock time predictable while still exercising the retry path. Do not raise this for new E2E tests without an explicit reason, and do not lower it to 0 — that would hide regressions in the retry layer.

MCP Inspector (manual testing)

# Interactive web UI
SEARXNG_URL=http://127.0.0.1:8888 npx @modelcontextprotocol/inspector ./searxng-mcp-go

# Or with a built binary
go build -o searxng-mcp-go .
SEARXNG_URL=http://127.0.0.1:8888 npx @modelcontextprotocol/inspector ./searxng-mcp-go

Opens a web UI where you can:

  • View all registered tools and their schemas
  • Call tools interactively
  • See all JSON-RPC messages in real time

Test Layers

The repository tests MCP at five layers, each catching a different class of defect. The three complementary MCP testing approaches from the intro above cover rows 2–4 of the table; unit tests and manual / smoke checks bookend them. Pick the lowest layer that can meaningfully assert what you want to verify.

Layer Method Files What it catches
Unit direct handler call (NewSearchToolHandler()) mcp_tool_test.go, internal/searxng/*_test.go Search logic, input validation, JSON response shape
MCP integration mcp.NewInMemoryTransports() (net.Pipe in-process) (unit-style tests of the MCP server surface) MCP protocol wiring, tool registration, schema, session lifecycle
CLI subprocess (default go test ./...) raw exec.Command on the built binary e2e_exitcode_test.go, e2e_cli_test.go, e2e_mockserver_test.go CLI exit code contract, stderr/stdout split, flag parsing
MCP E2E (stdio, -tags=e2e) exec.Command + mcp.CommandTransport (subprocess) e2e_mcp_test.go, e2e_functional_test.go, e2e_error_test.go, e2e_lifecycle_test.go Real stdio framing, env-var startup, process lifecycle, live SearXNG behavior
MCP E2E stress (stdio, -tags='e2e stress') exec.Command + mcp.CommandTransport (subprocess) e2e_stress_test.go Concurrent searches, lifecycle concurrency; suite-wide goroutine-leak guard active during stress runs
Manual / smoke MCP Inspector, CI shell smoke .github/workflows/e2e.yml Interactive verification, deterministic exit-code smoke

Rule of thumb:

  • Use InMemoryTransport for fast, deterministic tests of the MCP protocol surface (no subprocess, no live network).
  • Use CommandTransport when the assertion depends on the binary actually starting, reading env vars, or talking over real stdin/stdout.
  • Use raw exec.Command for process-level assertions (exit code, stderr text) and carefully synchronized first-message or single-stateless-request E2E checks; use CommandTransport for multi-step MCP protocol behavior.

Key Gotchas

  1. Server must Connect before Client for SDK negotiation — go-sdk v1.7 clients send server/discover first and may fall back to initialize, so the server must be ready to receive either message
  2. Always call serverSession.Wait() in cleanup — ensures the server goroutine exits cleanly
  3. clientSession.Close() before serverSession.Wait() — client initiates shutdown

References