searxng-mcp-go/docs/AI_UX_TEST_GUIDE.md
XLion 7c60300b48
docs: fix documentation inaccuracies across 7 files
- #218: Correct --language CLI help from [default: auto] to [default: ""]
- #219: Unify pageno default semantics in MCP_TOOLS.md table
- #220: Add 'Fetch metadata header' to Sec-Fetch-Dest/Site purpose rows
- #221: Replace pipe-violating echo | commands with → pseudo-flow notation
- #223: Clarify unresponsive_engines logged at debug level (slog.Debug)
- #224: Warn about http:// SEARXNG_URL triggering non-suppressible warning
- #225: Remove misleading PR #78 reference in centralization_drift_test.go

Closes #218, #219, #220, #221, #223, #224, #225
2026-06-11 02:23:03 +08:00

240 lines
8.2 KiB
Markdown

# SearXNG MCP AI UX Testing Guide
> This document establishes an "UX testing" workflow conducted by AI — unlike unit tests that verify functional correctness, UX testing evaluates whether the tools feel good to use from the **user (AI agent) perspective**, and quantifies feedback to drive improvements.
---
## Testing Objectives
Evaluate the user experience of the SearXNG MCP server from an AI agent perspective:
1. **Schema clarity** — Can the AI infer correct usage from the schema alone?
2. **Response format quality** — Are results easy to understand and further process?
3. **Stability and error handling** — Do various inputs cause unexpected behavior?
4. **Overall convenience** — Is the workflow smooth, or are there pain points?
---
## Prerequisites
### Required Environment
- **Project location**: repository root
- **Binary**: `./searxng-mcp-go`
- **SearXNG instance**: `http://localhost:8888/`
- **Available toolsets**: `terminal`, `file`, `web`, `skills`, `session_search`
### MCP Server Startup
The MCP server runs in stdio mode, meaning all communication happens via JSON-RPC over stdin/stdout.
**Two testing approaches:**
1. **Integration testing** (via Hermes Agent MCP tool):
```bash
# Directly call the mcp_searxng_search tool
# Suitable for end-to-end workflow testing
```
2. **Isolated testing** (direct JSON-RPC):
> **Note:** The following JSON-RPC message sequences use arrow notation (→/←) to show the bidirectional conversation. They are not directly executable shell commands — the MCP protocol requires a persistent bidirectional session (e.g., via MCP Inspector or a test harness that keeps stdin open).
```bash
# Initialize
→ {"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"test","version":"1.0"}}}
← {"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2024-11-05","capabilities":{...}}}
# Call tools/list
→ {"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}
← {"jsonrpc":"2.0","id":2,"result":{"tools":[...]}}
# Call tools/call (search)
→ {"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"search","arguments":{"query":"test","language":"en","safesearch":0,"time_range":"","categories":"","engines":"","pageno":1}}}
← {"jsonrpc":"2.0","id":3,"result":{"content":[{"type":"text","text":"..."}]}}
```
---
## Test Checklist
### 1. Schema Declaration Evaluation
**Method:** Call `tools/list` to inspect `inputSchema`. Do not look at the source code — infer usage purely from the schema text.
**Evaluation Dimensions:**
| Dimension | Question | Score (1-5) |
|-----------|----------|-------------|
| Parameter name intuitiveness | Are the names self-explanatory? | |
| Type declaration clarity | Does seeing `type: ["null","integer"]` make it clear what to pass? | |
| Default value documentation | Does the schema explain default values? | |
| Enum completeness | Are all valid values listed for fields like `safesearch` and `time_range`? | |
| Required/optional distinction | Can the AI determine which parameters are optional from the schema? | |
**Must-try scenarios:**
```bash
# Pass only query (test required field correctness)
# (Pseudo-flow — see note above on bidirectional session requirement)
→ {"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"search","arguments":{"query":"Golang MCP server"}}}
# Expected: success (only query is required)
# Pitfall: if required is incorrectly declared, a validation error will be returned
```
### 2. Response Format Evaluation
**Method:** Run several different queries and observe the returned text format.
**Evaluation Dimensions:**
| Dimension | Question | Score (1-5) |
|-----------|----------|-------------|
| Result count indication | Is it clear how many results were found? | |
| Field completeness | Does each record include title/URL/summary/source? | |
| Summary quality | Does the summary contain actual content rather than being empty? | |
| Format consistency | Is the format uniform across multiple results? | |
| Structured output | Is it plain text or JSON? Can the AI parse it programmatically? | |
**Must-try queries:**
```bash
# Basic English
query="MCP protocol specification", language="en"
# Chinese <!-- intentionally non-English: testing Chinese language search query -->
query="測試", language="zh-tw"
# Empty result expected
query="xyzzyqqq1234567890xyz", language="en"
# Long query
query="What are the best practices for building MCP servers in Go in 2024 and 2025", language="en"
```
### 3. Parameter Combination Testing
**Must-try combinations:**
| Parameter Combination | Expected Behavior | Observation |
|-----------------------|-------------------|-------------|
| `query="news", categories="news"` | News only | Are results filtered? |
| `query="AI", engines="google,bing"` | Only these two engines | Are results diverse? |
| `query="test", pageno=2` | Second page | Is the content different from the first page? |
| `query="test", time_range="day"` | Last 24 hours | Is the time filter applied? |
| `query="test", safesearch=2` | Strict filtering | Are there fewer results? |
### 4. Error Handling and Stability
**Must-try scenarios:**
```bash
# Empty query
{"query": ""} → Should return "validation error on \"query\": search query cannot be only whitespace" error
# Invalid time_range
{"query": "test", "time_range": "invalid"} → Should explain valid values
# Invalid safesearch
{"query": "test", "safesearch": 5} → Should explain valid range 0-2
# Invalid engine
{"query": "test", "engines": "nonexistent_engine_xyz"} → How does SearXNG handle this?
# Network errors (if simulable)
# Disconnect, target unreachable, etc.
```
**Observation points:**
- Is the error message human-readable (not a JSON parse error)?
- Does it panic or output non-JSON garbage?
- Does the server continue working normally after an error (or does it die after one use)?
### 5. Isolation Testing
**Objective:** Confirm that requests are cleanly isolated from each other.
```bash
# Request 1: Set a parameter
{"query": "apple", "language": "en"}
# Request 2: No parameters at all (test for leftover state)
{"query": "banana"}
# Observation: Is the second request affected by the first?
```
---
## Output Format
After completing the tests, produce a report in the following format:
```markdown
## 🔴 Critical Issues (Blocking)
## ⚠️ Moderate Issues (Affecting UX)
## 💡 Improvement Suggestions (Optional)
## Summary
| Item | Score (1-5) | Notes |
|------|-------------|-------|
| Schema clarity | | |
| Response format satisfaction | | |
| Stability | | |
| Error handling quality | | |
| Parameter validation completeness | | |
| Overall convenience | | |
```
---
## Delight and Frustration Log
In addition to quantitative scores, please also note the following:
**Impressive moments:**
- Which design decisions made the AI experience especially smooth?
- Were any error messages surprisingly precise?
**Frustrating moments:**
- Why did something simple require a roundabout approach?
- Which error wasted the most debugging time?
**"If I were the developer" thoughts:**
- What missing information made it hard to optimize usage?
- Which schema descriptions created false expectations?
---
## Appendix: Common JSON-RPC Interaction Examples
### Full Conversation Flow
```bash
# 1. Initialize
→ {"jsonrpc":"2.0","id":0,"method":"initialize","params":{...}}
← {"jsonrpc":"2.0","id":0,"result":{"protocolVersion":"...","capabilities":{...}}}
# 2. List tools
→ {"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}
← {"jsonrpc":"2.0","id":1,"result":{"tools":[...]}}
# 3. Execute search
→ {"jsonrpc":"2.0","id":2,"method":"tools/call","params":{...}}
← {"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"..."}]}}
```
### Quick Verification Commands
> **Note:** The →/← examples below are pseudo-flow for illustration. Use MCP Inspector or a bidirectional test harness for actual testing.
```bash
# Verify required fields bug (if passing only query fails = bug)
→ {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"search","arguments":{"query":"test"}}}
# List all tools and format output
→ {"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}
```