searxng-mcp-go/docs/SEARXNG_ANSWER_DEDUP.md
XLion 9617a6318f
test: consolidate answer contract ownership (#563)
* test: consolidate answer contract ownership

* test: address answer ownership review
2026-08-12 00:33:45 +08:00

136 lines
7.2 KiB
Markdown

# SearXNG Answer Deduplication
## Problem
SearXNG's DuckDuckGo engine returns Wikipedia summaries in **both** `answers` and `infoboxes` fields for entity queries (company names, people, places, etc.). This causes duplicate content in our output — the same text appears under "=== Answers ===" and again under "=== Infoboxes ===".
### Example: Query "apple inc"
SearXNG raw JSON (simplified):
```json
{
"answers": [
{
"answer": "Apple Inc. is an American multinational technology company headquartered in Cupertino, California...",
"engine": "duckduckgo",
"template": "wikipedia.html"
}
],
"infoboxes": [
{
"infobox": "Apple Inc.",
"content": "Apple Inc. is an American multinational technology company headquartered in Cupertino, California...",
"urls": [...]
}
]
}
```
The `answers[0].answer` text is a verbatim copy of `infoboxes[0].content` (or a prefix of it).
### Queries Affected
This primarily affects **entity queries** — searches that trigger DuckDuckGo's Wikipedia instant answer:
- Company names: "apple inc", "google", "microsoft"
- People: "elon musk", "albert einstein"
- Places: "tokyo", "mount everest"
- General knowledge: "python programming language"
### Queries NOT Affected
Queries where `answers` contains genuinely distinct information:
- IP lookup: "ip" → answer is your public IP, infobox may explain what an IP is
- Calculator: "2+2" → answer is "4"
- Timezone: "time in tokyo"
- Hash lookup: hash values in answers
## Solution
### Engine Scoping
Deduplication is applied **only** to answers from the `duckduckgo` engine. The " More at Wikipedia" suffix and 200-rune substring heuristic are specific to DuckDuckGo's response format, where the same Wikipedia summary appears in both `answers` and `infoboxes`. When dedup runs, answers with empty text are dropped before the engine gate; non-empty answers from other engines (calculator, IP, timezone, etc.), including those with unknown `Engine`, pass through without deduplication. This prevents false positives like discarding a calculator result "4" whose text happens to appear in unrelated infobox content.
### Deduplication Strategy
The function `DeduplicateAnswers` in `internal/searxng/answer/deduplicate.go` filters answers at the **search response layer**, before the result reaches JSON serialization or CLI formatting. This means both output modes benefit.
**Algorithm (empty-text filter + engine gate + exact-case fast path + lazy lowercase fallback):**
The dedup loop runs only when there is at least one non-empty infobox text; otherwise the original answers are returned unchanged. For each answer, the following steps are applied:
*Empty-text filter (all engines):*
1. If `ans.Answer == ""`, the answer is dropped regardless of engine.
*Engine gate:*
1. If `ans.Engine != "duckduckgo"`, the answer is retained without further checks.
*Fast path (no allocation, DuckDuckGo answers only):*
1. Strip the known suffix `" More at Wikipedia"` (exact case) that DuckDuckGo appends.
2. Take the first 200 characters of the stripped text as a prefix.
3. Check if any infobox content **contains** this prefix using exact-case substring matching.
*Lazy fallback (lowercase, built on first use, DuckDuckGo answers only):*
If the fast path did not match, a lowercased copy of all infobox contents is built once and reused for subsequent answers:
1. Lowercase the answer text.
2. Strip the known suffix `" more at wikipedia"` (lowercase).
3. Take the first 200 characters of the stripped text as a prefix.
4. Check if any lowercased infobox content **contains** this prefix (case-insensitive substring match).
If either path finds a match, the answer is considered a duplicate and is removed. The lazy fallback handles edge cases where the answer and infobox differ only in casing (e.g. `"Apple"` vs `"apple"`).
### Why Search Layer, Not Format Layer
Deduplication happens while normalizing the SearXNG response. The `deduplicateAnswers` wrapper in `internal/searxng/deduplicate.go` extracts infobox content and delegates to `DeduplicateAnswers` in `internal/searxng/answer/deduplicate.go`. This is done at the normalization layer rather than in `formatResults()` (format.go) so that:
- **JSON output** (`--json` flag or MCP mode) also has clean results
- **CLI output** benefits automatically
- Single point of truth — no scattered filtering logic
### Edge Cases Handled
| Scenario | Behavior |
|----------|----------|
| Empty `answers` | Return as-is (no filtering) |
| Empty `infoboxes` | Return as-is (no filtering) |
| Infobox with empty `content` | Skip that infobox (no filtering against it) |
| Empty `answer` text | Dropped when dedup runs (non-empty infobox exists); kept otherwise (early return) |
| Case differences | Case-insensitive comparison |
| Non-DuckDuckGo engine | Retain when non-empty (no dedup applied) |
| Empty `Engine` field | Retain when non-empty (conservative default) |
| Unknown `Engine` field | Retain when non-empty (conservative default) |
| Multiple answers, some duplicate | Keep non-duplicates, remove DuckDuckGo duplicates |
## Bounded Work Cap (CAND-33fe0b85-RUNTIME-003)
The dedup loop is O(len(answers) * len(infoboxes)) substring work, and for
each answer it scans every infobox twice (exact case, then lowercase) with
`strings.Contains`. A successful JSON body is already capped at
`MaxResponseBodySize` (2 MiB), but compact answer/infobox arrays can still
pack tens of thousands of entries into that envelope and force very large CPU
work before result limiting.
To bound that work, response normalization trims `Answers` and `Infoboxes`
to `MaxAnswers` and `MaxInfoboxes` (each 100) **before** the dedup call:
- Real SearXNG responses contain at most a handful of each, so 100 is a
generous cap.
- Worst-case dedup work is bounded at
`100 * 100 * 2 = 20,000` `Contains` calls.
- Truncation is logged with `slog.Warn` (`truncating answers/infoboxes
before deduplication`) and is visible in debug logs.
The cap is enforced in `normalizeResponse` (see
`internal/searxng/response.go`), not in `deduplicateAnswers` itself, so the
dedup function remains a pure data transform and tests can exercise it
without a searcher.
## Implementation
- **Function**: `DeduplicateAnswers(answers []Answer, infoboxContents []string) []Answer` in `internal/searxng/answer/deduplicate.go`
- **Called**: Via `deduplicateAnswers` wrapper in `internal/searxng/deduplicate.go` during response normalization after JSON unmarshalling, after the `MaxAnswers` / `MaxInfoboxes` cap is applied, before the response is returned to formatting/output code
- **Tests**:
- `internal/searxng/answer/deduplicate_test.go` — unit tests for `DeduplicateAnswers`, covering empty inputs, exact/prefix match, "More at Wikipedia" suffix stripping, case insensitivity, engine scoping, distinct answers, mixed scenarios, empty answer skipping, and truncation boundary cases
- `internal/searxng/deduplicate_internal_test.go` — focused adapter coverage that verifies only `Infobox.Content` is projected into the canonical deduplication function
- `TestNormalizeResponse` in `internal/searxng/response_internal_test.go` — integration coverage for typed-answer fallback before deduplication, answer/infobox caps before deduplication, no-op at cap, and the pathological-input time budget