* test: consolidate answer contract ownership * test: address answer ownership review
7.2 KiB
SearXNG Answer Deduplication
Problem
SearXNG's DuckDuckGo engine returns Wikipedia summaries in both answers and infoboxes fields for entity queries (company names, people, places, etc.). This causes duplicate content in our output — the same text appears under "=== Answers ===" and again under "=== Infoboxes ===".
Example: Query "apple inc"
SearXNG raw JSON (simplified):
{
"answers": [
{
"answer": "Apple Inc. is an American multinational technology company headquartered in Cupertino, California...",
"engine": "duckduckgo",
"template": "wikipedia.html"
}
],
"infoboxes": [
{
"infobox": "Apple Inc.",
"content": "Apple Inc. is an American multinational technology company headquartered in Cupertino, California...",
"urls": [...]
}
]
}
The answers[0].answer text is a verbatim copy of infoboxes[0].content (or a prefix of it).
Queries Affected
This primarily affects entity queries — searches that trigger DuckDuckGo's Wikipedia instant answer:
- Company names: "apple inc", "google", "microsoft"
- People: "elon musk", "albert einstein"
- Places: "tokyo", "mount everest"
- General knowledge: "python programming language"
Queries NOT Affected
Queries where answers contains genuinely distinct information:
- IP lookup: "ip" → answer is your public IP, infobox may explain what an IP is
- Calculator: "2+2" → answer is "4"
- Timezone: "time in tokyo"
- Hash lookup: hash values in answers
Solution
Engine Scoping
Deduplication is applied only to answers from the duckduckgo engine. The " More at Wikipedia" suffix and 200-rune substring heuristic are specific to DuckDuckGo's response format, where the same Wikipedia summary appears in both answers and infoboxes. When dedup runs, answers with empty text are dropped before the engine gate; non-empty answers from other engines (calculator, IP, timezone, etc.), including those with unknown Engine, pass through without deduplication. This prevents false positives like discarding a calculator result "4" whose text happens to appear in unrelated infobox content.
Deduplication Strategy
The function DeduplicateAnswers in internal/searxng/answer/deduplicate.go filters answers at the search response layer, before the result reaches JSON serialization or CLI formatting. This means both output modes benefit.
Algorithm (empty-text filter + engine gate + exact-case fast path + lazy lowercase fallback):
The dedup loop runs only when there is at least one non-empty infobox text; otherwise the original answers are returned unchanged. For each answer, the following steps are applied:
Empty-text filter (all engines):
- If
ans.Answer == "", the answer is dropped regardless of engine.
Engine gate:
- If
ans.Engine != "duckduckgo", the answer is retained without further checks.
Fast path (no allocation, DuckDuckGo answers only):
- Strip the known suffix
" More at Wikipedia"(exact case) that DuckDuckGo appends. - Take the first 200 characters of the stripped text as a prefix.
- Check if any infobox content contains this prefix using exact-case substring matching.
Lazy fallback (lowercase, built on first use, DuckDuckGo answers only): If the fast path did not match, a lowercased copy of all infobox contents is built once and reused for subsequent answers:
- Lowercase the answer text.
- Strip the known suffix
" more at wikipedia"(lowercase). - Take the first 200 characters of the stripped text as a prefix.
- Check if any lowercased infobox content contains this prefix (case-insensitive substring match).
If either path finds a match, the answer is considered a duplicate and is removed. The lazy fallback handles edge cases where the answer and infobox differ only in casing (e.g. "Apple" vs "apple").
Why Search Layer, Not Format Layer
Deduplication happens while normalizing the SearXNG response. The deduplicateAnswers wrapper in internal/searxng/deduplicate.go extracts infobox content and delegates to DeduplicateAnswers in internal/searxng/answer/deduplicate.go. This is done at the normalization layer rather than in formatResults() (format.go) so that:
- JSON output (
--jsonflag or MCP mode) also has clean results - CLI output benefits automatically
- Single point of truth — no scattered filtering logic
Edge Cases Handled
| Scenario | Behavior |
|---|---|
Empty answers |
Return as-is (no filtering) |
Empty infoboxes |
Return as-is (no filtering) |
Infobox with empty content |
Skip that infobox (no filtering against it) |
Empty answer text |
Dropped when dedup runs (non-empty infobox exists); kept otherwise (early return) |
| Case differences | Case-insensitive comparison |
| Non-DuckDuckGo engine | Retain when non-empty (no dedup applied) |
Empty Engine field |
Retain when non-empty (conservative default) |
Unknown Engine field |
Retain when non-empty (conservative default) |
| Multiple answers, some duplicate | Keep non-duplicates, remove DuckDuckGo duplicates |
Bounded Work Cap (CAND-33fe0b85-RUNTIME-003)
The dedup loop is O(len(answers) * len(infoboxes)) substring work, and for
each answer it scans every infobox twice (exact case, then lowercase) with
strings.Contains. A successful JSON body is already capped at
MaxResponseBodySize (2 MiB), but compact answer/infobox arrays can still
pack tens of thousands of entries into that envelope and force very large CPU
work before result limiting.
To bound that work, response normalization trims Answers and Infoboxes
to MaxAnswers and MaxInfoboxes (each 100) before the dedup call:
- Real SearXNG responses contain at most a handful of each, so 100 is a generous cap.
- Worst-case dedup work is bounded at
100 * 100 * 2 = 20,000Containscalls. - Truncation is logged with
slog.Warn(truncating answers/infoboxes before deduplication) and is visible in debug logs.
The cap is enforced in normalizeResponse (see
internal/searxng/response.go), not in deduplicateAnswers itself, so the
dedup function remains a pure data transform and tests can exercise it
without a searcher.
Implementation
- Function:
DeduplicateAnswers(answers []Answer, infoboxContents []string) []Answerininternal/searxng/answer/deduplicate.go - Called: Via
deduplicateAnswerswrapper ininternal/searxng/deduplicate.goduring response normalization after JSON unmarshalling, after theMaxAnswers/MaxInfoboxescap is applied, before the response is returned to formatting/output code - Tests:
internal/searxng/answer/deduplicate_test.go— unit tests forDeduplicateAnswers, covering empty inputs, exact/prefix match, "More at Wikipedia" suffix stripping, case insensitivity, engine scoping, distinct answers, mixed scenarios, empty answer skipping, and truncation boundary casesinternal/searxng/deduplicate_internal_test.go— focused adapter coverage that verifies onlyInfobox.Contentis projected into the canonical deduplication functionTestNormalizeResponseininternal/searxng/response_internal_test.go— integration coverage for typed-answer fallback before deduplication, answer/infobox caps before deduplication, no-op at cap, and the pathological-input time budget