Spanish full-text search collapses words that mean different things
2026-09-02 · Vorluno
The agent answers from a knowledge catalogue the business fills in — services, prices, policies, whatever it should know. Search over that catalogue is Postgres full-text search, which is the right default: no extra infrastructure, no embedding pipeline to keep in sync, ranking that is good enough for a few hundred short documents. Then real messages arrived and the hit rate was much worse than the corpus size suggested it should be.
Two independent problems, stacked. The first is the Spanish stemmer: it reduces words to a root aggressively enough that words with unrelated meanings collapse into the same token. A search for one concept matches documents about a different one, and — worse for us — ranking stops discriminating between them, because to the index they are the same word.
The second is subtler and is a property of the query parser, not the language. `websearch_to_tsquery` joins bare terms with AND. A person writing to a business on WhatsApp does not send a keyword: they send a sentence, and the more they explain themselves, the more terms have to appear *in the same document* for anything to match at all. Longer question, fewer results — the exact opposite of the intuition, and it degrades silently: an empty result set looks the same whether the corpus is missing the answer or the query demanded too much.
Neither problem announces itself. Both produce a plausible-looking outcome — a wrong-but-related document, or no document — and both are easy to misread as "the business has not written enough content yet". The way we found them was not by reading the query code: it was by running actual customer sentences against the actual index and comparing what came back to what should have.
The general form is worth keeping: a search stack has at least three places where meaning is silently transformed — how the text is tokenised, how the query is parsed, and how results are ranked — and each of them can be individually reasonable while the composition is useless. Testing search with the terms you would type yourself hides all three, because you are unconsciously writing queries the stack happens to handle.