Adam Innes · Blog

Your RAG Search Keeps Missing the Part Number

· 5 min · ai, rag, postgresql, search

A customer asks whether a replacement seal fits the pump they bought. Your support assistant retrieves a beautifully written installation guide, explains the procedure and links the manual. Unfortunately, the guide belongs to the next model up.

That is the kind of failure I’d want to catch before spending time on a better system prompt. The answer generator might have followed its instructions perfectly. The retrieval layer handed it the wrong evidence.

Product support makes this especially visible because similar language can describe incompatible parts. A paragraph about a seal for an imaginary pump called PX-410 may read almost identically to one about PX-410B. The extra letter matters more than all the surrounding prose.

Give identifiers their own place

I’d start by separating product identity from the text used to explain a product. Store a canonical part number alongside each chunk, with its document identifier, revision and section. Preserve the original text too. If an ingestion process loses the identifier before search starts, changing the embedding model cannot put it back.

For a query that explicitly names a recognized part, use that identity to constrain the search. A normal database equality comparison is a useful tool here. Embedding similarity is a ranking signal, not proof that two identifiers refer to the same item.

Normalization needs a domain decision. Uppercasing might be correct for a catalog where identifiers are case insensitive. Removing every separator might merge two distinct identifiers. I’d define that behavior against the catalog’s actual rules and retain the original value for display and troubleshooting.

An unknown identifier deserves its own path. The assistant can ask for clarification or offer possible matches without silently selecting one. A confident answer about the closest looking product creates a worse support experience than admitting the lookup failed.

Build a retrieval test before a chatbot test

A useful experiment can be small. Create fictional manuals for several related products, including a deliberately incompatible pair. Give each manual a section about fit, another about installation and another about maintenance. The examples should be invented and clearly labeled so nobody mistakes them for product advice.

Then write questions that expose different failure modes. An exact part number should locate the corresponding fit section. A question describing a symptom without naming a product should exercise semantic matching. An ambiguous question should remain ambiguous. A question about a nonexistent product should not receive a fabricated compatibility claim.

For each question, record the chunks that would provide sufficient evidence. Some questions need more than one. A question comparing two products may need both specifications, so treating any single relevant hit as success would conceal missing context.

Run the retriever without generating an answer and save the ordered chunk identifiers. Measure how often the required evidence appears within the number of chunks you actually intend to send to the model. Also inspect the failures individually. An aggregate score can hide the fact that all the wrong answers involve suffixes in part numbers.

Keep this question set fixed while comparing approaches. Otherwise it is easy to make the search look better by changing what you ask it.

PostgreSQL is a reasonable place to prototype this if your application already uses it. Its text search documentation describes parsing documents into lexemes and ranking matches. This is language processing, so it should not be confused with an exact lookup of an identifier.

For example, the following query shows a lexical search over a proposed manual_chunks table. The application supplies a canonical product identifier as $1 and the user’s descriptive question as $2. Those are bound parameters, not strings to interpolate into SQL.

SELECT id, body
FROM manual_chunks
WHERE product_id = $1
  AND to_tsvector('english', body)
      @@ plainto_tsquery('english', $2)
ORDER BY ts_rank(
  to_tsvector('english', body),
  plainto_tsquery('english', $2)
) DESC, id
LIMIT 5;

This is an illustrative query for a small experiment, not a complete schema or indexing strategy. In particular, plainto_tsquery joins surviving terms with AND, so a wordy question can be too restrictive. That limitation is something the fixture questions should reveal. The built in ranking function here is also not BM25.

Add an embedding search as a separate candidate retriever. pgvector’s version 0.8.0 README documents vector search inside PostgreSQL and includes a hybrid search section. For a small corpus, I’d establish an exact nearest neighbor baseline before introducing an approximate index. That keeps approximation error out of the first comparison.

The question is not whether keyword search or embeddings win universally. The useful result is knowing which questions each approach misses in your collection. Exact identifiers, unusual terminology and descriptive questions give the two approaches different opportunities to help.

Combine rankings without pretending scores are interchangeable

A lexical rank and a vector distance do not share a meaningful numerical scale. Adding their raw values can make one dominate for accidental reasons.

One option documented in pgvector’s hybrid search guidance is reciprocal rank fusion. It combines positions in ranked result sets rather than assuming the underlying scores mean the same thing. A chunk that appears near the top of both searches receives support from both.

For this experiment, I’d retrieve candidates from each approach, combine the rankings and send the same final number of chunks to the answer generator. Holding that final budget constant matters. Giving the combined approach twice as much context changes more than retrieval quality.

Identity constraints should apply to both candidate searches when the product is known. Ranking a wrong product slightly lower is not equivalent to excluding it when the user has specified an exact model.

Restore the context a chunk lost

A different problem appears when the relevant paragraph says only “this assembly.” The product name might live several pages earlier. Chunking has removed the information needed to interpret the passage.

Anthropic’s September 2024 contextual retrieval article describes prepending explanatory context to chunks before embedding them and building a BM25 index. That is a useful technique to investigate when isolated passages lose their meaning.

I’d try deterministic context first where the source already provides it. A document title, product identifier and section heading can travel with the paragraph without asking a model to invent a summary. If generated context is useful, retain it separately from the source passage and review examples for unsupported claims. A fabricated identifier in an indexing summary can poison retrieval before the answering model sees anything.

Make the answer earn its confidence

Once retrieval works acceptably, bring the answer generator back. Use the same questions and examine whether its claims are supported by the retrieved passages. Finding the right document and writing a correct answer are separate achievements.

Keep permissions in the retrieval path too. In a system with private customer manuals, document access should follow the authenticated caller before content reaches the model. A prompt asking the assistant to avoid mentioning other customers is not a replacement for that check.

I would start this work with a handful of troublesome questions and readable search logs. The most useful discovery might be a missing product field or a bad chunk boundary. Fixing that gives the assistant better evidence on every request, and it gives you something much more concrete to debug than another plausible answer about the wrong pump.

← all posts