Episode 5 — Retrieval: Top-K, Thresholds & Filtering

Our search is fast, but it isn’t smart

After Episode 4, MySQL ranks our chunks quickly and returns the top 3. Let’s test it with a question Nimbus’s knowledge base can’t possibly answer:

php episode-04/search.php "What's the weather in Mumbai today?"

#1  score=0.081  [refund-policy.md]
Customers may request a full refund within 14 days of purchase…

#2  score=0.074  [faq.md]
If you were billed twice for the same subscription…

#3  score=0.066  [getting-started.md]
Create your Nimbus account by clicking Sign Up…

It returned three chunks about refunds and signups for a question about weather. Not because it’s broken — because we never told it it was allowed to return nothing. “Give me the top 3” always produces a top 3, even when the best match is terrible.

Look at those scores though: 0.08, 0.07, 0.07. Compare that to the refund question in Episode 3, which scored 0.612. The system already knows these are bad matches — it’s screaming it through the numbers. We just aren’t listening.

This episode is about listening. We’ll turn a blunt “top 3” into genuine retrieval: returning the right number of good enough chunks, from the right subset of documents — and returning nothing at all when that’s the honest answer. This matters enormously for the next episode, because whatever we hand the model becomes the facts it answers from. Garbage retrieved is garbage generated.

Dial 1: Top-K — how many chunks?

Top-K simply means “return the K best matches.” K is a number you choose, and it’s a real trade-off.

Too small (K=1): You get only the single best chunk. If the answer spans two chunks — say the refund window is in one and the processing time in another — the bot gives an incomplete answer. Worse, if retrieval’s top pick is slightly wrong, there’s no backup.

Too large (K=20): You drown the model. You pay for far more tokens, responses slow down, and — counter-intuitively — accuracy often drops. Researchers call this the “lost in the middle” problem: language models pay closest attention to the beginning and end of a long context and can genuinely overlook facts buried in the middle. Twenty mediocre chunks are worse than three excellent ones.

For a support bot, K between 3 and 5 is the sweet spot: enough for an answer that spans a couple of passages, small enough to stay sharp and cheap.

💡 Why “K”? It’s just the mathematical convention for “some number you pick.” You’ll see “top-k” everywhere in search and AI. Nothing deeper than that.

Dial 2: The similarity threshold — a quality floor

Top-K controls how many. A threshold controls how good they have to be.

The idea is simple: set a minimum score, and discard any chunk that scores below it. If nothing clears the bar, return an empty list — which is the system’s way of saying “I have nothing relevant.” That empty list is what will let our bot say “I don’t know” in Episode 6 instead of inventing an answer.

The hard part is picking the number, and here’s the thing most tutorials won’t tell you:

⚠️ There is no universal threshold. Cosine scores are relative, not absolute. They shift depending on your embedding model, your content, and how people phrase questions. A “good” score with one model might be 0.75; with another, 0.35. Anyone who hands you a magic number without seeing your data is guessing.

So we don’t guess — we measure. Write a handful of questions you know should match your docs, and a handful you know shouldn’t, then look at the actual scores.

<?php
// calibrate.php — find a sensible threshold for YOUR content
require __DIR__ . '/db.php';
require __DIR__ . '/embed.php';
require __DIR__ . '/vector.php';

$shouldMatch = [
    'How do I get my money back?',
    'I was charged twice',
    'How long do refunds take?',
    'How do I create an account?',
];

$shouldNotMatch = [
    "What's the weather in Mumbai today?",
    'Who won the cricket match last night?',
    'Can you write me a poem about the sea?',
];

function topScore(string $question): float {
    $vec  = normalize(getEmbedding($question));
    $stmt = db()->prepare(
        'SELECT dot_product(embedding, CAST(:qvec AS JSON)) AS score
           FROM chunks
       ORDER BY score DESC
          LIMIT 1'
    );
    $stmt->bindValue(':qvec', json_encode($vec));
    $stmt->execute();
    return (float) ($stmt->fetchColumn() ?: 0);
}

echo "SHOULD match (relevant questions)\n";
$good = [];
foreach ($shouldMatch as $q) {
    $s = topScore($q);
    $good[] = $s;
    printf("  %.3f  %s\n", $s, $q);
}

echo "\nSHOULD NOT match (irrelevant questions)\n";
$bad = [];
foreach ($shouldNotMatch as $q) {
    $s = topScore($q);
    $bad[] = $s;
    printf("  %.3f  %s\n", $s, $q);
}

printf("\nLowest good score : %.3f\n", min($good));
printf("Highest junk score: %.3f\n", max($bad));
printf("→ Set your threshold between these two, e.g. %.2f\n",
    (min($good) + max($bad)) / 2);

Running it gives something like:

SHOULD match (relevant questions)
  0.612  How do I get my money back?
  0.548  I was charged twice
  0.501  How long do refunds take?
  0.437  How do I create an account?

SHOULD NOT match (irrelevant questions)
  0.081  What's the weather in Mumbai today?
  0.094  Who won the cricket match last night?
  0.112  Can you write me a poem about the sea?

Lowest good score : 0.437
Highest junk score: 0.112
→ Set your threshold between these two, e.g. 0.27

That gap between 0.112 and 0.437 is your safety margin, and now your threshold is an evidence-based decision instead of a vibe. Sit it comfortably in the middle — around 0.25–0.30 — so genuine questions always pass and nonsense always fails.

Dial 3: Filtering and diversity

Two smaller refinements that punch above their weight.

Metadata filtering narrows the search before scoring — only current policies, only public articles, only a given product. Because we indexed source and doc_type back in Episode 4, this is both easy and fast.

Per-source diversity solves a subtler problem. Sometimes all K chunks come from the same document — three consecutive passages of the refund policy that mostly repeat each other. You’ve burned your whole context budget on one narrow slice. Capping how many chunks any single source contributes forces a broader, more useful view.

Building the retrieve() function

Let’s fold all of it into one clean function. This is the piece Episode 6 plugs straight into, so it’s worth building properly.

<?php
// retrieve.php — turn a question into the best available context
require __DIR__ . '/db.php';
require __DIR__ . '/embed.php';
require __DIR__ . '/vector.php';

function retrieve(string $question, array $options = []): array {
    $topK          = $options['top_k']          ?? 4;
    $minScore      = $options['min_score']      ?? 0.27;
    $maxPerSource  = $options['max_per_source'] ?? 2;
    $maxChars      = $options['max_chars']      ?? 4000;
    $docType       = $options['doc_type']       ?? null;

    $queryVec = normalize(getEmbedding($question));

    // Fetch a few extra candidates so diversity capping has room to work
    $sql = 'SELECT source, title, content,
                   dot_product(embedding, CAST(:qvec AS JSON)) AS score
              FROM chunks';
    if ($docType !== null) {
        $sql .= ' WHERE doc_type = :doc_type';
    }
    $sql .= ' HAVING score >= :min_score
              ORDER BY score DESC
              LIMIT :k';

    $stmt = db()->prepare($sql);
    $stmt->bindValue(':qvec', json_encode($queryVec));
    $stmt->bindValue(':min_score', $minScore);
    $stmt->bindValue(':k', $topK * 3, PDO::PARAM_INT);
    if ($docType !== null) {
        $stmt->bindValue(':doc_type', $docType);
    }
    $stmt->execute();

    $results    = [];
    $perSource  = [];
    $charsUsed  = 0;

    foreach ($stmt->fetchAll() as $row) {
        $src = $row['source'];

        // Diversity: don't let one document dominate
        if (($perSource[$src] ?? 0) >= $maxPerSource) {
            continue;
        }
        // Context budget: stop before the prompt gets bloated
        if ($charsUsed + mb_strlen($row['content']) > $maxChars) {
            continue;
        }

        $results[] = [
            'source'  => $src,
            'title'   => $row['title'],
            'content' => $row['content'],
            'score'   => round((float) $row['score'], 4),
        ];

        $perSource[$src] = ($perSource[$src] ?? 0) + 1;
        $charsUsed      += mb_strlen($row['content']);

        if (count($results) >= $topK) {
            break;
        }
    }

    return $results; // empty array = nothing relevant found
}

Four things are happening, in deliberate order: filter by metadata, cut anything below the threshold, cap per-source dominance, and respect a character budget so the prompt can’t balloon.

💡 Why fetch topK * 3? If we asked MySQL for exactly 4 rows and then threw two away for diversity, we’d end up with 2. Over-fetching gives the filters room to discard without leaving us short. It’s cheap — we’re still returning a handful of rows, not the whole table.

Trying it out

<?php
// test-retrieve.php
require __DIR__ . '/retrieve.php';

$question = $argv[1] ?? 'How do I get my money back?';
$chunks   = retrieve($question);

echo "Question: $question\n\n";

if (empty($chunks)) {
    echo "No relevant information found.\n";
    exit;
}

foreach ($chunks as $i => $c) {
    printf("#%d  score=%.3f  [%s]\n%s\n\n",
        $i + 1, $c['score'], $c['source'],
        mb_substr($c['content'], 0, 140) . '…'
    );
}

The relevant question behaves as before. But now:

php episode-05/test-retrieve.php "What's the weather in Mumbai today?"

Question: What's the weather in Mumbai today?

No relevant information found.

That empty result is the most important output in this episode. A retrieval system that knows when it has nothing is what separates a trustworthy support bot from a confident liar. Next episode, that empty array becomes a polite “I don’t know — here’s how to reach a human.”

Design spotlight: tuning is a product decision, not a technical one

Where you set the threshold encodes a judgment about which mistake costs more.

Set it high (strict): the bot rarely says anything wrong, but says “I don’t know” more often — including sometimes when it could have helped. Frustrating, but safe.

Set it low (permissive): the bot answers almost everything, but occasionally answers from weakly-related chunks and gets things wrong. Helpful-feeling, but risky.

For Nimbus — where a wrong refund policy could mean a real financial promise the company must honour — lean strict. An honest “let me connect you with support” costs a little friction. A confidently wrong answer costs trust, and possibly money.

That’s a business call, not an engineering one. Make it deliberately, and revisit it once you see real questions.

Try it yourself

1. Run the calibration on your own docs. Your numbers will differ from ours — that’s the entire point. Write 5 questions that should match and 5 that shouldn’t, and read the gap.
2. Push the threshold to extremes. Set min_score to 0.9 (almost everything is rejected) and then 0.01 (weather questions return refund policies). Feel both failure modes.
3. Test diversity. Set max_per_source to 1, then 5, on a question your refund policy answers well. Watch how the mix of sources changes.
4. Try a filtered retrieval: retrieve($question, ['doc_type' => 'policy']) and confirm only policy chunks come back.

Recap & what’s next

• Raw “top 3” search always returns something, even for questions your docs can’t answer — that’s a hallucination waiting to happen.
• Top-K controls how many chunks come back; 3–5 balances completeness against noise and the “lost in the middle” effect.
• A similarity threshold sets a quality floor — and must be calibrated on your own content, never copied from a tutorial.
• Metadata filters narrow the field, and per-source caps stop one document from hogging the context.
• Returning an empty result is a feature: it’s how the bot learns to say “I don’t know.”
• We now have a reusable retrieve() function — the complete “R” in RAG.

We have the retrieval half of RAG finished and trustworthy. In Episode 6 — Generation: Grounded Answers with Citations, we finally close the loop: feed those retrieved chunks into a carefully built prompt, get the model to answer using only them, make it cite which article each fact came from, and wire up the empty-result case so the bot gracefully admits when it doesn’t know. That’s the episode where Nimbus HelpDesk AI becomes a real, working support bot.