From “seems good” to “measurably good” to “live”
For two episodes I’ve been repeating the same nag: measure it, don’t assume it. Now we build the thing that measures — and then we put Nimbus HelpDesk AI online.
Here’s why evaluation matters more in RAG than in normal software. In a normal app, a function either returns the right value or it doesn’t; you write a test, it passes or fails. RAG answers are fuzzy — there are many acceptable phrasings, and “good” is a judgment, not an equality check. Worse, RAG systems fail silently. Change your chunk size, your threshold, or your prompt, and nothing crashes — the bot just quietly gets a bit better or a bit worse, and you’d never know from eyeballing a few questions. Evaluation is how you replace “I tried five questions and it felt fine” with numbers you can actually compare across changes.
Then we ship: a JSON endpoint, a minimal chat interface, and a deployment checklist so the bot survives contact with real users.
Part 1 — Building an evaluation set
An evaluation set is just a list of questions with what you expect to happen. You write it once, by hand, and it becomes the ruler you measure every future change against. Keep it small and representative — 15–30 questions covering your common cases plus the tricky ones.
Create eval-set.json:
[
{ "question": "How long do refunds take?",
"expect_source": "refund-policy.md", "should_answer": true,
"expect_contains": "5" },
{ "question": "I was charged twice, can I get my money back?",
"expect_source": "refund-policy.md", "should_answer": true,
"expect_contains": "duplicate" },
{ "question": "How do I create an account?",
"expect_source": "getting-started.md", "should_answer": true,
"expect_contains": "Sign Up" },
{ "question": "What's the weather in Mumbai today?",
"expect_source": null, "should_answer": false, "expect_contains": null },
{ "question": "Do you offer refunds in cryptocurrency?",
"expect_source": null, "should_answer": false, "expect_contains": null }
]
Each entry encodes three checkable expectations: which document should be retrieved, whether the bot should answer at all, and a key fact the answer must contain. Notice we deliberately include questions the bot should refuse — measuring the “I don’t know” path is as important as measuring correct answers.
Part 2 — Three things worth measuring
Metric 1: Retrieval hit rate
Before you can judge an answer, check whether retrieval even found the right document. This isolates where a failure lives — bad retrieval or bad generation. If the right chunk never came back, no prompt on earth could have answered correctly.
Metric 2: Refusal accuracy
Did the bot answer when it should, and refuse when it should? A bot that answers everything (including nonsense) and a bot that refuses everything both score terribly here. This catches threshold drift instantly.
Metric 3: Answer quality (LLM-as-judge)
Grading fuzzy answers by hand doesn’t scale. The standard trick is LLM-as-judge: use a language model to grade the answer against the retrieved context on a simple scale. It’s not perfect, but it’s consistent and fast, and it correlates well enough with human judgment to catch regressions.
💡 What is “LLM-as-judge”? You ask a model to grade another model’s output — “on a scale of 1–5, how well does this answer match this reference?” It works because judging a finished answer is a much easier task than producing one. Treat its scores as a relative signal (did this change help or hurt?), not an absolute truth.
The evaluation script
<?php
// evaluate.php — score the bot against the eval set
require __DIR__ . '/bot.php'; // exposes ask() and retrieveHybrid()
require __DIR__ . '/chat.php';
function judge(string $question, string $answer, string $context): int {
$prompt = "Grade this support answer from 1 to 5.\n"
. "5 = fully correct and supported by the reference; 1 = wrong or unsupported.\n\n"
. "Question: {$question}\n\nAnswer: {$answer}\n\n"
. "Reference context:\n{$context}\n\n"
. "Reply with ONLY the number.";
$reply = chat([['role' => 'user', 'content' => $prompt]], 0.0);
return (int) filter_var($reply, FILTER_SANITIZE_NUMBER_INT);
}
$cases = json_decode(file_get_contents(__DIR__ . '/eval-set.json'), true);
$hits = $refusalOK = 0;
$qualityScores = [];
$answerable = 0;
foreach ($cases as $c) {
$chunks = retrieveHybrid($c['question']);
$sources = array_column($chunks, 'source');
$answer = ask($c['question']);
$refused = str_contains($answer, "I don't have that information");
// Metric 1: retrieval hit (only meaningful for answerable cases)
if ($c['expect_source'] && in_array($c['expect_source'], $sources, true)) {
$hits++;
}
// Metric 2: refusal accuracy
if ($refused === !$c['should_answer']) {
$refusalOK++;
}
// Metric 3: quality (answerable cases only)
if ($c['should_answer']) {
$answerable++;
$context = implode("\n", array_column($chunks, 'content'));
$qualityScores[] = judge($c['question'], $answer, $context);
}
$status = $refused ? 'REFUSED' : 'answered';
printf("[%s] %s\n", $status, $c['question']);
}
$n = count($cases);
printf("\n— Results over %d cases —\n", $n);
printf("Retrieval hit rate : %d/%d\n", $hits, $answerable);
printf("Refusal accuracy : %d/%d\n", $refusalOK, $n);
printf("Avg answer quality : %.2f / 5\n",
$qualityScores ? array_sum($qualityScores) / count($qualityScores) : 0);
Run it:
php episode-08/evaluate.php
[answered] How long do refunds take?
[answered] I was charged twice, can I get my money back?
[answered] How do I create an account?
[REFUSED] What's the weather in Mumbai today?
[REFUSED] Do you offer refunds in cryptocurrency?
— Results over 5 cases —
Retrieval hit rate : 3/3
Refusal accuracy : 5/5
Avg answer quality : 4.67 / 5
Now — and this is the whole point — go back and change something. Drop your threshold to 0.05, or switch retrieveHybrid() back to plain retrieve(), and re-run. The numbers move. You now have an objective way to answer “did that change help?” instead of guessing. That’s the superpower.
Part 3 — Cost and latency
Two numbers your finance team and your users care about.
Latency — how long a question takes. Wrap the call to time it:
$start = microtime(true);
$answer = ask($question);
printf("Answered in %.2f s\n", microtime(true) - $start);
Most of that time is the two API round-trips (embedding the question, then generation). A rerank pass adds a third. This is your budget for deciding whether the extra quality of reranking is worth the wait.
Cost — driven by tokens (remember Episode 2: ~4 characters ≈ 1 token). A single support answer is cheap, but multiply by thousands of questions a day and it adds up. The rough shape:
- Embedding the question: tiny — a few tokens, a fraction of a cent.
- Generation: the big one — you pay for the system prompt + all retrieved context + the question + the answer, every single time.
That last point is a lever you now understand deeply: retrieving fewer, better chunks (Episodes 5 and 7) isn’t just about accuracy — it directly lowers your bill, because context is the bulk of what you pay for on each call. Good retrieval is good economics.
⚠️ Gotcha — embeddings are a one-time cost, mostly. You embed each chunk once at ingestion, not per question. Re-embed only when a document changes. Don’t rebuild the whole index on every deploy; re-index only what changed.
Part 4 — Shipping it
Time to get Nimbus HelpDesk AI off the command line and onto the web.
Refactor: separate logic from interface
Our ask() function currently lives in a file with CLI code at the bottom. Move the reusable pieces into bot.php (the ask(), retrieveHybrid(), etc.) with no output code — just functions. Then both the CLI and the web share the same brain.
A JSON API endpoint
<?php
// api.php — HTTP endpoint for the bot
header('Content-Type: application/json');
require __DIR__ . '/bot.php';
$question = trim($_POST['question'] ?? '');
if ($question === '' || mb_strlen($question) > 500) {
http_response_code(400);
echo json_encode(['error' => 'Please ask a question (1–500 characters).']);
exit;
}
try {
echo json_encode(['answer' => ask($question)]);
} catch (Throwable $e) {
http_response_code(500);
echo json_encode(['error' => 'Something went wrong. Please try again.']);
// Log $e->getMessage() server-side — never show it to the user.
}
Note the input guard (length limit) and the try/catch that hides internal errors from users while you log them privately.
A minimal chat interface
<!-- index.php — a tiny chat UI -->
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Nimbus HelpDesk AI</title>
<style>
body { font-family: system-ui, sans-serif; max-width: 640px; margin: 40px auto; }
#log { border: 1px solid #ddd; border-radius: 8px; padding: 16px; min-height: 240px; }
.q { color: #1E3A8A; font-weight: 600; margin-top: 12px; }
.a { white-space: pre-wrap; margin: 4px 0 12px; }
form { display: flex; gap: 8px; margin-top: 12px; }
input { flex: 1; padding: 10px; }
button { padding: 10px 16px; }
</style>
</head>
<body>
<h2>Nimbus HelpDesk AI</h2>
<div id="log"></div>
<form id="f">
<input id="q" placeholder="Ask about refunds, accounts, billing…" autocomplete="off" required>
<button>Ask</button>
</form>
<script>
const log = document.getElementById('log');
document.getElementById('f').addEventListener('submit', async (e) => {
e.preventDefault();
const q = document.getElementById('q');
const question = q.value.trim();
if (!question) return;
log.innerHTML += `<div class="q">You: ${question}</div>`;
q.value = '';
const res = await fetch('api.php', {
method: 'POST',
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
body: 'question=' + encodeURIComponent(question)
});
const data = await res.json();
log.innerHTML += `<div class="a">${data.answer || data.error}</div>`;
log.scrollTop = log.scrollHeight;
});
</script>
</body>
</html>
Point PHP’s built-in server at the folder and open it:
php -S localhost:8000 -t episode-08/
There it is — a working web chatbot answering from Nimbus’s real knowledge base, with citations, running on plain PHP and MySQL.
Deployment checklist
Before real users touch it:
- Secrets in environment variables, never in code or version control. Keep
.envout of your repo. - HTTPS only — you’re sending user questions to an API; encrypt the transport.
- Rate limiting — cap requests per IP/session so nobody runs up your API bill.
- Logging & monitoring — log questions, retrieved sources, latency, and cost. This is also your richest source of new eval-set questions.
- Re-indexing plan — a script (or admin button) to re-chunk and re-embed when docs change; only touch what changed.
- Guardrails from Episode 7 stay on: hardened prompt, delimited context, the empty-retrieval guard.
- Never let the bot take real actions (issue refunds, change accounts) on its own. It advises; humans execute.
Prompt spotlight: the judge prompt
The evaluator has its own prompt, and it follows a rule worth stealing: make the model’s job as narrow as possible.
Grade this support answer from 1 to 5.
5 = fully correct and supported by the reference; 1 = wrong or unsupported.
...
Reply with ONLY the number.
Three deliberate choices: a tiny scale (1–5, not 1–100 — models are far more consistent over a small range), an anchored rubric (we define what 5 and 1 mean, so the score isn’t arbitrary), and “reply with ONLY the number” so the output is trivial to parse. A judge you can’t parse reliably isn’t a judge. The broader lesson from this whole series: when you want a machine-usable result from an LLM, constrain the output format hard.
Try it yourself
- Run an A/B on a real change. Record your three scores. Switch
retrieveHybrid()toretrieve(), re-run, compare. Which retrieval actually wins on your data? - Grow the eval set. Add five questions from edge cases you’re unsure about. A bigger ruler catches more regressions.
- Trace the cost. Print the character count of the full prompt (system + context + question) for a few questions. Multiply by your expected daily volume to sanity-check your bill.
- Ship it locally. Get the web UI running and hand it to a colleague. Watching a real person ask questions you never thought of is the best eval there is.
Recap — and the series so far
- RAG systems fail silently, so evaluation is non-negotiable — it turns “feels fine” into comparable numbers.
- Measure three things: retrieval hit rate, refusal accuracy, and answer quality (via LLM-as-judge).
- Watch latency and cost; since you pay for context on every call, good retrieval is also cheaper retrieval.
- Ship by separating logic (
bot.php) from interface (api.php,index.php), guarding inputs, hiding errors, and following a deployment checklist. - Keep your guardrails on and keep the bot an advisor, not an executor.
Look back at what you’ve built across eight episodes: from “what even is RAG?” to a deployed, evaluated, cited, guardrailed support bot — in nothing but PHP and MySQL 8, no vendor lock-in, no exotic infrastructure. You understand every layer, because you built every layer.
What’s next — the bonus project
The core series is complete. But a pattern you can only apply to one project isn’t really yours yet. So in a two-part bonus, we’ll prove RAG transfers to a completely different domain — a Local News assistant that pulls fresh articles from an open news API, embeds them, and answers questions about today’s news with citations back to the source articles.
It’s the same skeleton you already know, aimed at live, constantly-changing data instead of static docs — which surfaces new challenges: freshness, re-indexing, deduplication, and recency-aware answers. Those two posts are code- and prompt-forward: less theory, more building, because you already own the theory.
- Bonus 1 — Build it: fetch and ingest live news, adapt the chunking and metadata, wire up retrieval.
- Bonus 2 — Prompts & polish: news-specific generation, date/recency handling, multi-article citations, and edge cases.
Bring your bot.php — we’re about to point it at the world.