LLM Evaluations

• Pitfalls of vibe testing

  Understand the pitfalls of “vibe testing” by examining real-world failure case studies like Air Canada.

• Deterministic vs probabilistic testing

  Navigate the differences between deterministic software testing and probabilistic LLM systems.

• Model evals vs application evals

  Distinguish between evaluating core foundation model capabilities and testing complete application workflows.

• The end-to-end evaluation workflow

  Implement an end-to-end evaluation process covering task definition, criteria, datasets, execution, analysis, and iteration.

Outcome · Understand the foundational need for and workflow of LLM evaluations compared to traditional software testing.

• Programmatic, human & LLM-as-a-Judge

  Categorize evaluation execution mechanisms into Programmatic, Human, and LLM-as-a-Judge approaches.

• Reference-based vs reference-free

  Understand reference-based vs reference-free evaluation.

• When a Golden Dataset is required

  Determine when a Golden Dataset answer key is required versus when evaluation can rely entirely on LLM judgment.

• Matching method to behaviour

  Choose the appropriate evaluation method based on whether the target behavior is deterministic, semantic, subjective, or operational.

Outcome · Select the appropriate evaluation execution method and paradigm for a given LLM testing scenario.

• Offline suites for pre-release testing

  Use offline evaluation suites for pre-release testing, version comparison, experimentation, and preventing performance regression.

• Post-deployment production risks

  Identify post-deployment production risks including unexpected inputs, emergent systematic failures, changing user behavior, and data drift.

• System metrics vs quality metrics

  Differentiate automatically captured system metrics such as latency, token usage, and errors from LLM-computed quality metrics such as hallucination or relevance.

• Continuous online evaluation pipeline

  Implement a continuous online evaluation pipeline through production logging, sampling, dashboarding, and alerting.

• Feeding failures back offline

  Feed important production failures back into the offline evaluation dataset.

Outcome · Design a dual-layer evaluation strategy that catches regressions before release and monitors quality after deployment.

• Core LLM capabilities

  Explore core LLM capabilities such as knowledge, reasoning, mathematics, coding, instruction following, long-context understanding, multimodal capability, safety, and tool use.

• Anatomy of a benchmark

  Analyze the anatomy of a benchmark through its dataset, prompts, run configuration, scoring method, and result aggregation.

• Public leaderboards

  Understand how public leaderboards compare foundation models.

• Benchmark pitfalls

  Identify benchmark pitfalls including data contamination, score saturation, benchmark overfitting, and configuration gaming.

Outcome · Critically assess public LLM benchmarks and understand how core model capabilities are formally measured.

• From MMLU to harder knowledge tests

  Trace the progression from benchmarks such as MMLU and TruthfulQA toward harder tests of knowledge and factual reliability.

• Human exams as benchmarks

  Understand the use of human exams in benchmarks such as AGIEval to compare human and model capabilities.

• GPQA, MMLU-Pro, SimpleQA, HLE

  Explore harder benchmarks such as GPQA, MMLU-Pro, SimpleQA, and Humanity's Last Exam.

• Why benchmarks keep getting harder

  Understand why benchmarks continually become harder as frontier models saturate older datasets.

Outcome · Understand the progression of knowledge benchmarks and select appropriate tests for increasingly capable models.

• Evaluating on your actual task

  Move beyond public benchmarks and evaluate models on the actual task your application needs to solve.

• Shortlisting candidate models

  Use leaderboards and aggregation platforms to shortlist candidate models based on capability, latency, context, and cost.

• Building a custom eval dataset

  Build a custom evaluation dataset for a real task such as Text-to-SQL.

• Like-for-like model comparison

  Compare shortlisted models using the same inputs, prompts, parameters, and scoring criteria.

• Estimating usage & token cost

  Estimate API usage and token costs and use techniques such as prompt caching where appropriate.

• Quality / latency / cost trade-off

  Select the model that offers the best task-specific quality / latency / cost trade-off, rather than simply choosing the highest-ranked general model.

Outcome · Evaluate and select the most capable and cost-effective foundation model for a specific application.

• Component, pipeline & application levels

  Structure evaluations across component level, pipeline level, and application level depending on what is being tested.

• Versioning tests, configs & results

  Maintain versioned test cases, evaluation configurations, prompts, models, and results.

• Experiment tracking & baselines

  Track experiments and compare new versions against established baselines.

• CI/CD quality gates

  Build automated CI/CD quality gates so changes cannot ship when important metrics regress.

• Self-improving golden dataset

  Create a self-improving loop by adding edge cases and failures discovered in production back into the offline golden dataset.

• Quality alongside latency & cost

  Track quality together with latency and cost so improvements in one area do not silently damage another.

Outcome · Build an automated evaluation suite that prevents regressions and improves continuously from real-world usage.

• Four levels of RAG evaluation

  Evaluate a RAG system at component, pipeline, application, and safety levels, rather than treating it as one black-box system.

Component-level evaluation

• Evaluating the retriever alone

  Evaluate the retriever independently to determine whether it finds the information required to answer the question.

• Diagnosing missed & noisy retrieval

  Diagnose cases where the retriever completely misses the relevant context or retrieves excessive irrelevant/noisy chunks.

• Retrieval metrics

  Measure retrieval effectiveness using metrics such as Contextual Recall, Contextual Precision, Recall@k, and other ranking-aware metrics where appropriate.

• Building the Golden Dataset

  Generate or curate relevant chunks and ideal answers for a Golden Dataset through hand-authoring, LLM-assisted generation, or tools such as DeepEval's synthetic-data generators.

• Evaluating the generator alone

  Evaluate the generator independently by providing known-good context and checking whether it produces faithful and relevant answers.

• Detecting unfaithful answers

  Detect unfaithful claims, irrelevant answers, missing information, and misuse of provided context.

• Component tests with DeepEval & RAGAS

  Implement component tests using frameworks such as DeepEval or RAGAS while understanding what each metric actually measures.

Pipeline-level evaluation

• The full retrieve → generate pipeline

  Evaluate the complete retrieval → context → generation pipeline.

• The RAG Triad

  Measure the RAG Triad:

• Contextual Relevancy

  Contextual Relevancy

• Faithfulness

  Faithfulness

• Answer Relevance

  Answer Relevance

• When good components interact badly

  Understand how individually strong components can still interact poorly when connected.

• Diagnosing pipeline-level failures

  Diagnose cases such as high retrieval precision but low overall contextual relevance or poor downstream answers.

• Locating the root failure

  Determine whether the root failure lies in retrieval, assembled context, or generation.

Application-level evaluation

• Evaluating from the user's perspective

  Evaluate the RAG system from the actual user's perspective rather than only testing internal components.

• Factual correctness

  Measure whether the final answer is factually correct given the available evidence.

• Answer completeness

  Evaluate completeness: does the answer address all important parts of the user's request?

• Tone, format & citation behaviour

  Evaluate application-specific requirements such as tone, style, persona, format, citation behavior, and instruction following.

• Operational metrics per query

  Track operational metrics including latency, token usage, cost per query, and error rates alongside quality.

Safety evaluation for RAG

• Harmful & disallowed responses

  Evaluate whether the system produces harmful, illegal, dangerous, biased, or otherwise disallowed responses.

• PII protection & redaction

  Test whether sensitive PII such as contact details, credentials, or payment information is appropriately protected or redacted.

• Direct & indirect prompt injection

  Test direct and indirect prompt injection, including malicious instructions hidden inside retrieved documents.

• Jailbreak attempts

  Test jailbreak attempts intended to bypass application guardrails.

• Poisoned retrieval content

  Evaluate whether retrieved malicious or misleading content can manipulate the model into leaking data or ignoring application instructions.

Outcome · Evaluate a RAG application comprehensively across retrieval quality, generator behavior, pipeline interaction, user-facing quality, operational performance, and safety.

• Six dimensions of agent evaluation

  Evaluate agentic systems at the level of decision-making, trajectories, tool usage, task completion, recovery, and safety.

Agent behavior & trajectory evaluation

• Why the final answer is not enough

  Understand why evaluating only the final answer is insufficient when an agent autonomously decides how to complete a task.

• Tool selection

  Evaluate whether the agent chooses the correct tool for a given situation.

• Tool argument correctness

  Test whether the agent generates accurate and properly formatted tool arguments.

• Trajectory quality

  Evaluate the sequence of actions taken by the agent and whether the trajectory is appropriate for the task.

• Loops, redundancy & early exit

  Detect unnecessary tool calls, redundant actions, loops, incorrect ordering, and premature termination.

• Task-completion success rate

  Measure overall task-completion success rate for complex multi-step user requests.

Failure recovery

• Injecting realistic failures

  Introduce tool failures, malformed results, unavailable services, missing information, and unexpected intermediate states.

• Noticing failure & choosing recovery

  Evaluate whether the agent notices failure and chooses an appropriate recovery strategy.

• Retry, fall back, replan, escalate

  Test whether it can retry, select an alternative tool, revise its plan, or escalate appropriately.

• Measuring recovery, not just success

  Measure successful recovery rather than only measuring success in ideal conditions.

Agent safety evaluation

• Unauthorized & destructive actions

  Test whether agents attempt unauthorized or destructive actions.

• Approval gates & permission boundaries

  Evaluate human-approval gates and permission boundaries around consequential actions.

• Resistance to prompt injection

  Test agent resistance to direct and indirect prompt injections designed to hijack its tool usage.

• Secret extraction & scope escape

  Attempt to manipulate an agent into revealing secrets, exceeding permissions, or using tools outside the intended scope.

• Cybersecurity capability benchmarks

  Use relevant cybersecurity benchmarks where appropriate to assess dangerous capabilities such as exploitation, reverse engineering, or other high-risk computer-use behavior.

• Outcome quality vs execution efficiency

  Track both outcome quality and execution efficiency through steps, latency, tool calls, token usage, and cost.

Outcome · Systematically evaluate whether an agent makes good decisions, uses tools correctly, completes tasks efficiently, recovers from failures, and remains within its security boundaries.