Evaluating LLM Agents: Confidence Through Pre-Production Testing

This post focuses on Pre-production phase evaluation for LLM applications.
Before continuing, I would recommend reading the Design-Phase LLM Agent Evaluations article, which covers visual debugging, tracing, and unit testing during the design phase.
👉 Design-Phase LLM Agent Evaluations

---

Design-phase guardrails help your LLM application avoid obvious failures.
But before shipping to real users, there’s a more important question:

Is this version actually better than the last one — and did we accidentally break anything?

That’s the role of pre-production testing.

Unlike traditional software, LLM systems are:

• non-deterministic
• sensitive to small changes
• dependent on evolving models and retrieval layers

Pre-production testing isn’t about absolute correctness.
It’s about confidence, comparison, and regression prevention.

---

1. Where Pre-Production Testing Fits in the Lifecycle

By now, you’ve already:

• built your agent
• visualized execution paths
• added tracing
• written deterministic unit tests

Those steps answer:

“Does the system behave correctly in isolation?”

Pre-production testing answers a different question:

“How does this system behave on realistic inputs compared to earlier versions?”

It sits between design-phase testing and production monitoring.

---

2. What Pre-Production Testing Is (and Is Not)

It is:

• comparative
• trend-based
• regression-focused

It is not:

• strict pass/fail testing
• proof of correctness
• a replacement for production monitoring

Because LLMs are probabilistic, success is measured in patterns, not guarantees.

---

3. High-Impact Evaluation Datasets (The Foundation)

Every pre-production workflow starts with a dataset.

A dataset is a curated set of inputs your application should handle well.

Quality Beats Quantity

Most teams get strong signal from 10–50 examples if chosen carefully.

What Makes a Good Dataset

• frequently occurring user queries
• known failure cases
• edge cases found during development
• inputs that previously caused regressions

Sources of Dataset Examples

1. Manually curated cases

Encode product expectations and what good looks like.

2. Production logs

High-signal, real-world behavior.

Rule of thumb: Every production bug should eventually become a dataset entry.

3. Synthetic data

Useful for coverage, but should augment — not replace — real data.

Datasets should be treated as first-class artifacts:

• versioned
• reviewed
• continuously evolving

Example: A Small, High-Impact Dataset

A small, well-chosen dataset often surfaces more real problems than thousands of synthetic examples.

We can create a dataset in langsmith from Datasets & Experiments Tab

1. Click on + New Dataset button.
2. Choose Create from scratch option or Import from file option.
3. Enter the dataset name and description.
4. Click on Create button.

We can add examples either

• manually or
• directly from production traces (As shown below)

---

4. LLM-as-Judge: Evaluating with LLMs

Once you have a dataset, the next challenge is evaluation.

Hard-coded rules don’t scale well for natural language outputs.
This is where LLM-as-Judge evaluators come in.

LLM-as-Judge encodes human judgment into structured prompts.

Common Evaluation Dimensions

• Correctness — is the response factually correct?
• Relevance — does it actually answer the user’s question?
• Hallucinations — does it invent unsupported information?
• Safety and tone — is the response appropriate and compliant?

Types of Evaluators

Reference-based evaluators

Compare the model output against a known ground-truth answer.

Used when:

• correctness matters
• expected outputs are well-defined
• mistakes are clearly identifiable

Reference-free evaluators

Judge qualities like usefulness, safety, or clarity without a reference.

Used when:

• multiple valid answers exist
• subjective quality matters
• exact correctness is less important than behavior

Most real systems use a mix of both.

---

Creating Evaluators in Practice

Evaluators don’t need to be hard-coded or built from scratch every time.

Teams typically create evaluators in two ways:

1. From code

Custom evaluators allow:

• version control
• CI-style workflows

2. From the LangSmith UI

Useful for:

• rapid iteration
• experimenting with scoring criteria

Langchain Provides some Prebuilt evaluators as well such as Correctness, Hallucination, etc (As seen in the above picture) which can be used directly or can be customized as needed.

Best Practices

• prefer binary or very small scoring scales
• write prompts a human could realistically follow
• regularly audit evaluator decisions
• treat judges as signals, not truth

An example of a scoring scale for a relevance evaluator is shown below

---

5. Running Experiments

With datasets and evaluators in place, you can now run experiments.

An experiment is:

Running your application across the dataset and evaluating every output.

Pre-production experiments allow you to:

• compare prompt versions
• evaluate model upgrades
• assess retrieval or tool changes

Running Experiments in Practice

1. Running Experiments from the LangSmith UI

LangSmith allows you to:

• select a dataset
• choose one or more evaluators
• run experiments without writing code

This is especially useful for:

• quick comparisons
• exploratory testing

2. Running Experiments from Code

Code-driven experiments enable:

• automation
• CI-like workflows

Below is a simplified example of how to run experiments from code.

import { evaluate } from "langsmith/evaluation";
import { createLLMAsJudge } from "openevals";

const correctnessEvaluator = createLLMAsJudge({
  prompt: `
You are evaluating an AI assistant for correctness.
Return ONLY 1 (correct) or 0 (incorrect).

<input>{inputs}</input>
<output>{outputs}</output>
<reference>{reference_outputs}</reference>
`,
  feedbackKey: "correctness",
  model: "openai:gpt-4o-mini",
});

async function runAgent(inputs: { messages: string }) {
  const output = await myAgent.invoke(inputs.messages);
  return { output };
}

await evaluate(runAgent, {
  data: "sprint-planning-dataset",
  evaluators: [correctnessEvaluator],
  experimentPrefix: "sprint-planning-v2",
});

This run will:

• execute the agent across the dataset
• score each output using the evaluator
• store results as a named experiment

From there, you can:

• compare against previous experiments
• inspect failures example-by-example
• track regressions over time

We can see the results in the Experiments Tab of the dataset page.

We can check the individual outputs of each example by navigating to the experiment from the dataset page.

6. Testing Agents in Pre-Production

Agents introduce additional complexity because they control execution paths.

A useful strategy is to evaluate agents at three levels.

6.1 Final Output (Black-Box)

• did the agent complete the task?
• simple and end-to-end
• hard to debug failures

6.2 Single-Step Evaluation

• was the correct tool chosen?
• were inputs valid?
• faster and more targeted

6.3 Trajectory Evaluation

• number of steps taken
• tool call order
• loop detection
• efficiency and stability

Good agents aren’t just correct — they’re predictable and efficient.

7. What Pre-Production Testing Gives You

By the end of pre-production testing, you should be able to confidently say:

“This version improves X without hurting Y.”
“Any regressions are understood and explicitly accepted.”
“We know how this system fails.”
“Shipping this is a decision, not a gamble.”

That confidence is why every meaningful change should follow the same path:

Design phase → Pre-production testing → Production

---

A Useful Mental Model

LLM development works best as a loop, not a straight line:

Design → Pre-Prod → Production
   ↑                      ↓
   └─────── Feedback ─────┘

Every production failure feeds back into:

• new datasets
• new evaluators
• stronger guardrails

That feedback loop is where real reliability actually comes from.

Design-phase testing helps you build the right thing. Pre-production testing helps you ship the right version.

If design-phase testing answers:

“Does this agent work?”

Pre-production testing answers:

“Should this version go live?”

That confidence, not perfect scores, is the real goal.

---

So here’s to:

Building with visibility.
Testing with intent.
Shipping with confidence.

Toodaloo 🥂

References:

• LangSmith Evaluation Documentation