LLM Evals Are Not a Replacement for A/B Tests — Heres the Right Funnel

Why This Matters Now

We've all been there: a new prompt tweak, a model swap, a guardrail rewrite. The LLM judge says it's better. The team ships it. Then the metrics flatline or — worse — a secondary metric like session length or retention starts to drift. The eval missed something.

At Spotify, only about 12% of A/B tests end in a shipped positive result. But 64% produce valid learning: a regression caught, an idea ruled out, a hypothesis refined. The win rate understates the value of experimentation. Now we have a new capability: LLM evals can assess relevance, coherence, tone, and intent alignment at scale, faster and cheaper than human annotation.

The trap is treating evals as a replacement for experiments. They're not. The right relationship is a funnel, not a fork — as Schultzberg and Ottens (2024) describe in their evaluation funnel framework. Evals belong before your experiment, not instead of it.

Source: Spotify Engineering Blog

The Evaluation Funnel: Verification Then Validation

Schultzberg and Ottens draw a clean line between verification and validation:

• Verification (evals): Does the output conform to quality standards? Is the response coherent, non-toxic, aligned with the prompt?
• Validation (experiments): Do real users respond as predicted? Does the change drive the business outcome it was meant to?

Evals discard non-promising candidates before they consume experiment bandwidth. They raise the hit rate of the experiments that follow. But they can't tell you whether users who received the improved version actually had better long-term outcomes — whether the fix prevented the slow erosion of trust that eventually leads to churn. That question requires an experiment.

Two Calibration Layers, One Feedback Loop

Evals are proxies. They substitute a score for an outcome you actually care about. Now LLM judges add a second calibration layer on top of traditional quantitative metrics (ranking scores, precision, recall). Both layers need validation against online outcomes. Both can drift.

When the judge says Variant A is better, does it actually deliver a better user experience, or is the judge rewarding surface patterns that don't drive outcomes? For example, when Anthropic released Opus 4.5, Qodo's coding evals showed no improvement, but the model had improved substantially on longer tasks — a controlled experiment would have surfaced that. Miscalibration runs both ways.

# Simplified calibration loop pseudocode
import numpy as np

def calibrate_eval_weights(eval_scores, experiment_outcomes):
    """
    Adjust eval component weights to better predict online outcomes.
    
    Args:
        eval_scores: dict of eval dimensions (coherence, relevance, tone)
        experiment_outcomes: dict of business metrics (retention, engagement)
    Returns:
        updated_weights: dict of adjusted dimension weights
    """
    # Simple linear regression to find which eval dimensions
    # best predict the experiment outcome
    X = np.column_stack([eval_scores[d] for d in eval_scores])
    y = np.array(list(experiment_outcomes.values()))
    
    # Fit coefficients (weights) that minimize prediction error
    weights, _, _, _ = np.linalg.lstsq(X, y, rcond=None)
    
    return {dim: w for dim, w in zip(eval_scores.keys(), weights)}

What Evals Give You — and What They Don't

Beyond the dimensions you're measuring are the ones you aren't measuring. At Spotify, teams roll back about 42% of launched experiments to prevent regression in secondary metrics: session length dropping, crash rates climbing, retention eroding. No evals or offline evaluation flagged those.

Guardrail Metrics Matter

As described in Spotify's work on guardrail metrics, the point of a guardrail is to watch dimensions you care about but aren't optimizing for. An eval measures quality of implementation in one dimension. An experiment quantifies the impact on production systems and end users.

Teams under speed pressure sometimes call A/B tests "costly." But shipping without an experiment can be incredibly costly if a major regression goes undetected. The more complex the system, the more important it is to bound the risk.

Close the Loop

Run evals early and often to find the best treatments. Then let the experiment validate that real users and systems respond as predicted. Monitor the metrics you didn't optimize for.

Then: run your LLM evals on the A/B test data itself. Did the version the judge preferred actually perform better with users? This extends the traditional evaluation funnel. LLM judges let us ask not just "did the metric move?" but "did the qualitative aspects change?" When the gap between eval scores and experiment outcomes is large, that's diagnostic gold.

Limitations & Caveats

• Long-running tasks are hard to eval: By construction, long-running behaviors and delayed outcomes are challenging to capture with evals. Don't rely on evals alone for tasks with multi-day feedback loops.
• Calibration drift: As models and user behavior change, the mapping between eval scores and real outcomes can shift. Continuous recalibration is mandatory.
• Evals are opinions, not evidence: Without offline-online signal calibration, your evals are just opinions. Treat them as hypotheses, not facts.

Practical Advice for Your Team

1. Build your eval stack first — before you run any A/B test, have a set of LLM judges that can verify basic quality dimensions.
2. Use evals to narrow the candidate set — test 20 prompt variants with an eval, then take the top 3 into an experiment.
3. Calibrate continuously — after each experiment, compare eval scores with actual outcomes and adjust your judges.
4. Don't skip guardrail metrics — even if the eval and primary metric look good, monitor secondary dimensions you're not optimizing for.

Next Steps

If you're building an evaluation system today, start with a simple judge that measures one dimension (e.g., relevance or coherence). Run it on historical experiment data to see how well it predicts outcomes. Then iterate: add dimensions, adjust weights, and close the loop.

For a deeper look at how AI assistants are reshaping platform interactions, check out Beyond the Chatbot: How Cloudflare's Agent Lee Redefines Platform Interaction. And if you're working with Python, the latest Python Typing in 2025: 86% Adoption & The Challenges That Remain survey offers valuable context on modern development practices.