I ran the same tool-calling eval 20 times, changing nothing but the random seed, which also reshuffled the order of the scenarios. The pass rate still moved, from 85.0 percent in the worst run to 90.0 percent in the best.

Now put those runs through a release gate that blocks a candidate if its pass rate is 2 points or more below the baseline. Over all 380 ordered pairs of runs, the gate fires 75 times, so about one comparison in five would block a release that changed nothing. Independent runs would give about one in eight. The difference comes from a slow trend that I still cannot explain, and I would not have seen it without running the eval against itself.

The statistics are already worked out. Miller (2024) explains how to handle evals that vary between runs: resample each question, compare models question by question, and size the number of runs to the effect you care about. Wang (2025) separates the noise of sampling the model from the noise of choosing the questions. Bjarnason, Silva, and Monperrus (2026) and Mustahsan et al. (2025) measured how much agent evals move from run to run. In online experimentation this is called an A/A test, and it measures how often a pipeline reports a difference that is not there (Kohavi, Tang, and Xu, 2020).

I wanted to know how often a pass-rate gate on a tool-calling suite fires when nothing has changed, how many runs make it trustworthy, and whether extra runs should go only to the scenarios that flip. I wrote down six predictions before looking at any result. Two of them were wrong, and the gate I recommend at the end is one I had not planned to test.

How I ran it

The model is OLMo 3 7B Instruct, which I ran locally in 8-bit with its default sampling settings (temperature 0.6, top-p 0.95). I drew 200 single-turn tasks at random from the Berkeley Function Calling Leaderboard (Patil et al., 2025). Half give the model one tool, and half give it several to choose from. A small decoder I wrote pulls the function call out of each reply, and BFCL’s own checker grades it. Grading is deterministic, so all the noise comes from the model.

Each run answers all 200 scenarios once, in a new order, in a separate process with its own seed. I ran the unchanged configuration 30 times, using the first 10 as history and the other 20 for measurement. I also ran two real changes 20 times each, alternating with the unchanged runs, for 15,000 replies in all.

Where the noise comes from

Give each scenario its own probability of passing, q, and assume scenarios pass or fail independently within a run. Each run is then a set of coin flips, one per scenario. Scenarios that always pass or always fail add nothing to the noise, and only the ones in between move the pass rate. If v is the average of q(1 – q) over the n scenarios, the difference between two runs has a standard deviation of √(2v/n). Wang (2025, Appendix A) uses the same algebra for whole benchmarks.

In my 20 measured runs, 158 scenarios passed every time, 14 failed every time, and 28 went both ways. That works out to v = 0.024 and a standard deviation of 1.55 points, which puts a 2-point drop only 1.3 standard deviations from zero. When I simulated independent runs from a model fitted to these scenarios, a drop of 2 points or more showed up 13 percent of the time, about one comparison in eight.

Most flips came down to whether the model used a tool at all. In the 28 scenarios that went both ways, 59 percent of the failures were the model declining to call any tool, and 96 percent of all replies without a call started with “I’m sorry, but I can’t.” One scenario asks for next month’s exhibitions at the Museum of Modern Art and offers an exhibition lookup. The model called the lookup correctly in 9 of 20 runs. In the other 11 it declined, with replies like “I’m sorry, but I can’t provide art exhibitions specifically,” and then described the lookup it had just declined to use.

The trend

The real runs varied more than that, with a standard deviation of 1.95 points instead of 1.55. With 20 runs that is borderline, since the variance ratio of 1.6 sits just inside the 95 percent range of 0.47 to 1.73. I looked further anyway, with checks I had not planned, and found a trend. The first ten measured runs averaged 88.35 percent and the last ten averaged 86.85. Once I removed that straight-line trend, the variance ratio fell to 0.97, so the trend accounts for the whole gap between one in eight and one in five.

The trend also made the false alarms lopsided. When the earlier run was the baseline, the 2-point rule fired on 69 of 190 pairs, or 36 percent. When the later run was the baseline, it fired on 6. Averaging the baseline over many runs would not help much, because the candidate’s run keeps all of its own noise and the baseline goes stale as the pass rate trends.

I do not know what caused it. When I reran one of the first runs at the end of the night, it reproduced all 200 replies byte for byte, so a run depends on its seed and order, not on the machine’s state. The time of night does not fit either. The 10 history runs, which went first, averaged 87.75 percent, between the two halves. The runs of the rewritten prompt, which alternated with these, trended the other way. That leaves chance in the seeds, or something I have not found. Whatever the cause, a dashboard of these runs would have shown a decline in a suite where nothing changed.

Two real changes

For comparison, I made two changes of the kind a team makes without much thought. The first rewrote the default system prompt in plainer language. Every sentence changed. I did not test which change mattered, but I would bet on this one. “You may call one or more functions to assist with the user query” grants permission. Its replacement, “Call one or more of them if they help with the user’s request,” asks the model to judge whether a tool helps. The second change added one tool to each scenario’s list, taken at random from another scenario and nearly always unrelated, much as a shared tool registry grows.

Both changes hurt, and both made the model decline more often. The rewritten prompt cost 6.25 points and raised the share of replies without a call from 7 to 14 percent. The extra tool cost 11.0 points and raised that share to 17 percent, although the model never once called it in 4,000 replies. Wording is known to move scores, for prompt formats (Sclar et al., 2024) and for paraphrased instructions (Mizrahi et al., 2024).

Changes this large are easy to detect. In 399 of 400 pairings of an unchanged run with a rewritten-prompt run, the rewritten prompt came out 2 points or more lower, though single pairs put the drop anywhere from 1.5 to 11 points. The hard case is a real drop of 2 or 3 points.

Which gate catches a 2-point drop

I compared three gates that spend the same number of replies on each side.

• The 2-point rule averages each side’s runs and flags a drop of 2 points or more.
• A paired test flags the candidate when the average per-scenario drop is significant in a one-sided t-test at 5 percent. Pairing by scenario keeps the difference between easy and hard scenarios out of the comparison (Miller, 2024, section 4.2).
• Targeted reruns give one run to each of the 179 scenarios that never changed outcome in the 10 history runs and the rest of the budget to the 21 that did, then apply the same paired test. They also get the 2,000 history replies for free.

Measuring how often a gate catches a regression takes regressions of known size, so I simulated them from a model fitted to the 20 unchanged runs. I planted a 2-point regression in four shapes: the flaky scenarios get a little worse, 4 stable scenarios break, 8 stable scenarios start passing only half the time, or every scenario gets slightly worse. Each shape got 4,000 fresh sets of runs. When I tested the simulation beforehand on synthetic data with a known answer, its detection rates for the paired test and targeted reruns came within 3.5 points of the truth. The simulation has no trend, so my later checks also counted false alarms on the real runs.

With one run per side, none of the gates works well. The 2-point rule catches a real 2-point drop a little over half the time, but it also fires on 13 percent of unchanged comparisons in the simulation and 20 percent in the real runs. The paired test keeps false alarms at 5 percent in the simulation (10 percent on the real runs) and catches only 26 to 36 percent of 2-point drops.

More runs help. With two runs per side, the 2-point rule’s false alarms drop to about 5 percent in the simulation and 9 percent on the real runs, close to the paired test, while it still catches about half of the 2-point drops of every shape. That is a coincidence of this suite’s numbers, and a noisier suite would push the same rule back above 5 percent. With four runs per side, the paired test catches 79 percent of drops in the flaky scenarios and 72 percent when everything gets worse, but only 45 percent when 4 stable scenarios break. The trouble is its denominator. The t statistic divides the mean drop by the spread of the per-scenario drops, and a handful of scenarios going from always passing to always failing makes that spread large.

Targeted reruns were a disappointment. I had predicted that they would win when the regression hit flaky scenarios and lose when it hit stable ones. Both predictions were wrong. With two runs per side they beat the paired test in three of the four shapes and tied in the fourth, most clearly when the flaky scenarios got worse (69 percent against 53). But the 2-point rule beat them in the other three shapes. With four runs per side they lost to the paired test in every shape except one, where they tied. The history labels were a large part of the problem. Ten clean runs prove little, because they are consistent with a failure rate as high as 26 percent. Here, 8 of the 28 flaky scenarios had looked stable in the history. With labels from the 20 measured runs instead, a later check that flatters the method, targeted reruns beat the paired test in seven of the eight cases.

A gate I had not planned

The paired test’s weakness pointed to another gate. If the scenarios themselves are the suite, all the noise in a comparison comes from rerunning them, and an A/A run measures that noise directly. So you can set the threshold from the A/A run, at 1.645 standard deviations of a no-change difference, and flag any drop larger than that. Because the threshold is fixed before the candidate runs, a few broken scenarios cannot inflate it. I tried two versions after the fact, with the noise from the formula and from the measured spread. The table comes from a fresh simulation run, so the paired test’s numbers differ from those above by up to 2 points.

The formula threshold caught about four in five regressions of every shape, including the broken scenarios that the paired test mostly missed, but on the real runs the trend doubled its false alarms. The measured-spread threshold kept false alarms near 5 percent on the real runs, which is no surprise since it was set from them, and gave up about ten points of detection in exchange.

Two common suggestions

The first suggestion is to raise an alarm whenever a scenario that always passed starts failing, as classic CI does. Of the 165 scenarios that passed all 10 history runs, at least one failed in 14 of the 20 unchanged runs. Google reported a failure of a known flaky test only after it failed 3 times in a row (Micco, 2016), but that helps only slowly here. With 2, 3, or 4 failures in a row required, the alarm still fired in 6 of 19, 3 of 18, and 1 of 17 windows. Research on flaky tests mostly treats them as defects to fix, though it also covers machine-learning tests that are flaky because the algorithm is random (Parry et al., 2021, section 3.3.2), which is the situation here.

The second is to set the temperature to 0. On a single machine, greedy decoding removes most of the randomness. Two identical greedy runs matched reply for reply, while changing the batch size or the order changed up to 6 of the 200 replies, as Thinking Machines’ account would suggest (He, 2025). No verdict changed. With a hosted model you do not control the batch, though, and accuracy at temperature 0 has varied by up to 15 points between runs (Atil et al., 2024). Greedy decoding also hides the 28 scenarios that go both ways, which your users will meet if your product samples. The chance that a scenario passes every one of k attempts, which τ-bench calls pass^k (Yao et al., 2024), falls here from 87.6 percent for one attempt to 80.6 percent for eight.

What to do before you trust a gate

Before you put a gate on a tool-calling eval, run the eval against itself.

1. Run the unchanged configuration about ten times.
2. Estimate the noise of a one-run comparison two ways, with the formula √(2v/n) and as √2 times the standard deviation of the runs’ pass rates, and call the larger one s.
3. To catch a drop of δ points 80 percent of the time, use k ≥ (2.5 s/δ)² runs per side (2.5 is 1.645 + 0.84), and flag only drops larger than 1.645 s/√k.
4. Plot the A/A pass rates in the order you ran them. If they trend, any threshold will fire mostly in one direction.

For my suite, s was 1.95 points. Catching a 2-point drop would take 6 runs per side, and the threshold would be 3.2 points with one run per side, 2.3 with two, and 1.6 with four. Even so, the one-run threshold fired on 18 of the 190 pairs where the earlier run was the baseline, and on none the other way around. Adding scenarios shrinks the formula’s part of the noise as fast as adding runs does, if they are as flaky, and Mustahsan et al. (2025) discuss that trade-off. Extra run-to-run spread does not shrink that way. And when a gate fires, look at which scenarios moved before you decide anything.

"""A/A check for a pass/fail eval suite. Python 3.8+, standard library only.
Input: a CSV with the columns scenario,run,passed (passed is 0 or 1) from repeated runs of the unchanged
configuration, ten if you can afford them. A second CSV in the same format, from the candidate, adds a comparison.
    python3 aa_check.py baseline.csv
    python3 aa_check.py baseline.csv candidate.csv
"""
import csv
import math
import statistics
import sys
from collections import defaultdict
Z_ALARM, Z_POWER = 1.645, 0.842          # one-sided 5% false alarms, 80% detection
def load(path):
    by_scenario, by_run = defaultdict(list), defaultdict(list)
    with open(path, newline="") as f:
        for row in csv.DictReader(f):
            by_scenario[row["scenario"]].append(int(row["passed"]))
            by_run[row["run"]].append(int(row["passed"]))
    return by_scenario, [statistics.fmean(r) for r in by_run.values()]
def main():
    base, rates = load(sys.argv[1])
    n = len(base)
    assert all(len(r) >= 2 for r in base.values()), "every scenario needs at least two runs"
    # v: the average over scenarios of q(1 - q), unbiased for the number of runs of each scenario
    v = statistics.fmean(len(r) / (len(r) - 1) * statistics.fmean(r) * (1 - statistics.fmean(r)) for r in base.values())
    s_formula = math.sqrt(2 * v / n)                                        # one-run vs one-run, independent runs
    s_runs = math.sqrt(2) * statistics.stdev(rates) if len(rates) >= 3 else 0.0   # the same, measured
    s = max(s_formula, s_runs)
    mixed = sum(0 < sum(r) < len(r) for r in base.values())
    print(f"{n} scenarios, {len(rates)} runs, {mixed} scenarios with both passes and failures, v = {v:.4f}")
    print(f"noise of a one-run comparison: formula {100 * s_formula:.2f} points, measured {100 * s_runs:.2f}"
          + ("" if len(rates) >= 3 else " (needs 3 or more runs)") + f", using s = {100 * s:.2f}")
    if s == 0:
        print("no noise in these runs: any drop is a change")
    for k in (1, 2, 4, 8) if s else ():
        print(f"  {k} run(s) per side: flag only drops larger than {100 * Z_ALARM * s / math.sqrt(k):.2f} points")
    for d in (0.02, 0.03, 0.05) if s else ():
        k = math.ceil(((Z_ALARM + Z_POWER) * s / d) ** 2)
        print(f"  to catch a {100 * d:.0f}-point drop 80% of the time: {k} run(s) per side")
    if s and len(rates) >= 3:      # runs in file order: does the one-run threshold fire mostly in one direction?
        t = Z_ALARM * s
        early = sum(rates[j] - rates[i] < -t for i in range(len(rates)) for j in range(i + 1, len(rates)))
        late = sum(rates[i] - rates[j] < -t for i in range(len(rates)) for j in range(i + 1, len(rates)))
        m = len(rates) * (len(rates) - 1) // 2
        print(f"  the one-run threshold fired on {early} of {m} pairs with the earlier run as baseline and on {late} "
              f"the other way (runs in file order). Lopsided counts mean a trend: plot the pass rates in run order.")
    if len(sys.argv) > 2:
        cand, crates = load(sys.argv[2])
        common = sorted(set(base) & set(cand))
        drop = {x: statistics.fmean(cand[x]) - statistics.fmean(base[x]) for x in common}
        diff = statistics.fmean(drop.values())
        threshold = Z_ALARM * s / math.sqrt(2) * math.sqrt(1 / len(rates) + 1 / len(crates))
        verdict = "regression" if diff < -threshold else "no regression shown"
        print(f"candidate ({len(crates)} runs) minus baseline ({len(rates)} runs), {len(common)} scenarios: "
              f"{100 * diff:+.2f} points against a threshold of {100 * threshold:.2f}: {verdict}")
        print("largest drops:", ", ".join(f"{x} ({100 * drop[x]:+.0f})" for x in sorted(common, key=drop.get)[:8]))
if __name__ == "__main__":
    main()

Where this is wrong

• One model and one kind of task. Everything here comes from one 7B model on single-turn tasks with a single call. Agents that loop through several tool calls may be noisier. On SWE-bench, single-run scores of coding agents varied by 2.2 to 6.0 points (Bjarnason, Silva, and Monperrus, 2026).
• The suite as a fixed set. I treat the 200 scenarios as the whole suite. Wang (2025, section 3.1 of the third version) argues that most evals must also count the noise of choosing the questions, unless the questions are the point, as in a driving test. A regression suite is such a case, I think. If the runs are interchangeable, which the trend puts in doubt, the 95 percent intervals are 5.5 to 7.0 points for the rewritten prompt and 10.2 to 11.75 for the extra tool. Treated as a stratified sample of similar scenarios, the prompt’s interval widens to 3.8 to 8.9.
• Grading choices. My decoder reads true, false, and null as JSON values, as OLMo 3 writes them. Read literally, as BFCL’s parser would, they fail, the suite looks noisier (85.2 percent, v = 0.034), and the 2-point rule at two runs per side fires 6.5 percent of the time even in the simulation. The six predictions come out the same.
• The trend. Permutation tests I added afterward put its slope at p = 0.002 for the 20 measured runs and p = 0.056 for all 30 unchanged runs. The rewritten prompt trended the other way (p = 0.008), and the extra tool showed no trend (p = 0.64). The rewritten prompt’s measured cost moved with them, from 7.75 points over the first ten pairs of runs to 4.75 over the last ten.

In the terms of the Quality Locus, a regression gate is part of the evaluation surface, the part of a production AI system that decides what it can detect, compare, reject, ship, and improve. An eval that cannot tell a 2-point regression from a new seed can still compare models ten points apart, but it is not ready to block a release over a 2-point drop.

Section 11.5 of Beyond the Model, “Testing the Non-Deterministic,” covers why a test of a system whose outputs vary has to pass on a distribution rather than on a single run.

References

Atil, B., et al. “Non-Determinism of ‘Deterministic’ LLM Settings.” arXiv:2408.04667, 2024 (revised 2025).

Bjarnason, B. H., Silva, A., and Monperrus, M. “On Randomness in Agentic Evals.” arXiv:2602.07150, 2026.

He, H., and Thinking Machines Lab. “Defeating Nondeterminism in LLM Inference.” Thinking Machines Lab: Connectionism, September 2025.

Kohavi, R., Tang, D., and Xu, Y. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. Chapter 19, “The A/A Test,” 200-208.

Micco, J. “Flaky Tests at Google and How We Mitigate Them.” Google Testing Blog, May 27, 2016.

Miller, E. “Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations.” arXiv:2411.00640, 2024.

Mizrahi, M., Kaplan, G., Malkin, D., Dror, R., Shahaf, D., and Stanovsky, G. “State of What Art? A Call for Multi-Prompt LLM Evaluation.” Transactions of the Association for Computational Linguistics 12, 933-949, 2024.

Mustahsan, Z., et al. “Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation.” arXiv:2512.06710, 2025.

Parry, O., Kapfhammer, G. M., Hilton, M., and McMinn, P. “A Survey of Flaky Tests.” ACM Transactions on Software Engineering and Methodology 31(1), Article 17, 2021.

Patil, S. G., Mao, H., Yan, F., Ji, C. C.-J., Suresh, V., Stoica, I., and Gonzalez, J. E. “The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models.” ICML 2025, PMLR 267, 48371-48392.

Sclar, M., Choi, Y., Tsvetkov, Y., and Suhr, A. “Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design or: How I Learned to Start Worrying about Prompt Formatting.” ICLR 2024.

Wang, S. I. “Measuring all the noises of LLM Evals.” arXiv:2512.21326v3, 2025 (revised 2026).

Yao, S., Shinn, N., Razavi, P., and Narasimhan, K. “τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains.” arXiv:2406.12045, 2024. ICLR 2025.