Which perturbations hurt tool-use agents the most?

In RobustBench-TC, observation perturbations (like typos) reduce accuracy by less than 5%, but reward-relevant and transition perturbations reduce accuracy by roughly 40% and 30% respectively. Model scale alone does not close these gaps.

How do you make tool-use agents more robust?

One recipe is domain-randomized reinforcement learning: train the agent on perturbation-augmented trajectories spanning the statically encodable POMDP components, so it learns to handle noisy, ambiguous, and unreliable tool environments rather than only clean ones.

← Writing

How Do You Benchmark Tool-Use Agents for the Sim-to-Real Gap?

Q: What is the sim-to-real gap for tool-use agents?

It is the drop in performance between clean benchmarks — which assume perfect inputs, unambiguous tool registries, and reliable APIs — and real deployments, where users make typos, requests time out, and duplicate tool names collide. Framed as a partially observable Markov decision process (POMDP), these are perturbations to the observation, transition, and reward components.

By Aojie (Justin) Yuan · August 2026 · ~6 min read

TL;DR — Tool-use agents are graded on clean benchmarks but deployed into typos, timeouts, and duplicate tool names. That mismatch is a sim-to-real gap. Modeled as a POMDP, real-world noise perturbs the observation, transition, and reward components. RobustBench-TC measures it with 22 perturbation types: observation noise costs <5% accuracy, but reward- and transition-level noise cost ~40% and ~30% — and scaling the model does not fix it. A domain-randomized RL recipe trains on perturbed trajectories to close the gap.

If your agent scores 90% on a benchmark and then falls over in production, the benchmark was lying — not on purpose, but by assumption. Most tool-use evaluations assume clean inputs, an unambiguous tool registry, and reliable APIs. Real deployments violate all three.

What is the sim-to-real gap here?

Borrow the framing from robotics: the "simulator" is the clean benchmark; the "real world" is messy deployment. For a tool-use agent, the environment is a partially observable Markov decision process (POMDP), and real-world failures map onto its components:

Observation — the agent misperceives the request: user typos, paraphrased goals, noisy tool descriptions.
Transition — the world behaves unexpectedly: duplicate tool names across servers, misconfigured timeouts, redundant tools.
Reward — what counts as success shifts: distractor tools that look right but aren't.

How do you measure it?

You perturb each component deliberately and see what breaks. RobustBench-TC organizes 22 perturbation types by the four POMDP components, each grounded in a verified GitHub issue or a documented tool-calling failure — so the noise is realistic, not synthetic guesswork.

Run across 21 models (1.5B to 32B parameters, including closed-source), the pattern is sharp:

Observation perturbations — accuracy drops < 5%. Agents are surprisingly robust to messy inputs.
Reward-relevant perturbations — accuracy drops ~40%.
Transition perturbations — accuracy drops ~30%.

Scale alone does not close the gap.

Bigger models help a little, but they do not rescue an agent from a world where the tools themselves are ambiguous or unreliable. Robustness is a training-distribution problem, not a parameter-count problem.

How do you close it?

If the failure is distributional, fix the distribution. ToolRL-DR is a domain-randomization reinforcement-learning recipe: train the agent on perturbation-augmented trajectories that span the statically encodable POMDP components. The agent stops assuming a clean world and learns to operate in a noisy one — exactly the shift domain randomization brought to sim-to-real robotics.

Why it matters

A benchmark number is only trustworthy if it predicts deployment behavior. Measuring the sim-to-real gap — and training against it — is part of making agents you can actually trust and audit in the wild.

Frequently asked questions

What is the sim-to-real gap for tool-use agents?

The performance drop between clean benchmarks and real deployments, where typos, timeouts, and duplicate tool names perturb the observation, transition, and reward parts of the tool-use POMDP.

Which perturbations hurt most?

Observation noise costs <5% accuracy; reward- and transition-level noise cost ~40% and ~30%. Scale alone does not close the gap.

How do you improve robustness?

Domain-randomized RL: train on perturbation-augmented trajectories across POMDP components so the agent handles noisy, ambiguous tool environments.

Written by Aojie (Justin) Yuan. "When Simulation Lies" appears at NeurIPS 2026 (arXiv:2605.11928).