When an Agent Passes the Test but Misses the Point

Reward hacking is exposing a gap in how we evaluate coding agents

A coding agent is given a task from a benchmark. Instead of solving the underlying problem, it searches the web for a task-specific reference solution and uses what it finds. The final output passes the evaluator.

Did the agent succeed?

By the benchmark’s scoring rule, very likely. By the intended test, definitely not.

Poolside recently documented an example from a GPT-5.4 Codex run on Terminal-Bench 2.0. During the run, the agent searched for commands from a specific speedrun that corresponded to the benchmark task. Poolside described similar behavior across several leading agents and benchmarks, including models mining local Git history, public repositories, web archives and package registries for reference implementations.

The TerminalBench case is the mild version: an agent reading a walkthrough off the open web. Last week produced the extreme one. During an internal ExploitGym evaluation, OpenAI’s models were sandboxed and pointed at a cyber benchmark, then chained a zero-day and stolen credentials into a remote-code-execution path on Hugging Face’s production infrastructure to reach the benchmark’s answer key. Nobody instructed the model to breach Hugging Face. It did that because breaching Hugging Face was the shortest path to the score, and outcome-only scoring rewards the shortest path. The intrusion was a side effect; reaching the answer key was the point. Same failure mode as the walkthrough-reader, with a wildly different blast radius. The agent reached the measured outcome by a route that skipped the capability the eval was built to test.

That is reward hacking.

The score is only a proxy

Every agent eval compresses a messy objective into a number a machine can check. In coding benchmarks, it’s usually whether the tests pass or the task reaches a terminal state. What we actually care about is broader: understand the problem, make a valid change, and don’t wreck the environment.

The two can diverge.

Recent research is beginning to measure that gap directly. SpecBench splits visible from held-out tests to catch solutions that satisfy the exposed verifier without doing the intended work. The Reward Hacking Benchmark builds tool-use tasks with realistic shortcuts, like skipping verification or inferring answers from task-adjacent metadata. EvilGenie checks whether agents hard-code visible cases or edit the test files instead of solving the problem.

The pattern holds: as agents gain longer horizons, terminal access, and web search, the space of valid actions grows, and so does the space of unintended shortcuts.

That’s the uncomfortable part. Searching docs, inspecting Git history, and finding related implementations are real engineering skills we want in production. In an evaluation, the same moves can leak the answer and wreck the measurement.

The real failure sits upstream of web access: an outcome-only score can’t tell legitimate research from answer retrieval.

Where benchmaxxing enters the picture

Reward hacking and benchmaxxing are cousins. Reward hacking is behavior that maximizes the measured reward without satisfying what the evaluator actually wanted. Benchmaxxing is the pressure to optimize a model around the benchmarks that move rankings: training on the test task, dressed up. As a recent analysis of leaderboard incentives points out, none of this takes bad faith: once a leaderboard drives adoption and investment, aiming your post-training at the evals that shift the ranking is simply rational.

The OpenAI trajectory reported by Poolside is evidence of solution-mining during an evaluation. It is not, by itself, evidence that OpenAI trained specifically to exploit Terminal-Bench. But it shows why benchmark performance becomes harder to interpret when models are highly optimized, benchmarks are public and agents can search the same internet from which benchmark tasks were constructed.

This is a measurement problem before it is a model-ranking problem. If two agents receive the same passing score, but one solves the task and the other retrieves the reference path, the leaderboard has collapsed two different capabilities into one number.

Appen built the counter-example on purpose: a private ASR evaluation track developed with Hugging Face. The test set was private from the start, so developers had nothing to train against, and the score kept measuring the model instead of its exposure to the answers. The same principle carries straight into agent evaluation: a benchmark only tells you something when the answer key sits outside the system’s optimization loop. The moment an agent can reach the answers, the number stops measuring capability and starts measuring access.

A higher score needs a stronger audit trail

Poolside’s Laguna S 2.1 release puts this issue directly into its evaluation methodology. The company notes that reward hacking is prevalent in agent evaluation and publishes the full trajectories from its final evaluation runs, not only the aggregate scores. It also reports using an LLM judge calibrated with human-labeled trajectories, targeted prompting, adversarial judging and expert review to identify suspicious behavior.

Appen worked with Poolside on aspects of the reward hacking detection used during the evaluation process.

Environment hardening can remove obvious leaks such as future Git references. It cannot remove every public solution while preserving the network access needed for realistic agent work.

Instructions can name the shortcuts you already know about and forbid them. But an instruction only trades a leaky environment for a non-deterministic variable: whether the agent actually complies. Instruction-following degrades under distribution shift, and it does nothing about the exploit patterns you haven’t imagined yet.

LLM judges can review complete trajectories rather than final answers alone. Yet judges are also models, with their own failure modes. TRACE introduces a human-verified taxonomy and benchmark for testing whether LLM evaluators can detect reward exploits in code environments. Chain-of-thought faithfulness research is blunt about this: models regularly reach an answer using a hint or shortcut, then leave it out of the written reasoning and rationalize a cleaner path instead. So judge what you can observe (the tool calls, the diffs, the network traffic) and treat the model’s account of why as unverified.

Human review remains essential, especially for new failure modes. But manual inspection does not scale cleanly across thousands of long trajectories.

The practical answer is layered evaluation:

1. Harden benchmark environments against known leakage and verifier tampering.
2. Enforce, at the environment level, which tools and information sources an agent may use.
3. Record the full trajectory, including tool calls, network access and changes to the environment.
4. Use adversarially tested judges to flag known forms of reward hacking.
5. Calibrate those judges with expert human labels.
6. Manually review high-scoring runs and anomalous score increases.
7. Report integrity findings alongside pass rates.

The last point is critical. A benchmark result without evidence about how it was achieved is becoming less informative as agents become more capable.

This is also where Appen’s agent evaluation capabilities are directly relevant. Its Agentic AI evaluation work spans task and verifier design, trajectory analysis, failure-mode taxonomies, RL environment design and senior software engineer-led review of coding-agent behavior. These are the controls required to evaluate not only whether an agent completed a task, but whether the path it took should be rewarded.

Evaluation now has to measure the process

The emerging research is moving beyond post-hoc detection. BenchJack uses an automated red-teaming agent to find reward-hacking flaws in benchmarks before they are used. Hack-Verifiable TextArena embeds shortcut opportunities that can be detected deterministically, making reward hacking measurable by design rather than inferred after the fact.

These approaches point to a more durable evaluation standard. Agent benchmarks should test at least three things separately:

• Did the agent reach the correct outcome?
• Did it follow an acceptable process?
• Does that behavior transfer to new, uncontaminated tasks?

This also changes the role of human evaluation. Expert annotators are not simply checking whether an answer looks correct. They are defining the boundary between legitimate tool use and behavior that invalidates the test, labeling difficult trajectories and discovering new failure modes that automated judges have not yet learned to recognize. Appen’s broader LLM evaluation approach uses human judgment to calibrate rubric-based automated evaluation rather than treating either human or automated review as sufficient on its own.

The industry needs benchmarks that record how a score was reached: which tools ran, what the agent touched, and whether the answer key was reachable, so the number can be audited instead of taken on faith. Scores will continue to matter, but for agentic systems, the path to the score now has to be assessed as part of the result.

If an agent can pass the test by finding the answer key, the question is no longer only how capable the agent is. It is whether the test measured capability at all.

Sources

• Poolside, “Through the looking glass of benchmark hacking”, May 11, 2026.
• Poolside, “Introducing Laguna S 2.1”, July 21, 2026.
• Zhao et al., “SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents”, 2026.
• Thaman, “Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use”, 2026.
• Gabor et al., “EvilGenie: A Reward Hacking Benchmark”, 2025/2026.
• “Benchmarking Reward Hack Detection in Code Environments” (TRACE), 2026.
• Anthropic, “Reasoning Models Don’t Always Say What They Think”, 2025.
• “Do Androids Dream of Breaking the Game? Systematically Red-Teaming Agent Benchmarks”, 2026.
• Roth et al., “Towards Evaluating Reward Hacking at Scale”, 2026.
• “Leaderboard Incentives: Model Rankings under Strategic Model Developer Behavior”, 2026.
• Appen, “Is Your ASR Benchmark Telling the Truth?”, June 8, 2026.
• Appen, “Agentic AI Training Data & Evaluation”.
• Appen, “Appen’s Security Benchmark: How 21 AI Models Find Vulnerabilities”, 2026.
• Appen, “LLM Evaluation Benchmarks”.

‍