Red-teaming & QC for RL environments

Your policy will find every bug in your grader. We find them first.

We attack the tasks, graders and harness the way a policy under training would, and, in an independent audit, measure what each environment actually teaches. Vendors run us on every release; labs use us before they train.

Scroll ↓

Labs now buy RL environments the way they used to buy data. Most are never audited.

Every reward comes from a grader. When the grader is wrong, the model learns the grader, not the task.

And passing isn't value. Some tasks are broken, some are trivial, and some teach nothing your model doesn't already know.

ShapLab does red-teaming and QC for RL environments, built by researchers, repeat founders and AI engineers. We find the hacks before your model does, and measure what each environment is worth.

§1Verifiers

Where rewards go wrong.

Test suites, answer checkers, schema validators and LLM judges all make mistakes, and the mistakes aren't random noise. They repeat on the same tasks, so training finds them. The harness around them adds its own.

59%

of the hard SWE-bench Verified tasks OpenAI audited had flawed tests. [1]

28.5%

of SWE-bench Verified tasks can be passed by reward hacking. [2]

41 pts

spread in how often four math verifiers accept correct, equivalent answers. [3]

0.80

correlation between a pre-training grader audit and the false positives training later rewarded. [4]

  1. 01

    False positives

    The grader accepts wrong work: tests that only check visible cases, printed output taken as proof, the first number anywhere taken as the answer. The model learns the bug.

  2. 02

    False negatives

    The grader rejects right work: an equivalent answer in a different form, a test that demands a name the task never mentions. Correct behavior gets punished.

  3. 03

    Leaks

    The answer is reachable: readable answer files, a grader the agent can edit, tasks already in the model's training data.

  4. 04

    Judge bias

    LLM and rubric graders reward length, format and keywords. In one study, rubric judges preferred the hacked outputs; rubric-free judges preferred the original model. [5]

  5. 05

    Harness noise

    Non-deterministic environments, incomplete resets between rollouts and flaky infrastructure give the same work different rewards. The model learns from the noise.

  1. [1] OpenAI, Why we no longer evaluate SWE-bench Verified (2026)
  2. [2] Auditing Reward Hackability in Code RL Training Environments (2026)
  3. [3] Where the Verifier Fails: A Category-Level Audit of Reward Signals in RLVR (2026)
  4. [4] When the Reward Suite Is Leaky (2026)
  5. [5] Reward Hacking in Rubric-Based Reinforcement Learning (2026)
§2Case study

We red-teamed Mercor APEX-Agents 1.1.

A public agentic benchmark of 240 consulting, banking and law tasks, graded by an LLM judge against expert rubrics. We read every rubric, re-graded the judge on Mercor's own published eval traces, and inspected the sandbox. Every exhibit below is quoted from those public files.

240

tasks run in a sandbox with open internet, while every task's answer key is public.

13

rubric criteria, in 8 of 240 tasks, whose penalty clause contradicts the gold answer.

19%

of tasks rest on a single criterion, so one judge call sets the whole reward.

1–2%

of the LLM judge's verdicts are wrong: 400 re-graded by two independent judges, checked by hand.

Exhibit ALeak

The sandbox is online, and the answer key is public.

task.toml, every tasknetwork_mode = "public"
eval trace, world433-js-07URL: https://apps.legislature.ky.gov/law/statutes/chapter.aspx?id=53606 Status: Success, Length: 30104
public release, every tasktests/golden_responses.json · tests/grading_config.json

Agents can reach the open web mid-task, and the gold answers and rubrics for all 240 tasks are published. A policy trained in this sandbox can learn to look answers up instead of working them out.

Exposure confirmed. Not exploited: none of the 480 eval trajectories tried it, and we haven't yet tested a download from inside the sandbox. Fix: an egress allowlist.

Exhibit BGrader

Penalty clauses that contradict the gold answer.

gold answer, world224-sk-task03The Sponsor Equity Value in Year 5 is $28,137 (USD millions). The IRR in Year 5 is 20.4%.
criterionStates Sponsor Equity Value in Year 5 is $28,137
penalty clauseStates Sponsor Equity Value in Year 5 is a value other than $28,517.

A failed criterion costs −1 only if the penalty clause also fires, so the penalty lands on the wrong answers: an agent reporting $28,517 escapes it and every other wrong answer pays. The IRR criterion in the same task has the same error (20.4% vs 20.8%).

Confirmed in 13 criteria across 8 tasks. Correct answers still score in full.

Exhibit CJudge

The judge grades wording as well as substance.

criterion, world421-mc-08States that SLL should use a hybrid delivery approach that includes (1) SMS notification and (2) a URL link to the full privacy notice
agent's deliverableFull notice must be accessible via mobile-friendly URL linked in the SMS.
judge verdictThe criterion is NOT met. … The word "hybrid" does not appear in the document.

Correct work fails for missing a rubric keyword, while a hedged list of seven candidate states passed as naming the right one. Training on this reward pushes a model toward rubric vocabulary and hedging.

Rare, but systematic: about 1–2% of verdicts, measured on 400 criteria from Mercor's own eval traces.

The headline result holds. None of this overturns the benchmark's reported gains: the judge is right on 98% of verdicts. These are the defects a policy finds first, ranked by what they would cost in training.

§3Products

Two products, one method.

For environment vendorsEvery release

Red-team & QC

Adversarial and end-to-end testing of your tasks, graders and harness on every release. Hacks and regressions are caught before an environment ships, and each release comes with a report your buyers can check.

Set up QC →
For labsBefore you buy or train

Independent audit

Everything in red-team & QC, run independently on the environments you're considering, plus their marginal lift: what each set adds on your held-out tasks, with a buy, fix or skip call.

Book an audit →
LayerWhat we checkRed-team & QCAudit
TasksSolvable and well specified; not leaked, contaminated or trivial●●
GradersFalse positives and negatives; judge bias; tampering●●
HarnessIsolation, resets between rollouts, determinism, flaky infrastructure●●
Marginal liftShapley value of each environment set on your held-out tasks—●

Table 1. Coverage by product. Marginal lift needs training runs, so it comes with the independent audit.

What an audit report looks like

Environment setTasksGrader false +/−HackableBroken/trivialLiftVerdict
Repo-level bug fixing1,2002% / 5%3%4%+4.1Buy
Web app clones4014% / 3%22%9%+1.2Fix first
Spreadsheet analysis8001% / 27%1%31%+0.2Skip
Terminal operations50033% / 4%41%6%−0.8Skip

Table 2. Example audit report. Environment sets and numbers are illustrative. Lift is marginal lift, in pass-rate points on held-out tasks.

Independent by design. We don't build or sell environments or datasets, and the method and results are the same whoever pays.

Shipping RL environments, or training on them?

Vendors: catch the hacks before your buyers, or their policies, do, on every release. Labs: know what an environment is worth before it trains your model.