Quality
Quality is the product.
A model learns exactly what its data rewards. One careless task or a grader that can be gamed teaches the wrong lesson at scale. Every step below exists so that the data we provide helps models get better at the work, and a high score only ever means the work was done well.
01
Select the experts
Specialists apply, complete an interview with a live work sample from their field, and do paid trial tasks. Only the strongest are invited, and everyone keeps being ranked on the quality of their work.
02
Map the real workflow
Experts describe how the job is actually done: the tools, the data, the decisions, what good looks like, and where people go wrong.
03
Build the world
We build the software and synthetic data from that map. The same experts walk through it and flag anything that wouldn't happen at a real company.
04
Write the tasks
Experts write and own every task, and multiple experts review each one before it moves on. Catching a flaw here costs minutes. Missing it costs a lab the compute spent training on it.
05
Refine the rubrics
Rubrics are tested on real model outputs and revised until independent experts grade the same answers the same way.
06
Attack the grader
We try to score well without doing the work. Every hole we find is closed, with a test that proves it stays closed.
07
Measure on frontier models
Frontier models and human experts take every task. Their results set the difficulty, and a held-out split is kept back.
08
Review before release
A final quality review of every task, rubric, and report before anything reaches a customer.
Grading
A high reward means the work was done.
We assume a model will look for the cheapest path to a high score, and we design every grader so that path does not exist.
End-state grading
The final state of the software is compared to an expert solution on the same starting data. What the agent says it did does not count.
Side-effect checks
Any change outside the task's scope fails the episode, even when the target change is correct.
Checked rubrics
Rubrics cover judgment only, and the model that applies them is measured against expert graders.
Sealed graders
Solutions and rubrics run separately from the agent, which only ever sees the software and its tools.
Adversarial release
A task ships only when doing nothing, refusing, guessing, and reading everything all score zero.
Held-out splits
Every environment keeps tasks that are never released, so results can be checked on work no model has seen.
Difficulty
Hard enough to teach.
Tasks a model always or never solves teach it nothing. Every task is placed by measured frontier pass rates.
| Level | Shape | Target frontier pass rate |
|---|---|---|
| Easy | Single records, a few actions | 60 to 90% |
| Medium | Work across records, business rules, and permissions | 25 to 60% |
| Hard | Long work with judgment, conflicting sources, and policy traps | 1 to 25% |
Judge it on your own models.
Every sample comes with its grader audit and frontier model baselines.