208. Out of 923 occupations in the US Department of Labor’s O*NET taxonomy, that’s how many have even a single task backed by at least five benchmark prompts. The other 715 occupations are, as far as our evaluation infrastructure is concerned, invisible.
That number comes from Mercor’s “Benchmark Map,” which overlaid 29 selected AI benchmarks onto O*NET’s catalog of 923 occupations and 18,797 tasks. The headline finding: roughly 90% of American working hours are spent on tasks with little or no coverage from existing benchmarks. Coverage clusters around high-wage, AI-exposed work — software engineering, legal analysis — while lower-wage roles, physical-world jobs, middle-income knowledge work, and smaller job categories go largely unmeasured.
I want to argue that this is not primarily a story about economics or labor displacement, even though it will be reported that way. It’s a story about the measurement substrate that the entire agent research field stands on, and about what happens to architecture when you optimize against a biased sample.
Evaluation as a selection pressure
Benchmarks are not passive thermometers. They are the loss function of the research community. When a benchmark becomes canonical, labs allocate compute, data collection, and architectural attention toward the distribution it represents. Agent scaffolds get designed around the shape of the tasks we can score.
Consider what the covered 10% actually looks like structurally. Benchmarked tasks tend to be text-in, text-out. They have a bounded context that fits in a prompt. They have a verifiable answer, or at least a gradeable artifact. They’re typically single-session, with no state that persists over weeks. They assume the environment holds still while the model thinks.
Now consider the properties of work that falls outside that 10%. Tasks that span days and require holding commitments in memory. Tasks where the relevant context lives in a shared physical space, a hallway conversation, or a colleague’s head. Tasks where the correct output depends on reading a room. Tasks that involve manipulating objects, or supervising a process that runs in real time, or deciding when to stop and escalate.
Those aren’t just harder versions of benchmarked tasks. They stress different architectural components: persistent memory with selective forgetting, multi-agent coordination under partial observability, interruptibility, calibrated uncertainty that triggers a handoff rather than a confident guess. We have very little systematic evidence about how current agent designs perform on any of it, because the benchmark map has a hole there.
What a 90% blind spot does to our beliefs
The practical consequence is that our aggregate sense of “how capable are agents now” is being computed from a sample that is both small and non-random. Non-random in a specific direction, too: toward the tasks that are easiest to digitize and score, which correlate with the tasks that were already most legible to software.
This produces two symmetric errors. We overestimate general capability by extrapolating from legible wins, and we underestimate the difficulty of the remaining work because we’ve never priced it. Both errors are expensive. The first inflates deployment confidence. The second distorts research agendas, because problems without benchmarks don’t attract graduate students.
There’s a third effect I find more interesting. A benchmark gap is also a feedback gap. Reinforcement learning from verifiable rewards works beautifully where verification is cheap. The uncovered 90% is, almost by definition, the region where verification is expensive — judgment calls, relational work, physical outcomes with slow feedback. If our strongest training signal only applies where scoring is easy, then capability growth itself inherits the same skew. The gap doesn’t close on its own; it compounds.
Where I’d put the effort
Mapping the hole is the useful contribution here. A taxonomy-anchored map turns “benchmarks are narrow” from a complaint into a coordinate system. You can point at a cell and say: nothing here.
For people building agent systems, a few things follow. Treat benchmark scores as evidence about a narrow slice, not a capability summary. Build your own task-level evaluation against the actual work you’re automating, especially if that work sits outside the covered region. Expect the architectural bottleneck to be memory, coordination, and knowing when to defer — not raw reasoning on a well-posed prompt.
For the research community, the uncovered region is where the open problems live. Hard-to-score tasks require different evaluation methods: longitudinal studies, human-in-the-loop rubrics, process evaluation rather than outcome checking. That’s slower, messier work than publishing a new leaderboard. It’s also the only path to knowing whether agent intelligence generalizes beyond the shapes we happened to build tests for.
We’ve been grading ourselves on the tenth of the economy that fits in a prompt. The rest of the exam hasn’t been written yet.
🕒 Published: