I no longer trust AI screenshots.
That sounds like a small concession. It isn’t. For most of my career, the raw material of safety work was reproducible: a prompt, a model version, a seed, a log. You could hand someone your setup and they could argue with your conclusions. In 2026, the dominant unit of AI safety discourse has become the viral anecdote, and the anecdote is exactly the artifact our systems are best at manufacturing.
TechCrunch made the point sharply this week, reporting that two separate AI safety conversations went viral specifically because they showed how hard it has become to tell AI fact from AI fiction. One involved Andrew Yang, the former presidential candidate. I am deliberately not going to relitigate the details, partly because the details are the contested object, and partly because that is the whole lesson. The story about verification failure cannot itself be casually verified.
Unexpected behavior is a measurement problem before it is a doom problem
The 2026 escalation in safety discussion is grounded in something real: models have shown behaviors their developers did not anticipate. ABC News and CBS News both ran segments on it in mid-September, pulling in outside experts to interpret incidents. That coverage exists because the underlying phenomenon exists.
But from inside an agent architecture, “unexpected behavior” is not one category. It is at least four, and they demand different responses:
- Specification gaps. The system did what we asked and we asked for the wrong thing. Boring, common, fixable, and responsible for a large share of scary-looking transcripts.
- Distribution shift in the use. The model is fine; the scaffolding around it — tool access, memory, retry loops — created a behavior no single component contains. Agent stacks generate emergent behavior at the orchestration layer, not the weights.
- Genuine capability surprise. The model can do something nobody measured for. This is the case worth losing sleep over, and it is the hardest to establish from a screenshot.
- Fabrication. The incident did not happen as described, or happened under conditions that were never disclosed.
Public conversation collapses all four into a single narrative slot labeled “AI did something creepy.” That collapse is not a rhetorical annoyance. It actively misallocates engineering attention. A lab chasing headline-shaped incidents spends its evaluation budget on the wrong layer of the stack.
Coordination is the encouraging signal
The more interesting development got less attention. Chris Lehane, OpenAI’s global policy chief, told reporters on Tuesday, September 15, that OpenAI, Anthropic, and Google had been in talks on AI safety for weeks. Rebecca Bellan reported it for TechCrunch.
I read that as the most substantive item in the whole news cycle. Competing labs talking to each other for weeks means the conversation has moved past press statements and into something that requires sustained attention from people with technical authority. Whatever comes out of it, the fact of ongoing cross-lab dialogue is a precondition for the thing our field actually lacks: shared definitions.
Because that is what is missing. Not concern — concern is abundant, and experts continue to warn about how fast capability is advancing relative to our ability to characterize it. What is missing is a common vocabulary for what counts as an incident, what counts as a reproduction, and what counts as a claim strong enough to act on.
What a usable incident report looks like
If I could ask the labs in those talks for one deliverable, it would not be a policy framework. It would be a disclosure format. Something closer to how security research handles vulnerabilities than how social media handles screenshots. Minimally:
- Model identifier and version, including any routing or fallback behavior
- Full system prompt and tool schema, or an explicit note that they are withheld and why
- Sampling parameters and whether the behavior reproduces across runs
- The layer at which the behavior appeared — weights, use, or human-in-the-loop
- An independent party who attempted reproduction, and what they found
None of that is technically difficult. It is socially difficult, because it slows down a story, and stories move faster than reproductions. An AI security professional interviewed for one of this week’s pieces pushed back on a widely shared safety claim, which is precisely the kind of correction that arrives after the original has finished traveling.
The discourse is now part of the threat model
Here is the part I think agent researchers underrate. We build systems that produce fluent, confident, well-formatted text at scale. We then form our collective understanding of those systems through fluent, confident, well-formatted text. The evaluation channel and the failure mode share a medium.
That is not a reason for cynicism about safety work. It is a reason to treat provenance as a first-class engineering concern rather than a journalism concern. Signed logs, attested runs, reproducible harnesses, and third-party replication are not bureaucracy. They are the only way a claim about model behavior stays meaningful once it leaves the lab.
The conversations have gotten unbelievable in the literal sense. Fixing that is our job, not the press’s.
🕒 Published: