What if the most dangerous thing an AI system can do is answer confidently and correctly formatted? Three hikers were rescued from California’s Mount Shasta this week after planning their expedition with Google’s Gemini. The model told them how much food and water to bring. The amount was too low. The Siskiyou County Sheriff’s Office followed up with a public warning: don’t plan trips with AI alone, use local authorities and the Forest Service.
The easy read is “AI hallucinated, people got hurt.” I think that read misses what actually broke. Nothing here suggests the model produced gibberish. It produced advice that looked exactly like advice. That’s the failure mode worth studying.
Plausible is not the same as calibrated
A language model asked “how much water should we bring for a Mount Shasta climb” is doing distributional recall. It has read trail guides, forum posts, gear lists, and blog recaps. It returns something near the center of that distribution. The center of a distribution of hiking advice is advice written for people who are already competent hikers on a normal day in normal conditions.
The three people asking were, per authorities, novices. They went off course. They spent a night in a steep canyon. None of that is in the distribution. The model wasn’t wrong about the average case; it was answering a different question than the one that mattered, which was “how much water should we, specifically, bring given what could go wrong.”
That gap is a calibration problem, not a knowledge problem. And calibration is exactly what current assistant architectures are weakest at, because the training signal rewards helpful-sounding specificity. A hedge-heavy answer that says “I can’t estimate this reliably without knowing your pace, acclimatization, and the current snow line” reads worse to a human rater than a clean number. We optimized for the wrong thing and then deployed it as a planning agent.
Advice systems need asymmetric loss
Here is the design point I keep coming back to. In most model evaluation, error is symmetric. Off by 20% high, off by 20% low, same penalty. Real-world planning does not work that way.
- Too much water: heavy pack, sore shoulders, mild annoyance.
- Too little water on an exposed volcanic slope: a rescue operation.
Any system giving consequence-bearing advice needs an explicit asymmetric loss baked into its output policy, not just its training objective. That means the recommendation should be deliberately biased toward the survivable direction of error whenever the downside is not symmetric. This is not a hard thing to implement. It is a thing almost nobody implements, because it makes outputs look less precise and precision is what users reward.
The missing grounding layer
The sheriff’s office answer was essentially a systems-architecture recommendation: talk to the Forest Service. Translated into agent terms, that’s retrieval against an authoritative, current, jurisdiction-specific source. Trail conditions, snow levels, closures, and water availability change weekly. Parametric memory in a model cannot track that. It’s frozen at training time and blended across years.
So the architecture you actually want for this task has parts that a general chatbot doesn’t have by default:
- A live source of truth. Ranger district conditions, permit data, current weather. Retrieval, not recall.
- User-state elicitation. Experience level, fitness, acclimatization, gear. The model should be asking, not assuming.
- Refusal and handoff behavior. Some questions should terminate in “contact this office,” and a well-designed agent treats that as a successful outcome rather than a failure to be helpful.
- Uncertainty made legible. A range with stated assumptions, not a number.
None of that is exotic. All of it costs something in perceived fluency, which is why it keeps losing product arguments.
Why this generalizes
Mount Shasta is a clean case study because the feedback loop closed fast and visibly. Someone got stranded, someone got rescued, a sheriff wrote a Facebook post. Most agent failures of this shape never close the loop at all. Medication timing, contract terms, dosage of anything, structural load, financial exposure, medical triage. The model answers, the user acts, and the error surfaces later or never gets attributed.
What makes the hiking case instructive is that the model’s answer was probably defensible in the abstract and still nearly got people killed in the particular. Agent intelligence, if the term means anything, has to include knowing when the particular case falls outside what the general case can cover. Right now we mostly build systems that cannot tell the difference, and then we call the difference a hallucination when it shows up in a news story.
The three hikers got down. The interesting question is what a system designed to know its own limits would have said instead.
đź•’ Published: