Heidi Health CEO: AI medical tools degrade under load, regulators missing the risk
Heidi Health CEO Dr. Tom Kelly told SmartCompany that healthcare AI regulation is missing a critical risk: models do not perform consistently under real-world pressure. "When we measure and monitor the performance of Heidi on days where it is particularly busy, like a Monday morning, you see meaningful degradations in performance of the notes, which can be a safety issue," Kelly said at Blackbird Ventures' Sunrise event. Heidi is a Melbourne-based AI medical scribe that automates clinical documentation during patient consultations. The company processes over 1 million patient consultations every week globally. They claim only 1 in 1,000 notes receive negative quality ratings, but Kelly's comments suggest that average hides variability. The issue: regulators treat AI models as repeatable systems. They are not. Performance fluctuates based on load, deployment environment, and demand spikes. For a clinical tool, that variability creates patient safety risk. Kelly says regulators are not keeping pace. "The area I think that is being chronically missed is they are treating models as repeatable. In practice, it is not true." ## What this means for healthcare AI sales If you are selling AI tools to hospital systems, this is the objection sitting across the table: what happens when your model degrades during peak hours? Clinical buyers care about consistency, not average performance. A tool that works 99% of the time but fails during Monday morning rushes is a liability. Heidi previously faced security scrutiny when a Mindgard audit exposed prompt-injection vulnerabilities that could bypass guardrails. The company has since moved beyond pure documentation into broader clinical workflows, raising the stakes for reliability. For sales teams: hospital procurement cycles are long because the risk of failure is high. Comp transparency, performance guarantees under load, and security audit results are table stakes. If your product degrades under real-world conditions, expect clinical buyers to walk. Heidi raised $98 million in October 2024 at a $704 million valuation. The company is scaling globally, but Kelly's warning suggests the regulatory gap could slow adoption if safety concerns escalate.