Excerpt: A new startup, OpenEvidence, believes that AI can solve this problem. They’ve developed an AI tool that can scan medical literature and quickly summarize key themes. For example, let’s say you ask the OpenEvidence AI a question like “How do I diagnose pancreatitis?” The tool would respond by listing out diagnostic criteria and the blood tests, imaging tests, physical evaluations, and patient questions for you to consider.

It’s easy to see that OpenEvidence wants to serve as a “co-pilot” for doctors. The tool has already been used by over 250,000 doctors in the United States and the company recently reached a $1 billion valuation. If you’re thinking about using OpenEvidence (or even if you’ve already used it), you might be wondering whether or not the information it presents is accurate. That’s an important question to ask because AI has been known to generate fake data and then present it as factual (researchers call this phenomenon “AI hallucination”).

So, is OpenEvidence reliable and trustworthy? The answer is: sometimes. When OpenEvidence took the US Medical Licensing Exam recently, it was wrong 9% of the time. While this performance was better than other AI tools (like ChatGPT), it still shows that OpenEvidence can make mistakes. If you rely on OpenEvidence to make clinical decisions, you could be giving patients misinformation which would create legal liability for you and your clinic.

To understand the risks, let’s take a look at a specific example where OpenEvidence recommends a treatment that would actually be harmful to patients. You may have heard about a complex neurological condition called Myalgic Encephalomyelitis (also known as “Chronic Fatigue Syndrome” or “ME/CFS”). This condition most commonly occurs after a viral infection — like mononucleosis or Covid — and the debilitating symptoms can last for years. It is estimated that millions of Americans have ME/CFS.

When you ask OpenEvidence “What is Chronic Fatigue Syndrome?” you get back a response that talks about symptoms, diagnosis, and potential biomarkers. You also get a short paragraph that describes suggested treatments for this condition:

“Treatment for CFS is primarily supportive and symptom-based. Cognitive behavioral therapy (CBT) and graded exercise therapy (GET) have shown moderate efficacy in improving fatigue levels, functional capacity, and quality of life. Pharmacologic treatments have not been consistently effective, and no specific medication is approved for CFS.”

OpenEvidence recommends exercise and therapy for ME/CFS because these treatments were mentioned in a medical guidebook that was published 13 years ago. Unfortunately, these treatments are no longer recommended because the research study supporting them was thoroughly debunked. In fact, the NIH published new guidance (in 2022) to point out the risks associated with these treatments:

“The British National Institute for Health and Care Excellence (NICE) recently published its updated guidelines for myalgic encephalomyelitis/chronic fatigue syndrome (ME/CFS). NICE concluded, after an extensive review of the literature, that graded exercise therapy (GET) is harmful and should not be used, and that cognitive behavioural therapy (CBT) is only an adjunctive and not a curative treatment.”

In other words, telling a patient with ME/CFS to start exercising is like telling a patient with lung disease to start smoking cigarettes. If a doctor followed OpenEvidence’s advice and recommended graded exercise therapy (GET) to a patient with ME/CFS, they would be harming that patient and opening the clinic up to legal liability. For this reason, doctors have to be extremely cautious when using OpenEvidence.

To be fair, this isn’t really the AI’s fault. It’s the company’s fault. The outputs from an AI can only be as good as the inputs. In this scenario, OpenEvidence has built a recommendation engine that is using outdated information. So the AI is doing an excellent job of summarizing the available data, but the underlying information is factually incorrect. If OpenEvidence is going to be successful, they need to make more of an effort to purge outdated sources from their database so that the AI will stop misleading doctors and harming patients.