arXiv.org / 2024

Ensuring Safety and Trust: Analyzing the Risks of Large Language Models in Medicine

Yifan Yang, Qiao Jin, Robert Leaman, Xiaoyu Liu, Guangzhi Xiong, Maame Sarfo-Gyamfi, Chang Gong, Santiago Ferriere-Steinert, W. Wilbur, Xiaojun Li, Jiaxin Yuan, Bang An

AI SafetyFoundation ModelsGenerative AILarge Language Models

The remarkable capabilities of Large Language Models (LLMs) make them increasingly compelling for adoption in real-world healthcare applications. However, the risks associated with using LLMs in medical applications have not been systematically characterized. We propose using five key principles for safe and trustworthy medical AI – Truthfulness, Resilience, Fairness, Robustness, and Privacy – along with ten specific aspects. Under this comprehensive framework, we introduce a novel MedGuard benchmark with 1,000 expert-verified questions. Our evaluation of 11 commonly used LLMs shows that the current language models, regardless of their safety alignment mechanisms, generally perform poorly on most of our benchmarks, particularly when compared to the high performance of human physicians. Despite recent reports indicate that advanced LLMs like ChatGPT can match or even exceed human performance in various medical tasks, this study underscores a significant safety gap, highlighting the crucial need for human oversight and the implementation of AI safety guardrails.

13 citations1 influential

Full paper

Read the original paper

Open PDF Source page

Learning resources

arXiv PDFPDF arXiv abstract pagearXiv Google Scholar referencesGoogle Scholar Papers with Code searchPapers with Code Semantic Scholar paper pageSemantic Scholar YouTube explanationsYouTube

Reading state

Discuss in ChatGPT

Uses your own ChatGPT account. The paper context is copied into a tutor prompt before ChatGPT opens.

Preview prompt

You are my AI/ML research paper instructor. I want to deeply understand the paper below.

First, teach it in layers:
1. One-paragraph intuition.
2. Problem statement and why it mattered.
3. Key method, architecture, or algorithm.
4. Important equations or mechanisms, explained intuitively.
5. Experiments and evidence.
6. Limitations, assumptions, and failure modes.
7. How this paper influenced later AI/ML/Deep Learning/GenAI work.
8. A 30-minute study plan with checkpoints.
9. Quiz me with 5 questions and wait for my answers.

When something is not available in the attached context, say what is missing and infer carefully.

### Paper attached as context
Title: Ensuring Safety and Trust: Analyzing the Risks of Large Language Models in Medicine
Authors: Yifan Yang, Qiao Jin, Robert Leaman, Xiaoyu Liu, Guangzhi Xiong, Maame Sarfo-Gyamfi, Chang Gong, Santiago Ferriere-Steinert, W. Wilbur, Xiaojun Li, Jiaxin Yuan, Bang An
Year: 2024
Venue: arXiv.org
Categories: AI Safety, Foundation Models, Generative AI, Large Language Models
Citations: 13
Paper URL: https://arxiv.org/abs/2411.14487v1
Open PDF: https://arxiv.org/pdf/2411.14487v1

Abstract:
The remarkable capabilities of Large Language Models (LLMs) make them increasingly compelling for adoption in real-world healthcare applications. However, the risks associated with using LLMs in medical applications have not been systematically characterized. We propose using five key principles for safe and trustworthy medical AI – Truthfulness, Resilience, Fairness, Robustness, and Privacy – along with ten specific aspects. Under this comprehensive framework, we introduce a novel MedGuard benchmark with 1,000 expert-verified questions. Our evaluation of 11 commonly used LLMs shows that the current language models, regardless of their safety alignment mechanisms, generally perform poorly on most of our benchmarks, particularly when compared to the high performance of human physicians. Despite recent reports indicate that advanced LLMs like ChatGPT can match or even exceed human performance in various medical tasks, this study underscores a significant safety gap, highlighting the crucial need for human oversight and the implementation of AI safety guardrails.

Learning resources:
- PDF: arXiv PDF (https://arxiv.org/pdf/2411.14487v1)
- arXiv: arXiv abstract page (https://arxiv.org/abs/2411.14487v1)
- Google Scholar: Google Scholar references (https://scholar.google.com/scholar?q=Ensuring%20Safety%20and%20Trust%3A%20Analyzing%20the%20Risks%20of%20Large%20Language%20Models%20in%20Medicine)
- Papers with Code: Papers with Code search (https://paperswithcode.com/search?q=Ensuring%20Safety%20and%20Trust%3A%20Analyzing%20the%20Risks%20of%20Large%20Language%20Models%20in%20Medicine)
- Semantic Scholar: Semantic Scholar paper page (https://www.semanticscholar.org/paper/b53417a0182fa07cc521ea2d7721e469d0df3182)
- YouTube: YouTube explanations (https://www.youtube.com/results?search_query=Ensuring%20Safety%20and%20Trust%3A%20Analyzing%20the%20Risks%20of%20Large%20Language%20Models%20in%20Medicine+paper+explained)