Li Y, Huang CK, Hu Y, Zhou XD, He C, Zhong JW. Exploring the performance of large language models on hepatitis B infection-related questions: A comparative study. World J Gastroenterol 2025; 31(3): 101092 [DOI: 10.3748/wjg.v31.i3.101092]
Corresponding Author of This Article
Cong He, Associate Chief Physician, MD, Department of Gastroenterology, Jiangxi Provincial Key Laboratory of Digestive Diseases, Jiangxi Clinical Research Center for Gastroenterology, Digestive Disease Hospital, The First Affiliated Hospital, Jiangxi Medical College, Nanchang University, No. 17 Yong Waizheng Street, Nanchang 330006, Jiangxi Province, China. hecong.1987@163.com
Research Domain of This Article
Gastroenterology & Hepatology
Article-Type of This Article
Basic Study
Open-Access Policy of This Article
This article is an open-access article which was selected by an in-house editor and fully peer-reviewed by external reviewers. It is distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Yu Li, Chen-Kai Huang, Yi Hu, Xiao-Dong Zhou, Cong He, Jia-Wei Zhong, Department of Gastroenterology, Jiangxi Provincial Key Laboratory of Digestive Diseases, Jiangxi Clinical Research Center for Gastroenterology, Digestive Disease Hospital, The First Affiliated Hospital, Jiangxi Medical College, Nanchang University, Nanchang 330006, Jiangxi Province, China
Co-corresponding authors: Cong He and Jia-Wei Zhong.
Author contributions: Li Y, Huang CK, Hu Y, and Zhou XD performed the data acquisition and statistical analysis; Li Y and Huang CK contributed equally as co-first author; Li Y wrote the manuscript; He C and Zhong JW designed the study and revised the manuscript, they contributed equally as co-corresponding authors; and all authors read and approved the final manuscript.
Supported by National Natural Science Foundation of China, No. 82260133; the Key Laboratory Project of Digestive Diseases in Jiangxi Province, No. 2024SSY06101; and Jiangxi Clinical Research Center for Gastroenterology, No. 20223BCG74011.
Institutional review board statement: Institutional review board approval was not required for this study since it is an analysis of data and no patients or animals were affected by the study.
Conflict-of-interest statement: All the authors report no relevant conflicts of interest for this article.
Data sharing statement: All data generated or analyzed during this study are included in the supplementary information files.
Open-Access: This article is an open-access article that was selected by an in-house editor and fully peer-reviewed by external reviewers. It is distributed in accordance with the Creative Commons Attribution NonCommercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: https://creativecommons.org/Licenses/by-nc/4.0/
Corresponding author: Cong He, Associate Chief Physician, MD, Department of Gastroenterology, Jiangxi Provincial Key Laboratory of Digestive Diseases, Jiangxi Clinical Research Center for Gastroenterology, Digestive Disease Hospital, The First Affiliated Hospital, Jiangxi Medical College, Nanchang University, No. 17 Yong Waizheng Street, Nanchang 330006, Jiangxi Province, China. hecong.1987@163.com
Received: September 4, 2024 Revised: October 29, 2024 Accepted: December 3, 2024 Published online: January 21, 2025 Processing time: 106 Days and 21.9 Hours
Abstract
BACKGROUND
Patients with hepatitis B virus (HBV) infection require chronic and personalized care to improve outcomes. Large language models (LLMs) can potentially provide medical information for patients.
AIM
To examine the performance of three LLMs, ChatGPT-3.5, ChatGPT-4.0, and Google Gemini, in answering HBV-related questions.
METHODS
LLMs’ responses to HBV-related questions were independently graded by two medical professionals using a four-point accuracy scale, and disagreements were resolved by a third reviewer. Each question was run three times using three LLMs. Readability was assessed via the Gunning Fog index and Flesch-Kincaid grade level.
RESULTS
Overall, all three LLM chatbots achieved high average accuracy scores for subjective questions (ChatGPT-3.5: 3.50; ChatGPT-4.0: 3.69; Google Gemini: 3.53, out of a maximum score of 4). With respect to objective questions, ChatGPT-4.0 achieved an 80.8% accuracy rate, compared with 62.9% for ChatGPT-3.5 and 73.1% for Google Gemini. Across the six domains, ChatGPT-4.0 performed better in terms of diagnosis, whereas Google Gemini demonstrated excellent clinical manifestations. Notably, in the readability analysis, the mean Gunning Fog index and Flesch-Kincaid grade level scores of the three LLM chatbots were significantly higher than the standard level eight, far exceeding the reading level of the normal population.
CONCLUSION
Our results highlight the potential of LLMs, especially ChatGPT-4.0, for delivering responses to HBV-related questions. LLMs may be an adjunctive informational tool for patients and physicians to improve outcomes. Nevertheless, current LLMs should not replace personalized treatment recommendations from physicians in the management of HBV infection.
Core Tip: Hepatitis B virus (HBV) infection remains a global health problem that may cause chronic hepatitis, liver cirrhosis, or hepatocellular carcinoma. There is a notable trend among the public to acknowledge HBV-related information to improve outcomes. Artificial intelligence is a large language model that provides updated and helpful knowledge. Since the ChatGPT was developed by OpenAI, an increasing number of studies have explored its utility in responding to medical questions. This study evaluates and compares the abilities of OpenAI’s ChatGPT and Google’s Gemini in answering test questions concerning HBV using both subjective and objective metrics.
