Exploring the performance of large language models on hepatitis B infection-related questions: A comparative study

Table 1 Quality indicators (scientific adequacy) for answers from ChatGPT-3.5, ChatGPT-4.0, and Google Gemini

Common questions	Sources of answers	Answer lengths, 1st run	Answer lengths, 2nd run	Answer lengths, 3rd run	Grades, mean	Grades, P value
Overall (mean)	ChatGPT-3.5	275	366	352	3.50	0.2
	ChatGPT-4.0	274	252	238	3.69
	Google Gemini	307	322	325	3.53
Risk factors
What are the transmission modes of hepatitis B virus?	ChatGPT-3.5	189	316	400	3.67	0.296
	ChatGPT-4.0	358	241	220	4
	Google Gemini	264	291	291	3.33
Clinical manifestation
What are the symptoms of hepatitis B infection?	ChatGPT-3.5	247	333	356	3.67	0.216
	ChatGPT-4.0	269	276	295	3.67
	Google Gemini	226	349	352	3
Diagnosis
What is the most accurate test for diagnosing Hepatitis B infection?	ChatGPT-3.5	223	341	348	3.67	0.027
	ChatGPT-4.0	307	349	280	4
	Google Gemini	281	281	281	3
Treatment
Can hepatitis B infection be cured clinically?	ChatGPT-3.5	334	357	395	3.67	0.216
	ChatGPT-4.0	271	324	264	3.67
	Google Gemini	268	360	359	3
What are the indications of antiviral therapy for patients infected with hepatitis B virus?	ChatGPT-3.5	368	367	402	3.67	0.296
	ChatGPT-4.0	351	334	296	3.33
	Google Gemini	385	384	392	3
Can patients infected with hepatitis B virus be pregnant during antiviral treatment?	ChatGPT-3.5	341	392	383	3.33	0.079
	ChatGPT-4.0	319	247	242	4
	Google Gemini	369	352	351	4
Do patients diagnosed with chronic hepatitis B during pregnancy need antiviral therapy?	ChatGPT-3.5	366	416	383	3	0.296
	ChatGPT-4.0	230	313	256	3.33
	Google Gemini	325	330	375	3.67
Can patients diagnosed with chronic hepatitis B during lactation be treated with antiviral therapy?	ChatGPT-3.5	366	419	391	3.33	0.296
	ChatGPT-4.0	245	190	218	3.67
	Google Gemini	362	328	330	4
Prevention
How long should a newborn receive the first dose of hepatitis B vaccine after birth?	ChatGPT-3.5	133	392	185	3.67	0.296
	ChatGPT-4.0	182	146	146	3.33
	Google Gemini	193	207	201	4
Can pregnant women receive hepatitis B vaccine?	ChatGPT-3.5	181	397	338	4
	ChatGPT-4.0	179	183	149	4
	Google Gemini	277	318	318	4
How often should patients with hepatitis B virus infection be reexamined?	ChatGPT-3.5	209	421	328	3	0.027
	ChatGPT-4.0	275	171	205	3.33
	Google Gemini	330	334	334	4
Prognosis
What are the complications of hepatitis B infection?	ChatGPT-3.5	343	235	305	3.33	0.216
	ChatGPT-4.0	300	245	280	4.00
	Google Gemini	405	326	318	3.33