Dietary counseling based on artificial intelligence for patients with nonalcoholic fatty liver disease

Abstract

BACKGROUND

About 25% of the general population in Japan are reported to have nonalcoholic fatty liver disease (NAFLD). NAFLD and nonalcoholic steatohepatitis carry a risk of progressing further to hepatocellular carcinoma. The primary treatment for NAFLD is dietary therapy. Dietary counseling plays an essential role in dietary therapy. Although artificial intelligence (AI)-based nutrition management software applications have been developed and put into practical use in recent years, the majority focus on weight loss or muscle strengthening, and no software has been developed for patient use in clinical practice.

AIM

To examine whether effective dietary counseling is possible using AI-based nutrition management software.

METHODS

NAFLD patients who had been assessed using an AI-based nutrition management software application (Calomeal) that automatically analyzed images of meals photographed by patients and agreed to receive dietary counseling were given dietary counseling. Blood biochemistry tests were performed before (baseline) and 6 mo after (6M follow-up) dietary counseling. After the dietary counseling, the patients were asked to complete a questionnaire survey.

RESULTS

A total of 29 patients diagnosed with NAFLD between August 2020 and March 2022 were included. There were significant decreases in liver enzyme and triglyceride levels at the 6M follow-up compared to baseline. The food analysis capability of the AI used by Calomeal in this study was 75.1%. Patient satisfaction with the AI-based dietary counselling was high.

CONCLUSION

AI-based nutrition management appeared to raise awareness of dietary habits among NAFLD patients. However, it did not directly alleviate the burden of registered dietitians, and improvements are much anticipated.

Key Words: Artificial intelligence; Dietary counselling; Nonalcoholic fatty liver disease; Nonalcoholic steatohepatitis; Nutrition management software applications

Core Tip: Use of artificial intelligence (AI)-based nutrition management software (Calomeal) appeared to raise awareness of nonalcoholic fatty liver disease patients’ dietary habits, and they showed significant decreases in liver enzyme and triglyceride levels at the 6-mo follow-up compared to baseline. The food analysis capability of the AI package used in this study was 75.1%, and patient satisfaction with the AI-based dietary counselling was high. However, due to the limitations of the food analysis capabilities of AI, it did not directly alleviate the burden of registered dietitians, and improvements in the analytical capabilities of AI are much anticipated.

INTRODUCTION

About 25% of the general population in Japan are reported to have nonalcoholic fatty liver disease (NAFLD), and this figure is expected to increase to 39.3% by 2030[1]. NAFLD includes nonalcoholic fatty liver (NAFL) and nonalcoholic steatohepatitis (NASH). NAFL and NASH carry a risk of progressing further to hepatocellular carcinoma. Although the risk of developing liver cancer from NAFL is low (0.44 per 1000 persons per year), the risk increases with the progression of liver pathology (5.29 per 1000 persons per year in patients with NASH and 0.45 to 22.6 per 1000 persons per year in patients with liver cirrhosis)[2,3].

The primary treatment for NAFLD is dietary therapy. Dietary counseling plays an essential role in dietary therapy. In dietary counseling, patients self-report all the food items that they consumed over the previous 1 wk, and during the interview, registered dietitians calculate the caloric and nutritive values of the food items. Limitations exist with patients’ memory certainty and the listening and calculation skills of the dietitians in the field. In recent years, pictures taken with digital cameras have been used in combination with interviews. Neither of these is very efficient, because dietary counseling begins when patients come in for their consultation, and the caloric and nutrient intakes are calculated based on patients’ recalled food items consumed in the past.

Artificial intelligence (AI)-based image analysis has penetrated deeply in our daily lives, and the field of medicine is no exception. In gastroenterology, it is used to diagnose colon polyps, Helicobacter pylori infection, and stomach cancer[4-8]. Although AI-based nutrition management software applications have been developed and put into practical use in recent years, the majority focus on weight loss or muscle strengthening, and no software has been developed for patient use in clinical practice.

Thus, the present study examined whether effective dietary counseling is possible using food intake data of NAFLD patients that had been automatically analyzed with an AI-based nutrition management software application.

MATERIALS AND METHODS

Patients

This prospective study was conducted in compliance with the ethics guidelines of the 2008 Declaration of Helsinki. Approval was obtained from the Biomedical Ethics Committee of the authors’ affiliated hospital (No. 2014). Written informed consent was obtained from all patients.

Patients clinically diagnosed with NAFLD between August 2020 and March 2022 were included as subjects. NAFLD was diagnosed when fatty liver was observed in patients with an alcohol consumption equivalent to ≤ 30 g of ethanol per day in men and ≤ 20 g of ethanol per day in women[9]. Fatty liver was diagnosed when the following three criteria were confirmed on abdominal ultrasound examination: (1) Increased hepatic echogenicity; (2) positive liver-kidney contrast; and (3) deep ultrasound attenuation in the liver. Patients with chronic hepatitis B, chronic hepatitis C, autoimmune hepatitis, and primary biliary cholangitis were excluded. Patients with non-compensated liver cirrhosis and those with concomitant hepatocellular carcinoma were also excluded.

Dietary counseling using AI

“Calomeal” is a software application developed by Life Log Technology, Inc. (Tokyo, Japan) that can easily record and manage one’s daily meals and physical activity. It has been available in Japan since December 2015. The "Calomeal" software is commercially available, and anyone can purchase it. When a user takes a picture of a meal using devices such as smartphones, the cloud-based AI calculates the nutritive values of the food in the photograph, and the results are sent to the user’s device (Figure 1). The AI of Calomeal was compiled using machine learning of food image data collected from major restaurant chains, food manufacturers, and everyday home-cooked meals. It can identify approximately 18000 food images. In addition to total caloric intake, Calomeal simultaneously calculates the nutritive values of proteins, lipids, carbohydrates, glucides, dietary fiber, and salt content.

Figure 1 The concept of Calomeal. When users take a photograph of their meals with their smartphones, the artificial intelligence on the Calomeal server calculates the nutritive value of the food included in the photograph, and the results are sent to the user’s device.

This study used a customized version of Calomeal that was developed in cooperation with Life Log Technology. The following three areas were customized: (1) Changing the device from a smartphone to an iPad Mini; (2) simplifying food selection and offering more detailed options on consumed quantities; and (3) redirecting the AI analysis results to the study’s dedicated personal computer and not to the patients’ devices.

More specifically, since some NAFLD patients were older, iPad Minis were used as the device instead of smartphones because iPad Minis have larger display screens (Figure 2). When patients took pictures of a food item, the AI suggested the name of three possible food choices. The patients then selected from the three choices the food item that they consumed. When none of the offered three choices matched, “no suitable choice” was selected. Next, the patients chose the quantity of food that they consumed from the list of options. Eight options were available. Setting 100% as the normal serving size, the options were as follows: 200%, 150%, 100%, 66% (about two-thirds), 50%, 33% (about one-third), 10% (about one bite), and 0% (although a picture was taken, no food was consumed). AI calculated the nutritive values once these tasks were completed. The analysis results were forwarded to the study’s dedicated personal computer managed by the authors (Figure 3). Breakfast, lunch, dinner, or snacks were automatically determined based on the time when the foods were photographed.

Figure 2 Display on the iPad Mini. Instead of using a smartphone as the device, an iPad Mini, which has a larger screen, is used. When a patient photographs a food item, the artificial intelligence suggests the names of three possible food choices. The patient selects the food that he or she ate from one of the three choices. If none of the three choices matches, the patient chooses “no suitable choice.” Next, the patient selects the quantity of food consumed from the list of options. Eight options are available. Setting 100% as the normal serving size, the choices are as follows: 200%, 150%, 100%, 66% (about two-thirds), 50%, 33% (about one-third), 10% (about one bite), and 0% (picture was taken but no food was consumed). Breakfast, lunch, dinner, or snacks are automatically determined based on the time that the foods were photographed.

Figure 3 Calomeal concepts that were customized in this study. The food images photographed with an iPad Mini are analyzed by the artificial intelligence on the Calomeal server. The following information is sent to the dedicated personal computer at the authors’ hospital: ID, basic information of patients, captured food images, food lists of last one week, calculation of nutritive value, and bar charts of the nutritive value. Registered dietitians conduct dietary counseling based on these data.

After photographing one week’s worth of pictures of meals, the patients came in for their dietary counseling session. By the time of the patients’ visits, the AI had prepared a list of a patient’s one week’s worth of photographed foods (Figure 4) and bar charts showing the amounts of caloric, protein, lipid, carbohydrate, glucide, dietary fiber, and salt intakes (Figure 5). Registered dietitians participating in this study conducted dietary counseling while presenting the abovementioned data.

Figure 4 Analysis results of the photographed food images. A list is created based on the results of the artificial intelligence calculation using food names and intakes (%) selected by patients. Energy, protein, lipid, carbohydrate, salt, glucide, and dietary fiber content of each meal over 1 wk are shown.

Figure 5 Bar charts of nutritive values for 1 wk. A: Daily energy intake; B: Daily protein intake; C: Daily lipid intake; D: Daily carbohydrate intake; E: Daily salt intake; F: Daily glucide intake; G: Daily dietary fiber intake. The bar charts show the daily energy, protein, lipid, carbohydrate, salt, glucide, and dietary fiber intakes for 1 wk.

The photographed food images for which the patients could not find a suitable choice at the time of taking the picture were examined by the registered dietitians before the patients’ visits, and after checking the photographed images, the registered dietitians entered the correct item name.

Principle of Calomeal

Around dozens of pieces of photograph data for one food (or one product) are prepared for machine learning. These photographs are learned by deep neural network, and foods (or a product) are analyzed by the pattern of the color and form.

The nutrient of common foods was calculated based on "standard table of food composition" announced by Japanese Ministry of Education, Culture, Sports, Science and Technology by dietitian of Life Log Technology company. The nutrient of foods of major restaurant chains and food manufacturers was calculated using the data published in the home page of each company. When the nutrient was not announced in home page of the company, the dietitian calculated the nutrient from announced raw materials.

Effects of dietary counseling using Calomeal

Of all the patients enrolled in the study, patients who agreed to receive dietary counseling were given dietary counseling using Calomeal. Blood biochemistry tests were performed before (baseline) and 6 mo after (6M follow-up) dietary counseling. Aspartate aminotransferase (AST), alanine aminotransferase (ALT), γ-glutamyltransferase (GGT), total cholesterol (T-cho), and triglyceride (TG) levels were compared between the baseline and 6M follow-up.

Body weight of all patients was compared between the baseline and 6M follow-up.

Analysis capability of AI

As noted earlier, the AI used in the present study was capable of analyzing approximately 18000 meal items. However, since most of these items were collected from menus of major restaurant chains and food manufacturers, its ability to identify everyday home-cooked meals was unknown. The AI’s ability to analyze food images was evaluated by calculating the percentage (%) of unidentified foods (i.e., foods for which the results of the AI’s automated analysis did not match the foods actually consumed by the patients, thereby categorized under “no suitable choice”) among all food items.

In addition, the time spent by registered dietitians before the patient visits in entering the correct food names of the unidentified “no suitable choice” food items was also measured.

Acceptance of Calomeal

After the dietary counseling, subjects in the Calomeal Group were asked to complete a questionnaire survey consisting of the following questions:

Question 1: Were you glad that you were given an AI-based dietary counseling session? (Yes or No).

Question 2: Did you find the dietary counseling rewarding? (Yes or No).

Question 3: Have you become more conscious of improving your dietary lifestyle? (Yes or No).

Question 4: Would you like to receive another AI-based dietary counseling session? (Yes or No).

Statistical analysis

Continuous data, such as those from the blood biochemistry tests, are shown as the mean ± standard deviation. The paired Wilcoxon test was used to test the difference in each parameter between the start of observation (baseline) and after 6 mo (6M follow-up). P < 0.05 was considered significant.

RESULTS

There were 29 patients who agreed and received dietary counseling using Calomeal. The operation of Calomeal using iPads was accepted by all 29 patients. Table 1 shows the patients’ characteristics. The patients had been taking all of these medications before starting this study, and no new drugs were initiated after the start of the observation in this study.

Table 1 Patients' characteristics (n = 29).

Male/Female	16/13
Age (yr)	56.4 ± 14.3
Body weight (kg)	74.1 ± 13.1
BMI (kg/m2)	28.4 ± 5.2
Metabolic diseases
Diabetes mellitus (Yes/No)	10/19
Dyslipidemia (Yes/No)	14/15
Concomitant drugs
SGLT2 inhibitor	2
DPP-4 inhibitor	3
Thiazolidinedione
GLP-1 agonist	1
Statin	7
Bezafibrate
Pemafibrate
EPA and DHA preparation	1
AST (U/L)	50.2 ± 33.4
ALT (U/L)	53.8 ± 39.0
GGT (U/L)	80.3 ± 84.3
T-B (mg/dL)	1.3 ± 0.9
Alb (mg/dL)	4.2 ± 0.6
eGFR (mL/min)	71.1 ± 11.5
HbA1c (%)	6.4 ± 0.8
T-cho (mg/dL)	191.3 ± 35.0
TG (mg/dL)	126.5 ± 57.8
WBC (103/μL)	5.6 ± 1.5
Hb (g/dL)	14.5 ± 1.7
Plts (104/μL)	17.7 ± 7.4

AST: Aspartate aminotransferase; ALT: Alanine aminotransferase; GGT: γ-Glutamyltransferase; T-Bil: Total bilirubin; Alb: Albumin; eGFR: Estimated glomerular filtration rate; HbA1c: Glycated hemoglobin A1c; T-chol: Total cholesterol; TG: Triglycerides; WBC: White blood cells; Hb: Hemoglobin; Plts: Platelets.

Effects of dietary counseling

Table 2 shows the AST, ALT, GGT, hemoglobin A1c (HbA1c), T-cho, and TG levels and body weight at baseline and 6M follow-up. AST, ALT, GGT, and TG levels were significantly lower (P = 0.0088, 0.0133, 0.0494, and 0.0246, respectively) at the 6M follow-up compared to the levels at baseline. Body weight was significantly lower (P = 0.0472) at the 6M follow-up compared to that at baseline.

Table 2 Changes of blood biochemistry parameters and body weight.

	Baseline	6M Follow-up	P value
AST (U/L)	50.2 ± 33.4	34.7 ± 14.7	0.0088
ALT (U/L)	53.8 ± 39.0	35.3 ± 16.8	0.0113
GGT (U/L)	80.3 ± 84.3	66.3 ± 82.9	0.0494
HbA1c (%)	6.4 ± 0.8	6.2 ± 0.6	0.27832
T-cho (mg/dL)	191.3 ± 35.0	189 ± 34.3	0.2109
TG (mg/dL)	126.5 ± 57.8	104.1 ± 60.6	0.0426
Body weight (kg)	74.0 ± 13.1	71.2 ± 12.3	0.0472

AST: Aspartate aminotransferase; ALT: Alanine aminotransferase; GGT: γ-Glutamyltransferase; HbA1c: Glycated hemoglobin A1c; T-chol: Total cholesterol; TG: Triglycerides.

Analysis capability of AI

Table 3 shows the total number of food items photographed in 1 wk of all patients in the Calomeal Group, the number of food items categorized under “no suitable choice”, and the food analysis capability of the AI. The mean number of total photographed food items was 62.6 (20 to 104), the mean number of food items categorized under “no suitable choice” was 15.0 (1 to 32), and the mean analysis capability was 75.1% (51.5 to 98.6%). Before dietary counseling sessions, registered dietitians spent on average 25.9 min (4.5 to 67.0 min) identifying the food items categorized under “no suitable choice.”

Table 3 Number of photographed food items, analysis rate, and time for confirmation.

Case	All foods	No of candidates	Analysis rate (%)	Time for confirmation (min)
1	57	13	77.2	25.0
2	31	2	93.5	4.5
3	90	31	65.6	67.0
4	40	7	82.5	20.5
5	74	20	73.0	31.0
6	69	1	98.6	6.0
7	24	3	87.5	9.5
8	20	9	55.0	12.0
9	47	9	80.9	28.5
10	87	29	66.7	35.5
11	26	12	53.8	14.0
12	35	8	77.1	10.0
13	85	25	70.6	36.5
14	95	8	91.6	8.5
15	53	15	71.7	20.0
16	84	32	61.9	39.5
17	77	13	83.1	14.0
18	51	7	86.3	13.0
19	33	16	51.5	20.0
20	103	9	91.3	19.5
21	32	6	81.3	5.5
22	68	25	63.2	32.5
23	88	30	65.9	50.0
24	41	18	56.1	66.0
25	74	22	70.3	36.0
26	104	16	84.6	34.0
27	84	22	73.8	47.5
28	83	13	84.3	25.5
29	61	13	78.7	20.5
Average	62.6	15.0	75.1	25.9

Acceptance of Calomeal

Table 4 shows the findings of the responses to the questionnaire survey. When the patients were asked “Were you glad that you were given an AI-based dietary counseling session?” in Question 1, all 29 patients said “Yes.” When the patients were asked “Did you find the dietary counseling rewarding?” in Question 2, 15 of the 29 patients responded “Yes.” When the patients were asked “Have you become more conscious of improving your dietary lifestyle after the dietary counseling session?” in Question 3, all 29 patients responded “Yes.” When the patients were asked “Would you like to receive another AI-based dietary counseling session?” in Question 4, four of the 29 patients responded “No.”

Table 4 Questionnaire results.

Question	Yes	No
1	29	0
2	15	14
3	29	0
4	25	29

DISCUSSION

AI helps diagnose NAFLD through its use in diagnostic imaging procedures such as ultrasound, computed tomography, and magnetic resonance imaging and pathological diagnostic procedures[10-12]. However, there have been no reports of using AI in nutritional therapies and dietary counseling for NAFLD. Thus, the authors focused on this aspect and planned the present study.

There is a long history of using AI in food analysis. In 1983, Chen et al[13] reported that the caloric intake of foods could be calculated using a small wearable computer that can be attached to clothing like a tin badge. This small computer called the eButton is being used for the healthy transformation of homemade foods[14,15]. When the eButton is used solely for identifying foods, the sensitivity is 85.0% and the specificity is 85.8%. Since the small eButton can be worn by attaching it to clothing, privacy is maintained[13]; however, its widespread use by the general public may be difficult because it is a purpose-built computer.

In Japan, there are five types of nutrition management software applications that can be downloaded on smartphones. Although there may be some differences in their analysis capability, all of them are very easy to use[16]. Of these software applications, Oka et al[17] used Asken for dietary counseling of type 2 diabetes mellitus patients and reported its benefit. Of these five software applications, the present study focused on Calomeal. The main reason for choosing Calomeal was its ease of operation.

In the present study, significant decreases in AST, ALT, GGT, and TG levels and body weight were observed at the 6M follow-up compared to the levels at baseline. However, there is an important limitation of this study. The present study did not compare dietary counseling using conventional methods with dietary counseling using Calomeal. The reason was that, in contrary to the authors’ expectations, fewer patients gave consent for dietary counseling after being diagnosed with NAFLD during the study period. Thus, it cannot be asserted that this study’s dietary counseling using Calomeal is superior to conventional methods. This issue needs to be addressed in future research.

The food analysis capability of the AI used by Calomeal in this study was 75.1%. The food analysis capability using Life Log Technology’s own data was reported to be 85.7%, which showed a discrepancy with the present study’s findings. One possible reason may be the small percentage of data for everyday home-cooked meals, since the food items registered in Calomeal’s AI consisted primarily of major restaurant chain menus, food manufacturers’ items, and convenience store products. In the present study, NAFLD patients, especially elderly patients, often consumed unique homemade dishes whose food names were difficult to identify just by looking at the pictures.

In other words, there were various daily diets, especially the cooking methods of family diets cannot be unified. These made food illegible and energy intake difficult to calculate. This can lead to significant error in the study.

Assuming that a patient eats four dish items per meal, three meals a day, this means that 4 × 3 × 7 = 84 items should have been photographed in 1 wk. However, as shown in Table 2, there was a large difference among patients, some with very few photographed food items (20 items) and some who had photographed many food items (104 items). The following reasons were given: (1) Patients could not take pictures while they were dining out because they were conscious of their surroundings; (2) patients could not bring the iPads to work; and (3) Japanese “teishoku” set lunches often plate various dishes on one plate, and some patients took pictures of the whole plate as one photograph. In the case of (3), the AI could not analyze multiple food items at once. This issue was reported to Life Log Technology as feedback, and the Calomeal software at the time of writing this paper had already taken this into account.

In the present study, the names of all food images that patients selected as “no suitable choice” on the screen were examined and entered individually by the registered dietitians. This procedure, which on average took 25.9 min (4.5 to 67.0 min), was conducted before the dietary counseling sessions. Since the duration of one dietary counseling session was 30 min, this means that preparation required about the same amount of time. However, when meeting the patient face-to-face, a list of food items photographed by the patient over the previous 1 wk and bar charts of various nutrients were already prepared. Thus, it is possible that more productive dietary counseling was offered within the allocated 30-min session.

The questionnaire results showed that all of the respondents said that they were glad to have undergone dietary counseling using Calomeal. Similarly, they all responded that they had become aware of their dietary habits. On the other hand, 15 of 29 patients found the content of the dietary counseling rewarding. The majority of patients in the Calomeal Group in this study had a good command of their smartphones, and these patients had free access to the internet. Perhaps there was a lack of originality in the hospital-based dietary counseling, since nutritional information, to some extent, is available when one searches the internet. When the patients were asked whether they would like to receive dietary counseling again using Calomeal, four patients responded “No.” During outpatient follow-up interviews, all four patients commented that they “did not want to bring the devices out of their homes or to work.” This issue could be resolved by changing the device to a smartphone.

CONCLUSION

When an AI-based nutrition management software application automatically analyzed images of meals photographed by NAFLD patients, there were significant decreases in AST, ALT, GGT, and TG levels after 6 mo (6M follow-up). Thus, this method appeared to raise awareness of dietary habits of NAFLD patients. On the other hand, due to the limitations of the food analysis capabilities of AI, it did not directly alleviate the burden of registered dietitians, and improvements in the analytical capabilities of AI are much anticipated.

ARTICLE HIGHLIGHTS

Research background

Approximately 27000 people a year die from liver cancer in Japan. Liver cancer from non-viral liver disease increases while cancerogenesis from viral liver decreases. In the non-viral liver disease, nonalcoholic fatty liver disease (NAFLD) increases in particular. Therefore, carcinogenesis restraint from NAFLD is urgent business to reduce liver cancer death. Diet therapy is the first choice for the treatment of NAFLD and nutrition education for this purpose becomes extremely important.

Research motivation

The authors paid attention to the nutrition education using the artificial intelligence and led to the idea of this study using the application software called the "Calomeal". The authors have the patients understand the importance of the diet by performing the nutrition education using the artificial intelligence for the NAFLD patients and want to help inhibit the cancerogenesis from NAFLD. A study on optimization of the nutrition education using the artificial intelligence (AI) for NAFLD is the attempt that leads the world and thinks with pioneer positioning of the future health promotion medical care.

Research objectives

Patients clinically diagnosed with NAFLD between August 2020 and March 2022 were included as subjects. "Calomeal" as a software application developed by Life Log Technology, Inc. (Tokyo, Japan) was used for the nutrition education. Blood biochemistry tests were performed before (baseline) and 6 mo after (6M follow-up) dietary counseling. After the dietary counseling, the patients were asked to complete a questionnaire survey.

Research methods

There were significant decreases in liver enzyme and triglyceride levels at the 6M follow-up compared to baseline. The food analysis capability of the AI used by Calomeal in this study was 75.1%. Patient satisfaction with the AI-based dietary counselling was high.

Research results

The authors have the patients understand the importance of the diet because the NAFLD patients receive a nutrition education using the artificial intelligence, and the purpose of this study is to carry a help of the cancerogenesis restraint.

Research conclusions

When an AI-based nutrition management software application automatically analyzed images of meals photographed by NAFLD patients, liver function was improved significantly. On the other hand, due to the limitations of the food analysis capabilities of AI, improvements in the analytical capabilities of AI are much anticipated.

Research perspectives

The direction of future research is nutrition education using more advanced artificial intelligence to inhibit the carcinogenesis from NAFLD.