Utilization of artificial intelligence in the diagnosis of pes planus and pes cavus with a smartphone camera

Abstract

BACKGROUND

Pes planus (flatfoot) and pes cavus (high arch foot) are common foot deformities, often requiring clinical and radiographic assessment for diagnosis and potential subsequent management. Traditional diagnostic methods, while effective, pose limitations such as cost, radiation exposure, and accessibility, particularly in underserved areas.

AIM

To develop deep learning algorithms that detect and classify such deformities using smartphone cameras.

METHODS

An algorithm that integrated a deep convolutional neural network (CNN) into a smartphone camera was utilized to detect pes planus and pes cavus deformities. This case control study was conducted at a tertiary hospital with participants recruited from two orthopaedic foot and ankle clinics. The CNN was trained and tested using photographs of the medial aspect of participants’ feet, taken under standardized conditions. Participants included subjects with standard foot alignment, pes planus, or pes cavus determined by an expert clinician using the foot posture index. The model’s performance was assessed in comparison to clinical assessment and radiographic measurements, specifically lateral tarsal-first metatarsal angle and calcaneal inclination angle.

RESULTS

The CNN model demonstrated high accuracy in diagnosing both pes planus and pes cavus, with an optimized area under the curve of 0.90 for pes planus and 0.90 for pes cavus. It showed a specificity and sensitivity of 84% and 87% for pes planus detection, respectively; and 97% and 70% for pes cavus, respectively. The model’s prediction correlated moderately with radiographic lateral Meary’s angle measurements, indicating the model’s excellent reliability in assessing food arch deformity (P < 0.05).

CONCLUSION

This study highlights the potential of using a smartphone-based CNN model as a screening tool that is reliable and accessible for the detection of pes planus and pes cavus deformities, which is especially beneficial for underserved communities and patients with pain generated by subtle foot arch deformities.

Key Words: Artificial intelligence; Computer vision; Three-dimensional scanning; Lower extremity; High-arch; Flatfoot

Core Tip: The study demonstrates the integration of a deep convolutional neural network (CNN) with smartphone cameras to diagnose pes planus and pes cavus, common foot deformities, with high accuracy. By utilizing a non-invasive and accessible screening tool, this method eliminates the need for traditional radiographic assessments, making it particularly beneficial for underserved communities. The CNN model showed high specificity and sensitivity, suggesting its potential for early detection and management of foot arch deformities, ultimately enhancing patient care and reducing healthcare costs.

INTRODUCTION

Foot deformities that affect the medial longitudinal arch are very common affecting up to 75% in some populations in the form of pes planus or pes cavus. Pes planus, or flatfoot, is a condition characterized by the collapse of the medial longitudinal arch, foot abduction at the talonavicular joint, and hindfoot valgus[1]. Its prevalence varies between 15%-50% based on the demographics of the population and age group[2-5]. It is often associated with pain and discomfort following periods of exertion, particularly in children who might be affected congenitally[6]. In contrast, pes cavus, also known as pes cavo varus, is a condition characterized by an abnormal elevation of the medial longitudinal arch of the foot, plantar flexion of the first ray, forefoot pronation, and hindfoot inversion[7]. It is commonly seen in up to 10% of the population[8].

Individuals with pes planus are prone to developing conditions such as tendinitis, plantar fasciitis, metatarsalgia, knee pain, and lower-back pain[9-13]. On the other hand, individuals with pes cavus may develop chronic lateral ankle instability, peroneal tendinitis, and anteromedial ankle arthrosis due to altered pressure distribution at the ankle joint[14,15]. Thus, patients with either deformity benefit from a timely diagnosis followed by prompt subsequent management to minimize and prevent the long-term sequelae[14,16-18].

The most commonly used diagnostic methods for identifying pes planus or cavus deformities include clinical assessment and weightbearing imaging techniques. The foot posture index (FPI) is a standardized diagnostic clinical tool used to evaluate the multidimensional aspects of the foot, which scores feet along a spectrum of cavus to planus features[19,20]. While not commonly used, weightbearing computed tomography or plain radiographs are diagnostic or adjunctive assessment modalities for foot arch deformities. Nevertheless, these modalities are costly, expose the patient to radiation, and may not be accessible to the general population, especially in underserved areas[21].

Recently, there have been significant advancements in computer vision technologies, particularly in deep learning (DL) and deep convolutional neural network (CNN)[22]. These advancements have improved the capabilities of such technology making them more widely available and accessible. One notable development is their integration with mobile devices, broadening their accessibility and application. Our study aimed to evaluate the efficacy and utility of a CNN algorithm for detecting and classifying foot arch deformities using smartphone cameras in comparison to traditional clinical assessment using the FPI. We hypothesize that our model can detect the presence of deformity, particularly when subtle, with high accuracy, using standardized phone camera images of the medial aspect of the foot.

MATERIALS AND METHODS

Data collection

This case control study was approved by the Institutional Review Board (IRB) (No. 2022P001722). Patients were recruited from two orthopaedic foot and ankle clinics at our institution, which is a tertiary hospital in Boston, Massachusetts, United States. Based on their FPI, feet that were deemed as having pes planus or pes cavus deformity by a foot and ankle surgeon were labeled as cases, while feet that did not have any arch deformity were labeled as normal controls. The inclusion criteria for the recruited individuals in this study were adults aged ≥ 18 years old experiencing foot pain, able to ambulate and bear weight on their feet, and capable of providing informed consent for participation. The exclusion criteria were the presence of fractures in the foot including stress fractures and traumatic injuries, peripheral neuropathy, neurological injuries, or structural deformities related to chronic diseases (i.e., Charcot's foot).

As with other prediction models, our algorithm required training and testing datasets (link to smartphone application tool). The training (online) set was obtained from volunteers who submitted two photographs (one per foot) in accordance with the IRB showing the medial aspect of their feet while weightbearing in a single-leg stance. A thorough data cleaning process was performed to remove any photos submitted that did not adhere to the instructions or were of low quality. Selection boxes that bound the entire foot and ankle were placed manually to create the final dataset (Figure 1). A group of expert clinicians then independently categorized the collected photos into four classes of severity based on gross anatomy: (1) Normal arch; (2) Mild; (3) Moderate; and (4) Severe pes planus.

Figure 1 An example of an unscaled rectangular box that bounds the entire foot and ankle for images collected for deep learning algorithm training.

On the other hand, the testing (offline) set was collected in person, labeled clinically by a trained researcher, and confirmed by a foot and ankle surgeon, categorizing patients into three groups: (1) Those with pes planus; (2) Those with pes cavus; and (3) Those without either condition as normal controls by using the FPI. The photos were collected in a controlled environment to ensure the quality of data used in the algorithm. For participants with lateral radiographs of their feet, a radiologist measured the lateral tarsal-first metatarsal angle (LTMA), also called lateral Meary’s angle, and calcaneal inclination angle (CIA) for the correlation with the output score from the algorithm and to establish the severity of the deformity, if present.

Data pre-processing

The input image resolution for the algorithm was set to 360 pixels × 640 pixels (height × width) for analysis. The online testing set was divided into two sets using an 8-to-2 ratio; the larger set was designated for training purposes, while the other set was utilized for calculating intermediate metrics. The sub-training set was further divided using a 9-to-1 ratio. Once again, the larger subset was utilized for model training, and the smaller subset served as a validation set for early stopping to prevent overfitting.

Selection boxes that bound the entire foot and ankle on the photographs submitted online were used to crop and process the images to an aspect ratio of 360 pixels × 640 pixels. Data augmentation, such as flipping, blurring, and color jittering, was applied to the photographs to artificially increase the training set for propagation into the CNN. The offline testing set was also cropped to a 360 × 640-pixel aspect ratio, using the bottom edge of the image as a reference. Each resultant image and its horizontally flipped counterpart were passed through the algorithm during DL inference. The final prediction probability was calculated as the average of the outputs of the two mirrored images (Figure 2).

Figure 2 Flowchart showing the deep learning inference data processing pathway for the testing (offline) set.

Classification model

The pre-trained Memory Efficient Vision Transformer with Cascaded Group Attention (EfficientViT_M0) classification neural network model was fine-tuned using our data on pes planus binary classification task[23]. The input resolution of the images was set to 360 pixels × 640 pixels, attention biases within the neural network were reset, and the output layer of the neural network was replaced with a linear layer of a single output to create a one-class classification model. The same model was utilized for pes cavus classification but with reversed predictions.

Model training

Hyperparameters-learning rate (LR), weight decay (WD), and weightings for binary cross entropy (BCE) loss-were determined through trial-and-error experimentation with EfficientViT_M0 on a five-fold cross-validation process until area under the curve (AUC) achieved 0.85. The Adam optimizer with an LR of 1e-4 and WD of 1e-5 was used to fine-tune the model. The batch size for model training was set to 16 random images created from the online set. Training was interrupted when the criteria for early stopping were met to prevent overfitting. The weighted BCE loss was used as training loss and a validation metric to determine early stopping with a weight of 8 for normal feet and severe pes planus, a weight of 2 for moderate pes planus, and a weight of 4 for mild pes planus. BCE loss in this study was obtained by measuring the difference between the target (0 for no pes planus; 1 for pes planus) and the neural network output. The early stopping procedure involved calculating the BCE loss of the predetermined validation set after every 100 iterations or 1600 random images from the testing set post-augmentation. If the BCE loss of the validation set fails to decrease after ten passes of such a process, model training is terminated. The appropriately fitted model (no under or overfitting) with the minimal BCE loss on the validation set after training termination was chosen as the testing model.

Statistical analysis

The resulting algorithm was evaluated using the testing dataset without any additional training. For pes planus detection, the output was used as is. The predictions were reversed for pes cavus prediction. The AUC and the area under the precision-recall (PR) curve (PR-AUC) were calculated by using output scores for each condition without any post-processing and were utilized to assess model performance. The DeLong method was applied for AUC to acquire the 95%CI to ensure significance[24]. A threshold score was required to obtain binary predictions of a participant's condition. The F1 score for each possible threshold was calculated, and the threshold that maximizes the F1 score was acquired. While algorithm training was not required for this methodology, the threshold selection procedure might be viewed as a form of training. As a result, the leave-one-out cross-validation (LOOCV) method was employed to calculate the metrics. The LOOCV approach was repeated for each N, where N is the study population. For each participant i, the approach estimated the threshold using N-1 scans (i.e., scans that did not include person i) and compared the angle of the omitted scan to the calculated value. After cross-validation, a list of binarized predictions is produced and used to calculate the confusion matrix-based metrics. Pearson’s correlation coefficient was used to determine the correlation between LTMA and CIA measurements with the algorithm’s output score for severity analysis of pes planus. Python 3.9.6 (Python Software Foundation; Scipy 1.9.1) was used to perform statistical analyses. Outcomes are presented as mean ± SD or percentage. A cut-off alpha value of 0.05 was used to determine significance.

RESULTS

The resultant training dataset included 1433 unique participants with 2537 observations, 905 females (63.2%) and 528 males (36.8%), with an average age of 32.56 ± 14.76 years and body mass index (BMI) of 21.9 ± 4.53 kg/m². The cohort included 1683 cases of normal feet, 606 cases of mild, 188 cases of moderate, and 60 cases of severe pes planus. The resulting testing dataset consisted of 246 feet from 123 recruited participants, 73 females (59.3%) and 50 males (40.7%), with an average age of 37.88 ± 18.14 years and BMI of 25.26 ± 5.16 kg/m². Within the testing set, 70 feet were labeled and identified as pes planus (28.4%), and 20 feet (8.1%) were labeled and identified as pes cavus. There were 156 healthy feet (normal) in the testing set. A total of 36 lateral weightbearing radiographs were available for LTMA and CIA measurements.

Out of the total 246 feet scanned for testing, the number of feet without pes planus was 176 (71.5 %). AUC and PR-AUC curves for positive and negative classes were calculated on raw scores. For the pes planus model, the AUC score was 0.897 (95%CI: 0.852-0.943). The PR-AUC for the positive class was 0.75 and 0.95 for the negative class. The PR curves are represented in Figure 3A and B. Table 1 presents the classification metrics calculated on the binarized output after LOOCV for pes planus.

Figure 3 Precision-recall curves. A: Positive class for pes planus; B: Negative class for pes planus; C: Positive class for pes cavus; D: Negative class for pes cavus.

Table 1 Classification performance metrics calculated on the binarized output after leave-one-out cross-validation for pes planus.

Metric name	F1 score	Accuracy	Precision (positive predictive value)	Recall (sensitivity)	Specificity	Negative predictive value
Metric value	0.77	0.85	0.69	0.87	0.84	0.94

On the other hand, the reversed extrapolated model for pes cavus demonstrated reasonable metrics even though the model was not intentionally trained to identify this deformity specifically. There were 20 (8.1%) identified cases of pes cavus in our data set. For this class, The AUC score was 0.898 (95%CI: 0.826-0.970). The PR-AUC for the positive class was 0.56 and 0.99 for the negative class. PR curves for pes cavus are represented in Figure 3C and D. The classification metrics that were calculated on the binarized output after LOOCV for pes cavus are presented in Table 2.

Table 2 Classification performance metrics calculated on the binarized output after leave-one-out cross-validation for pes cavus.

Metric name	F1 score	Accuracy	Precision (positive predictive value)	Recall (Sensitivity)	Specificity	Negative predictive value
Metric value	0.68	0.95	0.67	0.70	0.97	0.97

Among the 123 patients recruited for the study, 25 patients had 36 lateral weightbearing foot radiographs obtained as a part of their diagnostic workup. The LTMA and CIA were derived from these radiographs and showed a correlation of-0.65 (95%CI: -0.81 to -0.41; P < 0.001) for LTMA (Figure 4A) and -0.23 (95%CI: -0.52 to 0.09; P = 0.16) for CIA (Figure 4B) with the neural network output score. There were 21 (58.3%) weightbearing radiographs for a random subset of normal healthy feet that showed a CIA of 19.19° (± 5.59°) and LTMA of -0.57° (± 4.87°). There were 11 (30.6%) weightbearing radiographs for a random subset of flatfeet showing a CIA of 15.64° (± 4.52°) and LTMA of -13.91° (± 7.08°). For a random subset of cavus feet, the 4 (11.1%) weightbearing radiographs showed a CIA of 25.7° (± 6.13°) and LTMA of 12.75° (± 4.11°).

Figure 4 Cluster plot. A: It showing the neural network's output when compared to lateral tarsal-first metatarsal angle, also known as lateral Meary’s angle; B: It showing the neural network's output when compared to calcaneal inclination angle.

DISCUSSION

Our study showed that a DL CNN algorithm using smartphone cameras can identify foot arch deformities by analyzing photos of the medial foot reliably and accurately independently of clinical examination. The algorithm was able to detect pes planus deformity with 84% specificity and 87% sensitivity. Likewise, it was able to detect pes cavus deformity with 97% specificity and 70% sensitivity. As our method is focused on the sagittal plane, which is clinically represented by the medial aspect of the foot, we used the LTMA and CIA as radiographic measures to correlate with the algorithm’s output score. This also enabled the model to evaluate the severity of the deformity when present. The LTMA is the most reliable radiographic measure of the foot arch, and its correlation with the algorithm’s classification output was statistically significant. The CIA, on the other hand, did not correlate with the algorithm severity output. The lack of CIA’s correlation with the clinical alignment may be due to abnormal associated foot pronation in pes planus deformity, which may limit the reliability of this radiographic measurement to assess the severity of the deformity for certain cases. This phenomenon is substantiated by Lautzenheiser and Kramer[25] and Agoada and Kramer[26] in an anthropometric study showing that an excessively pronated foot can occasionally manifest with a normal CIA. However, when our sample population was taken as a whole, the CIA measurements performed on our random subset of patients showed appropriate mean measurements that conform with cut-off measures established for foot-type classification by Lautzenheiser and Kramer[25] for pes planus (< 16°), normal arch (16°-24.99°), and pes cavus (> 24.99°). The inverse correlation between the CNN output and LTMA (and CIA) is expected as the output of the algorithm is the probability that pes planus is present (ranging between 0 and 1); in other words, the smaller the LTMA, the higher the probability of the foot being labeled as pes planus, and vice-versa.

Pes planus and pes cavus deformities are often diagnosed in the clinical setting. However, radiographic confirmation and assessment of etiologies are commonly required as they provide additional information about the nature of the deformity and any associate bony abnormalities. Although these foot pathologies are typically non-emergent when isolated, congenital or acquired pes planus and pes cavus deformities may indicate underlying conditions that can lead to functional disability and other comorbidities if left untreated[27-29]. Moreover, patients might experience remarkable pain and associated symptoms from subtle, not readily recognizable foot arch deformities. The algorithm developed in this study could be used as a prescreening tool prior to any clinical encounter to identify suspected foot arch deformities at an early stage that may prompt physicians to pursue more conservative management pathways to address a pain-generating, even subtle, foot arch deformity. Our DL neural algorithm detects the presence of foot deformities and their severity remotely, omitting the need for radiological examination and increasing the possibility of early diagnosis, especially for patients living in underserved areas. Screening tools like this algorithm expand the applicability of telehealth and facilitate their use for a larger number of pathologies, especially in musculoskeletal diseases[21]. Thus, the application of this algorithm becomes two-fold: (1) To screen for pes planus and pes cavus deformities; and (2) To facilitate telehealth encounters and associated assessments.

To the best of our knowledge, this is the first study to apply a neural network model to aid the diagnosis of pes planus or pes cavus deformity without using plain radiographs. Several extant studies have used similar DL-based tools to automate pes planus detection. For example, Eksen et al[30] have similarly developed a prototype deep neural network to automate pes planus identification using smartphone cameras, albeit using a relatively smaller dataset of only 34 participants. Their algorithm conveyed an excellent performance for detecting pes planus when referencing clinical classification of foot arch deformity by a clinician. However, their algorithm was not validated using the FPI and compared to radiographic assessments, which we have done in our study. Gül et al[31] successfully developed and used a CNN to diagnose pes planus deformity on plain radiographs. However, radiographic imaging was still required for their algorithm, implying burdensome visits to the clinic and additional imaging appointments. Patients often do not see their primary care providers or obtain a referral to a podiatrist or foot and ankle surgeon until they experience persistent pain in their feet. However, this novel mobile application and algorithm enables patients who are cognizant about their foot health to explore their curiosity freely and independently before further invasive interventions are indicated. This algorithm can also be a supporting tool for periodically customizing orthotics and footwear solutions for patients over the long term.

It is important to note, however, that our training and testing datasets were obtained exclusively from an adult population (> 18 years), whereas the majority of pes planus and pes cavus cases are first identified in children[27,32]. Additional limitations of our algorithm are the availability or accessibility of our technology to our target population, especially in underserved areas, and how the quality of the image can limit their interpretation. The algorithm requires a certain photo quality threshold not all phones are currently capable of achieving, in addition to the bandwidth of wireless connection required to support the data transfer. With that being said, the 360 × 640 resolution requirement is supported by many recent smartphone models, so we consider this a minor limitation that will likely disappear as mobile technology continues to develop. What’s more important for optimizing the performance of this model is obtaining properly oriented photos and following appropriate weightbearing instructions. Lastly, while the algorithm can reliably detect the presence of pes planus and pes cavus deformity, distinguishing between moderate and severe cases remains a limitation. For pes planus specifically, our method does not elaborate on whether the deformity is flexible or rigid, as the photos are static representations of the foot arch during weightbearing conditions that may not adequately evaluate associated heel deformity.

CONCLUSION

This study introduces a novel approach to detecting and classifying pes planus and cavus deformities using a DL algorithm via smartphone cameras. This method, marked by high specificity and sensitivity, makes foot arch deformity detection more accessible and non-invasive, particularly when subtle and not readily recognized by routine physical examination. Notably, it reduces the need for radiographic imaging, thus decreasing radiation exposure and healthcare costs, which provides significant benefits, especially for underserved communities.

ACKNOWLEDGEMENTS

We want to acknowledge Guss D, Waryasz G, Kwon J, Nassour N, and Toy K for their contributions in technical suggestions and help during the project.