Interobserver and intraobserver agreement for Letournel acetabular fracture classification system using 3-dimensional printed solid models

Abstract

BACKGROUND

Acetabular fractures pose diagnostic and surgical challenges. They are classified using the Judet-Letournel system, which is based solely on X-ray. However, computed tomography (CT) imaging is now more widely utilized in diagnosing these injuries. The emergence of 3-dimensional (3-D) printing technology in varying orthopedic fields has provided surgeons a solid model that improves their spatial understanding of complex fractures and ability to plan pre-operatively.

AIM

To evaluate the reliability of the Judet-Letournel classification system of acetabular fractures, when using either CT imaging or 3-D printed models.

METHODS

Seven patients with acetabular fractures underwent pelvic CT imaging, which was then used to create solid, 3-D printed models. Eighteen orthopaedic trauma surgeons responded to questionnaires regarding fracture classification and preferred surgical approach. The same questionnaire was completed using only CT imaging, and two weeks later, using only 3-D printed models. The inter- and intra-observer agreement rates were then analyzed.

RESULTS

Inter-observer agreement rates based on CT imaging or 3-D printed models were moderate for fracture classification: κ = 0.44, κ = 0.55, respectively (P < 0.001) and fair for preferred surgical approach: κ = 0.34, κ = 0.29, respectively (P < 0.005). Intra-observer agreement rates for fracture classification and preferred surgical approach comparing CT imaging or 3-D printed models were moderate: κ = 0.48, κ = 0.41, respectively. No significant difference in intra-observer agreement was detected when comparing orthopedic pelvic specialists to general orthopedic traumatologists.

CONCLUSION

The Judet-Letournel classification demonstrated only moderate rates of agreement. The use of 3-D printed models increased the inter-observer agreement rates with respect to fracture classification, but decreased it with respect to the preferred surgical approach. This study highlights the role of 3-D printed models in acetabular fractures by improving spatial understanding of these complex injuries, thus providing more reliable fracture diagnoses and alternative viewpoints for pre-operative planning.

Key Words: Acetabulum; Pelvic trauma; Acetabular fracture; Three-dimensional printing; Three-dimensional reconstruction; Judet-Letournel

Core Tip: Three-dimensional (3-D) imaging and printing is an emerging technique in the field of orthopedic surgery. This study highlights the utility of 3-D printing in the treatment of complex acetabular fractures, as it relates to the traditionally used Judet-Letournel classification and the accustomed surgical approaches.

INTRODUCTION

Acetabular fractures pose a great diagnostic challenge due to the complexity and variability of each fracture pattern. Treatment, whether conservative or surgical, is influenced by the initial diagnosis of the fracture pattern and classification. Therefore, it is essential that these fractures are accurately characterized before decision making[1-4]. The Judet-Letournel classification system for acetabular fractures is the current gold standard for describing these injuries. The system is based on the theory of anterior and posterior walls and columns within the acetabulum[5,6]. Based on this anatomical description, they described five elementary fracture patterns and five associated fracture patterns.

The Judet-Letournel classification system was described using specific pelvic radiographs, termed Judet views, which include iliac oblique and obturator oblique views[5]. However today, most trauma centers perform computed tomography (CT) for high-energy or polytrauma patients. Using these CT scans, three-dimensional (3-D) reconstructions are created, and have become standard diagnostic tools in many trauma centers (Figure 1). Studies have shown that using two-dimensional (2-D) and 3-D CT reconstructions enable improved understanding of fracture patterns and anatomy, in particular with complex pelvic and acetabular fractures[7,8]. Some novel CT-based classification systems have been proposed as well[9], challenging traditional methods.

Figure 1 Computed tomography 3-dimensional reconstruction of an acetabular fracture.

3-D printing technology can utilize 3-D CT reconstructions to manufacture a tangible model out of a variety of materials, ranging from plastics to metals. This technology is becoming more widespread in various industries, thus making it more affordable and accessible[10,11]. 3-D printing first entered the field of medicine in the early 2000s, when dental implants and tailored prostheses were produced. The use of this technology had increased substantially in several areas of medicine, creating anatomic models, prostheses, implants, and other accessories[12,13].

In orthopedic surgery, 3-D printing has been used both for surgical planning as well as patient specific instrumentation (PSI). PSI utilizes 3-D printing to manufacture instrumentation that is unique to a patient’s anatomy. Its use has been reported on in varying fields of orthopedics including arthroplasty, orthopedic oncology, trauma, and spine surgery. Several studies have demonstrated that the use of 3-D printed models and implants decreased operative time and improved patient outcomes[14-18].

The reliability of the Judet-Letournel classification has been investigated in several trauma centers. Previous studies evaluating inter-observer agreement of acetabular fractures based on CT scans demonstrated high variability between studies and their reported rates of agreement, with a range of κ = 0.6-0.7[7,19-21].

To our knowledge, the use of 3-D printed models to test the reliability of the Judet-Letournel classification system has not yet been reported on in the literature. Therefore, we raised the hypothesis that the use of 3-D printed models of acetabular fractures may improve the interobserver and intraobserver agreement of fracture classification, and influence surgeons’ preferred surgical approach.

MATERIALS AND METHODS

Seven patients with varying acetabular fractures that were surgically treated at our institution were included. No initial attempt was made to characterize them according to Letournel’s system. Each patient underwent non-contrast CT imaging of the pelvis with 3-D reconstructions. The CT images of each patient were de-identified and assigned a coded number (Figure 2). Each case was stored onto several portable drives and distributed to the reviewing physicians.

Figure 2 Computed tomography image: an axial cross-section illustrating a right acetabular fracture. Circled in red is the anonymously assigned coding of the case. All identifying details have been omitted from the test.

3-D printed models of the fractured acetabuli were produced, using a UP Plus 2 3-D printer (PP3DP, United States), using Acrylonitrile Butadiene Styrene (ABS), a chemical compound (C₈H₈)_x·(C₄H₆)_y·(C₃H₃N)_z. Technical limitations mandated a 0.6-0.8/1 ratio of model to real size. The models were engraved with serial numbers (Figure 3). An encoding table was created, with the legend available only to the statistician following data collection. This was a blinded study in that reviewing surgeons were unable to identify which CT scan corresponded to each 3-D printed model.

Figure 3 Three-dimensional printed models of acetabular fractures.

Eighteen orthopedic trauma surgeons from various hospitals in Israel were selected. Nine of those surgeons specialize in pelvic and acetabular injuries and routinely operate on these injuries. The other nine orthopedic trauma surgeons do not routinely operate on pelvic and acetabular injuries. Each participating surgeon completed the same questionnaire twice. Initial responses were based only off of CT images. They then completed the same questionnaire two weeks later using only the 3-D printed models. They were asked to answer the following two questions: (1) What is the fracture type according to Judet-Letournel classification system? and (2) What is the preferred surgical approach for each case?

Statistical analysis

In order to comply with previous literature, the (Cohen's) Kappa coefficient was used to evaluate Interobserver Agreement between all participants with respect to fracture classification and corresponding preferred surgical approach using either CT images or 3-D printed models. Intraobserver Agreement was also tested for each individual surgeon with respect to fracture classification and corresponding preferred surgical approach using either CT images or 3-D printed models. For each of these parameters a correlation test was performed between the two abovementioned groups of surgeons.

In order to calculate the inter-observer agreement between the various examiners with respect to fracture classification using either CT images or 3-D printed models, we compared each individual’s responses to the remaining examiners' responses. This was done for each test that was performed. The number of matches was then correlated to the number of comparisons made (removing duplicate matches), and the results from each imaging method were compared using a Wilcoxon signed-rank test. Statistical analyses were performed using SPSS program (version 20, IBM), with a P value below 0.05 considered significant.

RESULTS

Seven cases of acetabular fractures were selected with differing levels of complexity, at the discretion of the authors. The eighteen participating surgeons examined all CT scans two weeks before the 3-D printed models were examined. The results from all questionnaires are shown in Tables 1-4 and Figures 4 and 5.

Figure 4 Mean Kappa rate describing agreement between reviewers regarding the fracture classification and preferred surgical approach using computed tomography and 3-dimensional printed models (mean kappa in circle). A: The fracture classification; B: The preferred surgical approach. CT: Computed tomography; 3-D: Three-dimensional; Fx: Fracture.

Figure 5 Decision change regarding the surgical approach. A: The effect of reviewing the 3-dimensional model; B: Division of the subgroups demonstrates a change in surgical approach in double rates when the classification is also changed.

The inter-observer agreement regarding fracture classification based on CT and 3-D models was moderate: κ = 0.44 (SE range: 0.0-0.24) and κ = 0.55 (SE range: 0.0-0.22) respectively, with a statistically significant difference between the two modalities (P < 0.001). The inter-observer agreement regarding the preferred surgical approach based on CT and 3-D models was fair: κ = 0.34 and κ = 0.29 (SE range: 0.0-0.39), with a statistically significant difference between the two modalities (P < 0.005) (Tables 1 and 2, Figure 4).

Table 1 Inter-observer agreement on classification, based on computed tomography or 3-dimensional printed models.

Fx classification
		CT			3-D model
#Surgeon	Match pairs	Mean kappa	SE min	SE max	Mean kappa	SE min	SE max
Mean	153	0.44	0.00	0.24	0.55	0.00	0.22
1	17	0.06	0.06	0.20	0.54	0.17	0.21
10	17	0.22	0.04	0.23	0.41	0.15	0.20
7	17	0.24	0.04	0.20	0.65	0.15	0.22
4	17	0.35	0.06	0.21	0.58	0.15	0.20
12	17	0.36	0.17	0.22	0.26	0.14	0.22
16	17	0.40	0.13	0.22	0.61	0.15	0.20
17	17	0.41	0.11	0.22	0.68	0.15	0.21
14	17	0.42	0.11	0.24	0.60	0.14	0.21
8	17	0.45	0.14	0.22	0.56	0.16	0.20
9	17	0.48	0.10	0.24	0.69	0.00	0.21
6	17	0.50	0.11	0.24	0.54	0.17	0.21
15	17	0.52	0.14	0.22	0.51	0.16	0.21
3	17	0.56	0.11	0.24	0.55	0.15	0.21
11	17	0.57	0.00	0.22	0.69	0.00	0.21
18	17	0.57	0.00	0.22	0.40	0.16	0.21
2	17	0.60	0.00	0.24	0.69	0.00	0.21
5	17	0.60	0.00	0.24	0.44	0.14	0.21
13	17	0.66	0.00	0.24	0.58	0.15	0.20

CT: Computed tomography; 3-D: Three-dimensional; Fx: Fracture.

Table 2 Inter-observer agreement on surgical approach, based on computed tomography or 3-dimensional printed models.

Surgical approach
		CT			3-D Model
#Surgeon	Match pairs	Mean kappa	SE min	SE max	Mean kappa	SE min	SE max
Mean	153	0.35	0.00	0.39	0.30	0.00	0.39
17	17	0.18	0.17	0.32	0.43	0.00	0.39
12	17	0.18	0.16	0.31	0.09	0.11	0.24
9	17	0.25	0.19	0.31	0.34	0.16	0.36
14	17	0.28	0.16	0.39	0.29	0.15	0.36
15	17	0.31	0.18	0.27	0.25	0.13	0.23
16	17	0.31	0.00	0.36	0.43	0.15	0.35
18	17	0.31	0.00	0.36	0.20	0.17	0.29
10	17	0.32	0.19	0.28	0.19	0.16	0.29
8	17	0.33	0.21	0.30	0.33	0.18	0.24
3	17	0.36	0.18	0.36	0.29	0.17	0.26
2	17	0.36	0.20	0.28	0.35	0.15	0.30
11	17	0.39	0.20	0.27	0.26	0.16	0.28
6	17	0.41	0.18	0.32	0.21	0.19	0.32
1	17	0.41	0.00	0.30	0.43	0.00	0.39
4	17	0.41	0.00	0.30	0.35	0.11	0.26
7	17	0.44	0.20	0.32	0.37	0.19	0.30
5	17	0.50	0.00	0.39	0.24	0.14	0.36
13	17	0.50	0.00	0.39	0.30	0.13	0.39

CT: Computed tomography; 3-D: Three-dimensional.

The intra-observer agreement regarding fracture classification among all 18 surgeons when comparing the two imaging modalities was moderate: κ = 0.48. The surgeons specializing in pelvic and acetabular injuries demonstrated a slightly lower rate of agreement (κ = 0.45) when compared to the general orthopedic trauma specialists (κ = 0.50), though this difference was not statistically significant (P = 0.592). The intra-observer agreement regarding the preferred surgical approach among all the surgeons when comparing the two imaging modalities was moderate: κ = 0.41. The surgeons specializing in pelvic and acetabular injuries demonstrated a lower rate of agreement (κ = 0.37) when compared to the general orthopedic trauma specialists (κ = 0.50), though this difference was also not statistically significant (P = 0.33) (Tables 3 and 4, Figure 5).

Table 3 Intra-observer agreement on classification and surgical approach, between computed tomography and 3-dimensional printed models.

#Surgeon		Fx classification					Surgical approach
1	1	-0.02					0.70
2	1	0.50					0.18
3	1	0.50					0.32
4	1	0.50					0.55
5	1	0.83					0.36
6	1	0.48					0.28
7	1	0.32					0.77
8	1	0.66					0.34
9	1	0.67					0.22
10	1	0.22					0.40
11	1	0.50					0.58
12	1	0.49					0.25
13	1	0.66					1.00
14	1	0.50					0.70
15	1	0.64					0.15
16	1	0.48					0.19
17	1	0.32					0.16
18	1	0.35					0.19
		Mean	Median	Min	Max	P value	Mean	Median	Min	Max	P value
Total	18	0.48	0.50	-0.02	0.83	0.592	0.41	0.33	0.15	1.00	0.331
Pelvis	9	0.45	0.50	-0.02	0.83		0.37	0.25	0.15	1.00
Trauma	9	0.50	0.50	0.22	0.67		0.45	0.36	0.18	0.77

Table 4 Reviewers' change of decision (computed tomography vs 3-dimensional model).

#Surgeon	Total decisions	Fracture classification		Surgical approach
		Number of decisions changed	% of decisions changed	Number of decisions changed	% of decisions changed
1	7	6	86	1	14
2	7	3	43	4	57
3	7	3	43	3	43
4	7	3	43	2	29
5	7	1	14	2	29
6	7	3	43	3	43
7	7	4	57	1	14
8	7	2	29	3	43
9	7	2	29	3	43
10	7	5	71	3	43
11	7	3	43	2	29
12	7	3	43	3	43
13	7	2	29	0	0
14	7	3	43	1	14
15	7	2	29	4	57
16	7	3	43	3	43
17	7	4	57	3	43
18	7	4	57	3	43
Total	126	56	44	44	35
Pelvis	63	29	46	20	32
Trauma	63	27	43	24	38

After examining the 3-D printed models, surgeons changed their initial responses regarding fracture classification and preferred surgical approach 56 out of 126 times (44%), and 44 out of 126 times (35%), respectively (Table 4). There were no significant differences between the two groups of specialists in this respect (Table 3).

A large variability in responses amongst the cases was evident. For case 2, only three (17%) reviewers changed their decision regarding fracture classification, and only one (6%) reviewer with respect to preferred surgical approach. In contrast, for case 5, fourteen (78%) reviewers changed their response regarding fracture classification, and 9 (50%) reviewers with respect to preferred surgical approach (Table 4).

A total of 70 out of 126 (55.5%) responses regarding fracture classification using CT imaging alone were not changed after examination of the 3-D models. However, in 22.8% of these cases the examiners changed their responses regarding the preferred surgical approach. Of the remaining 56 out of 126 (44.5%) responses that were changed regarding fracture classification, 50% of these surgeons changed their response regarding the preferred surgical approach (Table 4 and Figure 5).

DISCUSSION

The primary outcome of this study demonstrated that 3-D printed models of 7 different acetabular fractures significantly increased inter-observer agreement with respect to fracture classification, while decreasing inter-observer agreement with respect to preferred surgical approach. The use of 3-D printed models did not demonstrate significant difference in intra-observer agreement for both fracture classification and preferred surgical approach. No significant difference in intra-observer agreement for fracture classification and preferred surgical approach was detected when analyzing responses from pelvic and acetabular specialists and general orthopedic traumatologists, separately.

The Judet-Letournel classification system for acetabular fractures is widely used and establishes an algorithm for surgical treatment. This classification system is based on X-ray imaging using Judet views[5]. However, with the prevalent use of CT imaging, most patients with pelvic or actebular injuries are not treated without review of this advanced imaging modality.

Numerous studies have evaluated the reliability and effectiveness of the Judet-Letournel classification system[7,8,19-21], using plain radiographs, 2-D and 3-D CT imaging. We've raised the hypothesis that 3-D printed solid models could contribute to the spatial understanding of these highly complex and variable fractures. The use of 3-D printed models affords surgeons a more comprehensive spatial understanding of these injuries, thus improving their ability to plan pre-operatively. Advancement in 3-D printing technology has allowed a relatively simple way to create a real-size detailed model of a fractured acetabulum.

In contrast to previous studies involving a single medical center, this study recruited eighteen surgeons from numerous level 1 trauma centers in the nation, many of who regularly perform complex operations of the acetabulum. Our intention was to provide an accurate and generalizable picture regarding the reliability of the Judet-Letournel classification system. When we asked the surgeons to classify these fractures and provide their preferred surgical approach, our premise was that there is no “correct” answer, and that the agreement rate would be the sole reference.

Tables 1 and 2 demonstrates a moderate rate of agreement regarding the fracture classification based on CT imaging. This rate is lower than reported on previous literature[7,8,10]. One explanation might be that the fractures selected for this study were more complex and difficult to assess. Another explanation is the fact that the participating surgeons work at different medical centers. There's an element of habit and common practice at each medical center, or unified training under the same pelvic specialist, which may create a bias.

The degree of inter-observer agreement for the fracture classification based on the 3-D printed models was found to be only slightly higher, though still moderate. Although both imaging modalities were within moderate agreement, the difference between them was statistically significant (P < 0.001). The higher rate of agreement using the 3-D printed models is likely a result of the improved spatial visualization and understanding of each fracture and the possibility to examine each one from different viewpoints.

The inter-observer agreement rate among all the surgeons regarding the preferred surgical approach according to CT imaging (Tables 1 and 2) was found to be fair. This is in contrast to a slightly lower, yet still fair, rate of agreement when using the 3-D models, which was statistical significant (P = 0.005). This can be explained by the fact that different surgeons may decide to approach the same fracture using different methods. Varying personal preferences among each surgeon can explain the lower rate of agreement when compared to that of fracture classifications. The higher agreement rate for the surgical approach when using CT vs 3-D models may be due to various options revealed to a surgeon when holding a model in his hand. We believe that, in this respect, use of the 3-D printed models raises a more innovative way of thinking and undermines previous decision making patterns. Simply put, one might say that more information may only contribute to a problem's complexity.

The intra-observer agreement regarding the fracture classification and the preferred surgical approach based on CT imaging vs 3-D printed models was moderate (Table 3), with no significant difference between pelvic and trauma specialists in both parameters. Table 4 demonstrates the variance between the cases chosen for the study. It can be assumed that in some cases, the complex fractures posed a greater diagnostic challenge for the reviewers and raised more questions regarding surgical approaches. Another explanation for this can be attributed to technical reasons. Some non-displaced fracture lines, which can be identified through CT imaging, might have been “obliterated” in the printing process and are difficult to identify in the solid models. A higher resolution printer, larger scales, or possibly other modalities (e.g., 3-D holograms) may serve to reduce this effect.

Figure 5 illustrates the surgeons’ decision-making process regarding the appropriate surgical approach for each classified fracture, once the 3-D printed models were examined. Although 56% maintained the same fracture classification, 13% of those surgeons decided to change their preferred surgical approach. In our view, this represents a significant insight into how a fracture is evaluated, and moreover, how it would be addressed surgically. Our findings demonstrate that the Judet-Letournel classification system does not comprise all the information surgeons need for their decision-making. It is likely that the 3-D printed models provide additional information that affects a surgeon’s preferred surgical approach.

The collected data did not demonstrate significant differences between the pelvic specialists and general orthopedic traumatologists in all parameters. Presumably, as part of their work, trauma specialists who do not regularly operate on acetabular fractures are still thoroughly familiar with the theoretical material.

CONCLUSION

The results of this study indicate that the currently accepted Judet-Letournel classification system for acetabular fractures demonstrates only moderate rates of agreement by CT imaging alone. Use of 3-D printed models increased the interobserver agreement rates with respect to fracture classification, however decreased the interobserver agreement rates with respect to the preferred surgical approach. Due to the inherent anatomical complexity of acetabular fractures, these models allow for improved visuospatial understanding of these fractures and enable more accurate classification. Additionally, the 3-D printed models allow surgeons to examine a fracture from infinite perspectives and consider the best surgical approach before operating. The ability of a surgeon to hold a 3-D model in his hands as part of the preoperative planning process can improve their decision-making. This surgical aid may stimulate renewed thinking of fracture diagnosis and preferred surgical approaches for acetabular fractures, and may contribute to improved surgical outcome.

ARTICLE HIGHLIGHTS

Research background

There are numerous studies examining the reliability of the Judet-Letounel classification system for acetabular fractures using traditional radiographs and computed tomography (CT). However, 3-dimensional (3-D) printing is an emerging technology that hasn't been thoroughly investigated in the field of orthopedics in terms of imaging and pre-operative planning.

Research motivation

We evaluated the intra and inter-observer reliability of the Judet-Letournel classification system, with respect to fracture classification and preferred surgical approach. We compared the use of 3-D printed models of acetabular fractures to the current standard use of CT scans.

Research objectives

The study aims to illustrate the added value of 3-D printed models as a reliable method to more accurately characterize a patient’s acetabular fracture, and aid in the decision regarding the preferred surgical approach.

Research methods

Seven patients with acetabular fractures underwent a CT scan with 3-D reconstructions. We then created 3-D printed models of the fractured acetabula. Eighteen trauma surgeons were surveyed to classify each fracture and identify their preferred surgical approach, on two separate occasions, using one of each imaging modality alone.

Research results

The inter-observer agreement regarding fracture classification based on CT and 3-D printed models was moderate for both: κ = 0.44 (SE range: 0.0-0.24), and κ = 0.55 (SE range: 0.0-0.22), respectively; this difference was statistically significant (P < 0.001). The inter-observer agreement regarding the preferred surgical approach based on CT and 3-D printed models was fair for both: κ = 0.34, and κ = 0.29 (SE range: 0.0-0.39), respectively; this difference was statistically significant (P < 0.005). The intra-observer agreement regarding fracture classification among all 18 surgeons when comparing the two imaging modalities was moderate: κ = 0.48, as for the preferred surgical approach: κ = 0.41.

Research conclusions

3-D printed models improve the inter-observer reliability of the Judet-Letournel classification system, when compared to the use of standard CT scans. However, the inter-observer agreement regarding the surgical approach was decreased, likely due to the added perspective and visualization of the fractures.

Research perspectives

3-D printed models improve visuospatial understanding of complex fractures. Its utility and contribution for better patient outcomes should be investigated in future prospective randomized controlled trials.

ACKNOWLEDGEMENTS

The authors wish to express their gratitude to Professor Sosna J (Chairman, Department of Radiology, Hadassah Medical Center, Jerusalem, Israel) for his contribution in the process of realizing the 3-D printed models.