Copyright
©The Author(s) 2022.
World J Radiol. Jan 28, 2022; 14(1): 19-29
Published online Jan 28, 2022. doi: 10.4329/wjr.v14.i1.19
Published online Jan 28, 2022. doi: 10.4329/wjr.v14.i1.19
Table 1 Pooled inter-reader agreement with the reference standard
| Pre-training, k | Post-training, k | P value of the difference | |
| Composition | 0.46 (95%CI: 0.37 to 0.54), moderate | 0.52 (95%CI: 0.44 to 0.61), moderate | 0.32 |
| Echogenicity | 0.36 (95%CI: 0.29 to 0.44), fair | 0.44 (95%CI: 0.37 to 0.52), moderate | 0.30 |
| Shape | 0.09 (95%CI: 0.02 to 0.21), slight | 0.67 (95%CI: 0.56 to 0.78), substantial | < 0.001 |
| Margins | 0.03 (95%CI: -0.14 to 0.08), slight | 0.05 (95%CI: -0.05 to 0.15), slight | 0.71 |
| Echogenic Foci | 0.28 (95%CI: 0.19 to 0.37), fair | 0.45 (95%CI: 0.36 to 0.53), moderate | 0.004 |
| TI-RADS Level | 0.14 (95%CI: 0.08 to 0.20), slight | 0.36 (95%CI: 0.30 to 0.42), fair | < 0.001 |
| Recommendations | 0.36 (95%CI: 0.27 to 0.45), fair | 0.50 (95%CI: 0.41 to 0.59), moderate | 0.02 |
Table 2 Percentage reader agreement with the reference standard for sonographic features
| Sonographic feature | RS | R1 pre | R1 post | R2 pre | R2 post | R3 pre | R3 post |
| Composition | n | n (%) | |||||
| Spongiform | 4 | 0 (0) | 1 (25) | 1 (25) | 1 (25) | 3 (75) | 4 (100) |
| Cystic or almost completely cystic | 11 | 3 (27.3) | 5 (45.5) | 7 (63.6) | 8 (72.7) | 10(90.9) | 10(90.9) |
| Mixed cystic and solid | 12 | 9 (75) | 6 (50) | 5 (41.7) | 7 (58.3) | 5 (58.3) | 6 (50) |
| Solid | 27 | 26 (96.3) | 26 (96.3) | 25 (92.6) | 26 (96.3) | 18 (66.7) | 19 (70.4) |
| Echogenicity | |||||||
| Anechoic | 11 | 3 (27.3) | 5 (45.5) | 5 (45.5) | 5 (45.5) | 9 (81.8) | 8 (72.7) |
| Hyperechoic or isoechoic | 27 | 23 (85.2) | 23 (85.2) | 19 (70.4) | 21 (77.8) | 19 (70.4) | 20 (74.1) |
| Hypoechoic | 12 | 2 (16.7) | 4 (33.3) | 9 (75) | 8 (66.7) | 4 (33.3) | 4 (33.3) |
| Shape | |||||||
| Wilder than tall | 42 | 38 (90.5) | 39 (92.9) | 7 (16.7) | 39 (92.9) | 41 (97.6) | 40 (95.2) |
| Taller than wide | 8 | 7 (87.5) | 7 (87.5) | 7 (87.5) | 7 (87.5) | 6 (75) | 4 (50) |
| Margins | |||||||
| Smooth or ill defined | 47 | 36 (76.6) | 35 (74.5) | 35 (74.5) | 33 (70.2) | 43 (91.5) | 45 (95.7) |
| Lobulated or irregular | 3 | 1 (33.3) | 2 (66.7) | 1 (33.3) | 2 (66.7) | 0 (0) | 0 (0) |
| Echogenic foci | |||||||
| None or large comet tail artifact | 41 | 20 (48.8) | 36 (87.8) | 29 (70.7) | 39 (95.1) | 29 (70.7) | 29 (70.7) |
| Macrocalcification | 3 | 1 (33.3) | 1 (33.3) | 0 (0) | 2 (66.7) | 2 (66.7) | 2 (66.7) |
| Punctate echogenic foci | 6 | 5 (83.3) | 4 (66.7) | 2 (33.3) | 5 (83.3) | 3 (50) | 3 (50) |
Table 3 Percentage reader agreement with the reference standard for American College of Radiology Thyroid Imaging Reporting and Data System levels
| ACR TI-RADS level | RS, n | R1 pre, n (%) | R1 post, n (%) | R2 pre, n (%) | R2 post, n (%) | R3 pre, n (%) | R3 post, n (%) |
| 1 | 11 | 1 (9.1) | 5 (45.5) | 1 (9.1) | 7 (63.6) | 10 (90.9) | 8 (72.7) |
| 2 | 9 | 3 (33.3) | 4 (44.4) | 0 (0) | 4 (44.4) | 3 (33.3) | 3 (33.3) |
| 3 | 9 | 4 (44.4) | 5 (55.5) | 1 (11.1) | 6 (66.7) | 4 (44.4) | 6 (66.7) |
| 4 | 13 | 4 (30.8) | 5 (38.5) | 5 (38.5) | 9 (69.2) | 5 (38.5) | 5 (38.5) |
| 5 | 8 | 7 (87.5) | 4 (50) | 6 (75) | 5 (62.5) | 3 (37.5) | 3 (37.5) |
Table 4 Percentage reader agreement with the reference standard for American College of Radiology Thyroid Imaging Reporting and Data System recommendations
| Recommendations | RS, n | R1 pre, n (%) | R1 post, n (%) | R2 pre, n (%) | R2 post, n (%) | R3 pre, n (%) | R3 post, n (%) |
| No follow up | 25 | 13 (52) | 17 (68) | 10 (40) | 19 (76) | 21 (84) | 22 (88) |
| Follow up | 5 | 3 (60) | 1 (20) | 1 (20) | 3 (60) | 3 (60) | 3 (60) |
| FNA | 20 | 17 (85) | 15 (75) | 18 (90) | 17 (85) | 11 (55) | 13 (65) |
Table 5 The relative sensitivity, specificity, positive predictive value, and negative predictive value per Thyroid Imaging Reporting and Data System Level on the pre-training assessment compared to the reference standard
| Pre-training, Statistics | TI-RADS 1, % | TI-RADS 2, % | TI-RADS 3, % | TI-RADS 4, % | TI-RADS 5, % |
| Sensitivity | |||||
| R1 | 9.1 (0.2-41.3) | 33.3 (7.5-70.1) | 44.4 (13.7-78.8) | 30.8 (9.1-61.4) | 87.5 (47.4-99.7) |
| R2 | 9.1 (0.2-41.3) | 0 (0-33.6) | 11.1 (0.3-48.3) | 38.5 (13.9-68.4) | 75 (34.9-96.8) |
| R3 | 90.9 (58.7-99.8) | 33.3 (7.5-70.1) | 44.4 (13.7-78.8) | 38.5 (13.9-68.4) | 37.5 (8.5-75.5) |
| Pooled | 36.4 (20.4-54.9) | 22.2 (8.6-42.3) | 33.3 (16.5-54) | 35.9 (21.2-52.8) | 66.7 (44.7-84.4) |
| Specificity | |||||
| R1 | 100 (91.0-100) | 90.2 (76.9-97.3) | 92.7 (80.1-98.5) | 62.2 (44.8-77.5) | 76.2 (60.6-88) |
| R2 | 100 (91-100) | 97.6 (87.1-99.9) | 80.5 (65.1-91.2) | 81.1 (64.8-92) | 50 (34.2-65.8) |
| R3 | 66.7 (49.8-80.9) | 97.6 (87.1-99.9) | 95.1 (83.5-99.4) | 89.2 (74.6-97) | 90.5 (77.4-97.3) |
| Pooled | 88.9 (81.8-94) | 95.1 (89.7-98.2) | 89.4 (82.6-94.3) | 76.6 (67.6-84.1) | 72.2 (63.5-79.8) |
| Positive predictive value | |||||
| R1 | 100 | 42.9 (16.8-73.6) | 57.1 (26.4-83.2) | 22.2 (10.3-41.6) | 41.2 (27.7-56.1) |
| R2 | 100 | 0 | 11.1 (1.8-46.8) | 41.7 (21.5-65.1) | 22.2 (14.8-32.1) |
| R3 | 43.5 (32.2-55.5) | 75 (26-96.2) | 66.7 (30.1-90.3) | 55.6 (28.3-79.8) | 42.9 (17.1-73.2) |
| Pooled | 48 (31.8-64.6) | 50 (25.9-74.1) | 40.9 (24.8-59.2) | 35 (23.9-48) | 31.4 (23.5-40.5) |
| Negative predictive value | |||||
| R1 | 79.6 (76.4-82.5) | 86.1 (79.4-90.8) | 88.4 (80.8-93.2) | 71.9 (62.2-79.9) | 97 (83.5-99.5) |
| R2 | 79.6 (76.4-82.5) | 81.6 (80.9-82.4) | 80.5 (75.8-84.5) | 79 (70.4-85.6) | 91.3 (75.3-97.3) |
| R3 | 96.3 (79.8-99.4) | 87 (80.7-91.4) | 88.6 (81.2-93.4) | 80.5 (72.6-86.5) | 88.4 (81.5-92.9) |
| Pooled | 83.2 (79.2-86.6) | 84.8 (81.9-87.3) | 85.9 (82.3-88.9) | 77.3 (72.5-81.5) | 91.9 (86.5-95.3) |
Table 6 The relative sensitivity, specificity, positive predictive value, and negative predictive value per Thyroid Imaging Reporting and Data System Level on the post-training assessment compared to the reference standard
| Post-training, Statistics | TI-RADS 1, % | TI-RADS 2, % | TI-RADS 3, % | TI-RADS 4, % | TI-RADS 5, % |
| Sensitivity | |||||
| R1 | 45.5 (16.8-76.6) | 44.4 (13.7-78.8) | 55.6 (21.2-86.3) | 38.5 (13.9-68.4) | 50 (15.7-84.3) |
| R2 | 63.6 (30.8-89.1) | 44.4 (13.7-78.8) | 66.7 (29.9-92.5) | 69.2 (38.6-90.9) | 62.5 (24.5-91.5) |
| R3 | 72.7 (39-94) | 33.3 (7.5-70.1) | 66.7 (29.9-92.5) | 38.5 (13.9-68.4) | 37.5 (8.5-75.5) |
| Pooled | 60.6 (42.1-77.1) | 40.7 (22.4-61.2) | 63 (42.4-80.6) | 48.7 (32.4-65.2) | 50 (29.1-70.9) |
| Specificity | |||||
| R1 | 92.3 (79.1-98.4) | 97.6 (87.1-99.9) | 90.2 (76.9-97.3) | 70.3 (53-84.1) | 81 (65.9-91.4) |
| R2 | 94.9 (82.7-99.4) | 97.6 (87.1-99.9) | 95.1 (83.5-99.4) | 73 (38.6-90.9) | 90.5 (77.4-97.3) |
| R3 | 66.7 (49.8-80.9) | 95.1 (83.5-99.4) | 97.6 (87.1-99.9) | 86.5 (71.2-95.5) | 90.5 (77.4-97.3) |
| Pooled | 84.6 (76.8-90.6) | 96.8 (91.9-99.1) | 94.3 (88.6-97.7) | 76.6 (67.6-84.1) | 87.3 (80.2-92.6) |
| Positive predictive value | |||||
| R1 | 62.5 (32-85.5) | 80 (33.6-96.9) | 55.6 (29.4-79) | 31.3 (16.3-51.5) | 33.3 (16.5-56) |
| R2 | 77.8 (45.8-93.6) | 80 (33.6-96.9) | 75 (41.8-92.6) | 47.4 (32.2-63.1) | 55.6 (29.9-78.6) |
| R3 | 38.1 (25.8-52.2) | 60 (22.6-88.5) | 85.7 (45.1-97.8) | 50 (25.6-74.4) | 42.9 (17.1-73.2) |
| Pooled | 52.6 (40.1-64.8) | 73.3 (48.6-88.9) | 70.8 (52.8-84.1) | 42.2 (31.5-53.8) | 42.9 (29-57.9) |
| Negative predictive value | |||||
| R1 | 85.7 (77.6-91.2) | 88.9 (81.7-93.5) | 90.2 (81.6-95.1) | 76.5 (66.8-84) | 89.5 (80.7-94.5) |
| R2 | 90.2 (80.8-95.3) | 88.9 (81.7-93.5) | 92.9 (83.7-97) | 87.1 (74.5-94) | 92.7 (83.7-96.9) |
| R3 | 89.7 (76.3-95.9) | 86.7 (80.3-91.2) | 93 (84.1-97.1) | 80 (71.9-86.2) | 88.4 (81.5-92.9) |
| Pooled | 88.4 (83.2-92.1) | 88.2 (84.5-91.1) | 92.1 (87.6-95) | 81 (75.5-85.4) | 90.2 (85.9-93.2) |
- Citation: Du Y, Bara M, Katlariwala P, Croutze R, Resch K, Porter J, Sam M, Wilson MP, Low G. Effect of training on resident inter-reader agreement with American College of Radiology Thyroid Imaging Reporting and Data System. World J Radiol 2022; 14(1): 19-29
- URL: https://www.wjgnet.com/1949-8470/full/v14/i1/19.htm
- DOI: https://dx.doi.org/10.4329/wjr.v14.i1.19