Comparison of diagnostic validity of two autism rating scales for suspected autism in a large Chinese sample

Abstract

BACKGROUND

Autism is the most common clinical developmental disorder in children. The childhood autism rating scale (CARS) and autistic autism behavior checklist (ABC) are the most commonly used assessment scales for diagnosing autism. However, the diagnostic validations and the corresponding cutoffs for CARS and ABC in individuals with suspected autism spectrum disorder (ASD) remain unclear. Furthermore, for suspected ASD in China, it remains unclear whether CARS is a better diagnostic tool than ABC. Also unclear is whether the current cutoff points for ABC and CARS are suitable for the accurate diagnosis of ASD.

AIM

To investigate the diagnostic validity of CARS and ABC based on a large Chinese sample.

METHODS

A total of 591 outpatient children from the ASD Unit at Beijing Children’s Hospital between June and November 2019 were identified. First, the Clancy autism behavior scale (CABS) was used to screen out suspected autism from these children. Then, each suspected ASD was evaluated by CARS and ABC. Receiver operating characteristic (ROC) curve analysis was used to compare diagnostic validations. We also calculated the area under the curve (AUC) for both CARS and ABC.

RESULTS

We found that the Cronbach alpha coefficients of CARS and ABC were 0.772 and 0.426, respectively. Therefore, the reliability of the CARS was higher than that of the ABC. In addition, we found that the correlation between CARS and CABS was 0.732. Next, we performed ROC curve analysis for CARS and ABC, which yielded AUC values of 0.846 and 0.768, respectively. The cutoff value, which is associated with the maximum Youden index, is usually applied as a decision threshold. We found that the cutoff values of CARS and ABC were 34 and 67, respectively.

CONCLUSION

This result indicated that CARS is superior to ABC in the Chinese population with suspected ASD.

Key Words: Suspected autism spectrum disorder, Children, Childhood autism rating scale, Autism behavior checklist, Receiver operating characteristic curve, Cutoff value

Core Tip: This study compared the diagnostic validities of childhood autism rating scale (CARS) and autism behavior checklist (ABC) based on a large Chinese sample. We found that the CARS was superior to the ABC in terms of its diagnostic validity for assessing suspected autism spectrum disorder (ASD) cases in children. In the clinical evaluation for suspected ASD, our findings suggest that the cutoff values of CARS and ABC were 34 and 67, respectively.

INTRODUCTION

Autism is a neurodevelopmental disorder that occurs in early childhood and results in stereotypical interests, communication deficits, social deficits and repetitive behaviors[1]. Autism spectrum disorder (ASD) has received increasing attention in recent years[2]. Moreover, early diagnosis and intervention play a critical role in the treatment of ASD patients[3]. However, early diagnosis lacks specific biological markers. The diagnosis of ASD was based on a detailed developmental history, parents’ report, observed behavior, and validated screening tools or criteria of the diagnostic and statistical manual of mental disorders, fifth edition (DSM-5)[4,5]. Therefore, clinical assessments are important for diagnosing ASD[4,6,7]. The scales most commonly used to diagnose ASD in children are the autism behavior checklist (ABC) and childhood autism rating scale (CARS).

There are numerous suspected ASD cases (showing one or more symptoms of ASD but no final diagnosis) that originate from community health-service centers and preschools in China, most of whom are initially screened via the Clancy autism behavior scale (CABS)[8]. The cutoff point of 14 for CABS is always used as the criterion for suspected ASD in China. When a suspected ASD case was identified, his or her parents received suggestions to go to a hospital for a final diagnosis. When they reach hospitals for final diagnoses, most of them might undergo further assessments, such as ABC or CARS. According to previous studies on various assessments of ASD, CARS exhibits better diagnostic validation than ABC[9]. However, the diagnostic validations and the corresponding cutoff for CARS and ABC in individuals with suspected ASD remain unclear[4]. Notably, it remains unclear whether CARS is a better diagnostic tool than ABC for suspected ASD in China. Furthermore, it is unclear whether the current cutoff points for ABC and CARS are suitable for the accurate diagnosis of ASD.

Therefore, the purpose of this study was to compare the diagnostic validities of CARS and ABC for suspected ASD, as well as to obtain more updated and appropriate cutoff scores for each assessment scale. For the definition of suspected ASD, we used the CABS as a screening tool with a cutoff score of 14[8]. A receiver operating characteristic curve was used to compare the diagnostic validities of CARS and ABC, as well as the corresponding cutoff determinations. Our present findings provide insights into the usage of optimal assessment scales for suspected ASD in Chinese mental health hospitals.

MATERIALS AND METHODS

Participants

A total of 591 outpatient children from the ASD Unit at Beijing Children’s Hospital between June and November 2019 were identified. First, they were initially screened with CABS. The cutoff point of 14 for CABS was always used as the criterion for suspected ASD. Based on these criteria, a total of 117 outpatient children were excluded and 474 were identified as suspected ASD. The total sample size included 407 boys and 67 girls, aged between 18 mo and 14 years (4.1 ± 1.93). Then, each suspected ASD was evaluated by CARS and ABC. Parents filled in the ABC scale. After filling in all of them, the specialist gave the due load score for the items that were answered “yes” according to the provisions of the scale. Then, CARS was assessed by two specialists. Prior to the study, two specialists conducted studies and consistency training on evaluation. Second, the DSM-5 was used to confirm the diagnosis of ASD via more than two attending physicians. They diagnosed or excluded autism based on the parents’ detailed description of the child’s development history, observed behavior, and the DSM-5 criteria for diagnosing autism in children. A total of 399 children were diagnosed with ASD (Figure 1) (a total of 75 suspected ASDs did not meet the DSM-5 criteria).

Figure 1 Flowchart of the recruitment of participants in the present study. ASD: Autism spectrum disorder; CABS: Clancy autism behavior scale; ABC: Autism behavior checklist; CARS: Childhood autism rating scale; DSM-5: Diagnostic and statistical manual of mental disorders, fifth edition.

Assessment scales

The CARS is one of the most widely used autism assessment scales[10]. Several studies assessed the internal consistency of the CARS by measuring Cronbach’s alpha, which resulted in a range of 0.82 to 0.95[11]. Park and Kim[12] investigated the construct validity of the CARS in the context of DSM-5 criteria and found that the two-factor model had good fit indices. It is suitable for children over 18 mo old and exhibits good reliability and validity. The CARS is a clinician-rated questionnaire with four frequency levels from 1 to 4 based on observations of individuals and their corresponding information, such as teacher and/or parents reports[13]. The CARS is a behavioral rating scale, consisting of 15 items, that is invariably used to quantitatively describe the severity of suspected ASD symptoms[14]. According to the CARS manual, ASD is defined as a CARS score of ≥ 30 points. A score of 30 or more strongly indicates the existence of ASD. A score of 30-36 suggests mild symptoms, whereas a score of 37 or above suggests moderate to severe ASD[15].

ABC is a well-established assessment scale for screening and diagnosing ASD, and been successfully used in the differential diagnosis of ASD. There was a preliminary study on the validity of the ABC in 2005. The results showed that ABC was effective in differentiating children with autism from children with language disorders and those without complaints[16]. In addition, Yousefi et al[17] evaluated the psychometric features of the Persian version of ABC and found that the internal consistency was 0.73; they also verified the instrument’s concurrent validity with the Gilliam Autism Rating Scale and the correlation between total scores was 0.94. The ABC scale contains 57 items segmented into five categories: Social and self-help, body and object use, relating, language, and sensory features[18]. Based on the degree of association with pathological behavior, each item is rated four frequency levels from 1 to 4. Calculation of the scores for each of the five domains yields the partial and overall scores for each domain[19]. Based on the sum of these scores, severe behavioral characteristics can then be analyzed. Higher scores indicate more autistic behavioral symptoms. In the present study, we used 68 as the ABC cutoff score since this value has been previously recommended[20].

In addition, there were some studies on the application of CABS[4,21-24]. The results all showed that CABS was highly sensitive to screening autism and autism tendencies. Therefore, CABS is the most commonly used screening tool on the Chinese mainland[24]. For this assessment scale, parents completed the Chinese version of the CABS, which is based on its first edition in 1969[24]. A total of 14 items are included, each of which has three frequency levels: “Never’’ (score of 0), ‘‘Occasionally’’ (score of 1), and ‘‘Frequently’’ (score of 2)[8]. In the present study, any participant with a total CABS score ≥ 14 was identified as a suspected ASD case.

Procedure

Given that the ABC and CARS were developed in English, we needed to translate these two scales. First, permission to translate and evaluate the psychometric features of the CARS and ABC was obtained from the publisher of the instrument. The original version of the profile was translated into Chinese according to the International Quality of Life Assessment approach. First, the two scales were translated into the Chinese language by two independent Chinese professionals familiar with special education. The forward translations were compared and discussed in a group meeting of the two translators and two of the authors. Differences were discussed until consensus was reached about the final Chinese version. Then, to examine the equivalence of this translated version with the original version, back-translation to English was performed by a Chinese-English bilingual professional. Third, a committee of 10 professionals including six speech and language pathologists and four child psychiatrists were asked to confirm the validity of the translation and revise the Chinese version.

Statistical analysis

The present study used the statistical package, MedCalc 19.0, for all statistical analyses. We primarily used receiver operating characteristic (ROC)[25] curve analysis to determine the best cutoff values for CARS and ABC and to evaluate the sensitivities, specificities, and accuracies of CARS and ABC[10]. ROC curve analysis was also used to compare diagnostic validations. We also calculated the area under the curve (AUC) for both CARS and ABC. Larger AUCs were indicative of improved prediction efficacies. Each cutoff point and its corresponding sensitivity and specificity were also calculated. A P value < 0.05 was considered to be statistically significant.

Ethical approval

The ethics committees of Capital Medical University and Beijing Children's Hospital authorized the protocols used in the present study. The institutional review board number is 2019-k-396. All of the guardians of the participants offered written informed consent.

RESULTS

Table 1 presents our assessments of ABC and CARS for suspected ASD. The mean and standard deviation (SD) of CARS total scores were 35.72 and 4.10, respectively, while the mean and SD of ABC total scores were 70.05 and 1.19, respectively. According to the results of t tests (both P > 0.05), there were no significant differences in CARS or ABC scores between male and female participants. The skewness coefficient and kurtosis coefficient of CARS were 0.99 and 1.39, respectively. In contrast, the skewness coefficient and kurtosis coefficient of ABC were -0.04 and 0.39, respectively.

Table 1 The description of autism behavior checklist and childhood autism rating scale.

Variables	Mean	SD	Kurtosis	Skewness	Cronbach's α	AUC	AUC (95%CI)
ABC	70.05	1.19	0.39	-0.04	0.426	0.768	0.727-0.805
CARS	35.72	4.10	1.39	0.99	0.772	0.846	0.810-0.877

SD: Standard deviation; ABC: Autism behavior checklist; CARS: Childhood autism rating scale; AUC: Area under curve; CI: Confidence interval.

The most commonly applied measure of scale reliability is the Cronbach's alpha coefficient (α), originally developed by Cronbach (1951), which is used for estimating internal consistency[26]. For this coefficient, larger α values (namely those greater than 0.7) are indicative of higher reliability. We found that the Cronbach alpha coefficients of CARS and ABC were 0.772 and 0.426, respectively (Table 1). Therefore, the reliability of the CARS was higher than that of the ABC. In addition, we found that the correlation between CARS and CABS was 0.732.

Next, we performed ROC curve analysis for CARS and ABC, which yielded AUC values of 0.846 and 0.768, respectively (Figure 2). Notably, ROC curves (AUCs) represent the most commonly applied global index of diagnostic accuracy. The diagnostic capacity of an assessment tool is usually not evaluated by a single number but is instead usually assessed via two or more diagnostic procedures[27]. Diagnosis is generally based on a cutoff or threshold value[28]. It is often recommended that the Youden index be used to define the best cutoff point. The cutoff value, which is associated with the maximum of the Youden index, is usually applied as a decision threshold[29]. Table 2 shows the cutoff scores for ABC and CARS with their corresponding sensitivity and specificity values. The results showed that the differences in AUC values and specificities between CARS and ABC were statistically significant (P < 0.05). The false-positive rate (1-specificity) was indicative of a lower misdiagnosis rate[27]. We found that the cutoff values of CARS and ABC were 34 and 67, respectively. For more details see Table 2 and Figure 2.

Figure 2 Receiver operating characteristic curves of autism behavior checklist and childhood autism rating scale. ABC: Autism behavior checklist; CARS: Childhood autism rating scale.

Table 2 The cutoff points and corresponding sensitivity and specificity of autism behavior checklist and childhood autism rating scale.

Criterion	Sensitivity	95%CI	Specificity	95%CI	+LR	-LR
ABC
> 62	80.45	76.2-84.2	58.67	46.7-69.9	1.95	0.33
> 63	77.94	73.6-81.9	61.33	49.4-72.4	2.02	0.36
> 64	76.19	71.7-80.3	65.33	53.5-76.0	2.2	0.36
> 65	73.68	69.1-77.9	68	56.2-78.3	2.3	0.39
> 66	72.18	67.5-76.5	69.33	57.6-79.5	2.35	0.4
> 67	68.17	63.4-72.7	76	64.7-85.1	2.84	0.42
> 68	63.41	58.5-68.1	80	69.2-88.4	3.17	0.46
> 69	58.9	53.9-63.8	82.67	72.2-90.4	3.4	0.5
> 72	48.12	43.1-53.1	82.67	72.2-90.4	2.78	0.63
> 73	44.36	39.4-49.4	84	73.7-91.4	2.77	0.66
> 74	41.1	36.2-46.1	85.33	75.3-92.4	2.8	0.69
CARS
> 30	92.98	90.0-95.3	40	28.9-52.0	1.55	0.18
> 31	85.46	81.6-88.8	60	48.0-71.1	2.14	0.24
> 32	78.7	74.3-82.6	76	64.7-85.1	3.28	0.28
> 33	68.42	63.6-73.0	90.67	81.7-96.2	7.33	0.35
> 34	57.64	52.6-62.5	94.67	86.9-98.5	10.81	0.45
> 35	33.33	28.7-38.2	97.33	90.7-99.7	12.5	0.68
> 36	29.07	24.7-33.8	98.67	92.8-100.0	21.8	0.72
> 37	1	0.3-2.5	98.67	92.8-100.0	0.75	1

ABC: Autism behavior checklist; CARS: Childhood autism rating scale; CI: Confidence interval; +LR: Positive likelihood ratio; -LR: Negative likelihood ratio.

The negative predictive values (NPVs) and positive predictive values (PPVs) of CARS and ABC are shown in Table 3. The PPV for ASD of a screening test is defined as the proportion of children screened as positive who received an ASD diagnosis divided by the total number of screen-positive cases. PPVs and NPVs are affected by the specificity and sensitivity of the screening tool, as well as by the baseline prevalence of ASD in the population being screened[3]. Moreover, we performed a chi-square test on the PPV and NPV values of ABC and CARS, which revealed that there was no significant difference identified between CARS and ABC.

Table 3 The positive predictive value and negative predictive value for autism behavior checklist and childhood autism rating scale.

Variables	ABC	CARS	χ2 value	P value
PPV	93%	95%	1.048	0.306
NPV	32%	40%	2.243	0.134

ABC: Autism behavior checklist; CARS: Childhood autism rating scale; PPV: Positive predictive value; NPV: Negative predictive value.

Based on these results, we suggest the diagnostic procedures for suspected ASD was as follow Figure 3.

Figure 3 The suggested diagnostic procedures of suspected autism spectrum disorder. ABC: Autism behavior checklist; CABS: Clancy autism behavior scale; CARS: Childhood autism rating scale; ASD: Autism spectrum disorder; DSM-5: Diagnostic and statistical manual of mental disorders, fifth edition.

DISCUSSION

In this study, we found that the AUC of CARS was larger than that of ABC. This finding suggests that the CARS is better than the ABC in terms of its diagnostic validity for suspected ASD. We also found that the cutoff scores of the CARS and ABC for suspected ASD were 34 and 67, respectively. Sensitivity and specificity values included in criterion-validity measures are known to be particularly helpful in clinical settings[30]. The results of a t test on the specificities between these two assessments also revealed a significant difference, indicating that the specificity of the CARS was higher than that of ABC. Furthermore, we verified that the Cronbach alpha coefficient of CARS was 0.772, while that of ABC was 0.426. This finding suggests that the CARS may be more suitable for diagnosing suspected ASD.

Early diagnosis of ASD plays an important role in the intervention and rehabilitation. However, as the etiology of ASD is not clear, it is difficult to make diagnosis based on biochemical indicators at present. The CARS is one of the most important tools for the assessment of ASD, such that both clinical and research practices often use it[31]. Recently, CARS-2 was exploited based on the original edition of the CARS[32]. CARS-2 (normalized form) is the same as original the CARS, whereas CARS-2-HF (high-functioning form) is a newly developed optional diagnostic for evaluating ASD in children over a certain age and with intelligence quotient (IQ) scores above 80[11]. In this study, we revisited the validation of the CARS and found that it functioned as a better diagnostic than ABC. We also identified an updated cut-off score of the CARS for its further usage in diagnosing suspected ASD.

One of the advantages of our study is the introduction of the concept of suspected ASD, which differs from concepts offered in previous studies. In China, there is an increasing number of suspected ASDs that have been identified at community health-service centers and preschools[24]. It has been reported that early diagnosis plays a critical role in improving the outcomes of ASD[33]. In this context, preliminary screening tools are a critical step for the timely diagnosis and intervention of ASD[34]. As a preliminary screening tool, CABS can help childcare physicians, teachers, and parents to quickly screen children with suspected autism[8].

Moreover, most children with suspected ASD require further assessments, such as via ABC and/or the CARS. Based on the results of our present study, we suggest that the CARS may be sufficient for further assessment of suspected ASD.

Previous studies have suggested that the cutoff scores of the CARS and ABC for distinguishing autism and non autism are 30 and 68, respectively[17]. However, for patients with suspected ASD, it has been suggested that these previously proposed cutoff values may no longer be accurate. Based on the results of the present study, we suggest a new cutoff value of the CARS (namely, a score of 34) for the diagnosis of suspected ASD. Based on our present findings, we suggest that children with suspected ASD be initially screened via CABS and that any suspected cases be further confirmed via CARS.

Based on clinically suspected children with ASD in the present study, we found that the diagnostic validation of CRAS was better than that of ABC. Although previous studies have confirmed the strength of the CARS, the sample sizes have been limited[8]. In the present study, we confirmed that the CARS may be more suitable than ABC for diagnosing ASD in China, especially for suspected ASD[12]. However, there are few qualified physicians after receiving training in this examination method in China. We need a scale that is relatively simple and easy to operate to quickly screen suspected autistic patients.

It should be noted that the only available means of ASD diagnosis are behavioral assessments rather than blood tests or noninvasive assessments[35]. Furthermore, to conduct the most comprehensive evaluation of ASD, different measurement tools are required in different assessment environments. The CARS is a valid and reliable assessment tool that is used for the diagnosis and screening of ASD in a number of countries[5]. As mentioned above, the main purpose of this study was to explore the diagnostic validation of the CARS in a large Chinese sample. Our results further confirmed that the CARS can effectively and efficiently diagnose patients with suspected ASD. Therefore, to comprehensively evaluate ASD, we recommend the combined use of the CABS and CARS, which might improve the efficiency of clinical work in hospitals.

Three specific limitations needed to be addressed. First, the adult ASD group was not included in this study, and future studies should clarify the diagnostic validation of ABC and CARS in different age groups. Second, although a total of 474 outpatients were included in this study, the sample was still small. A large sample of ASD is needed to confirm these results in future studies. Third, CARS-2 has been well developed[36], but there is currently no Chinese version of CARS-2. More new tools for the assessments of ASD in China are needed, especially the original tools designed by a Chinese researcher in a Chinese setting.

CONCLUSION

This study demonstrated that the CARS was superior to the ABC in terms of its diagnostic validity in assessing suspected ASD cases in children. In the clinical evaluation for suspected ASD, our findings suggest that the cutoff values of CARS and ABC were 34 and 67, respectively. Based on our results, we recommend that the CARS could be used for assessments of suspected ASD cases in Chinese hospitals.

ARTICLE HIGHLIGHTS

Research background

Autism is the most common clinical developmental disorder in children. The childhood autism rating scale (CARS) and autism behavior checklist (ABC) are the most commonly used assessment scales for diagnosing autism. However, the diagnostic validations and the corresponding cutoffs for CARS and ABC in individuals with suspected autism spectrum disorder (ASD) remain unclear. Furthermore, for suspected ASD in China, it remains unclear whether CARS is a better diagnostic tool than ABC. Also unclear is whether the current cutoff points for ABC and CARS are suitable for the accurate diagnosis of ASD.

Research motivation

According to previous studies on various assessments of ASD, CARS exhibits better diagnostic validation than ABC. However, the diagnostic validations and the corresponding cutoff values for CARS and ABC on individuals with suspected ASD remain unclear. Furthermore, for suspected ASD in China, it remains unclear whether CARS is a better diagnostic tool than ABC. Furthermore, it is unclear whether the current cutoff points for ABC and CARS are suitable for the accurate diagnosis of ASD.

Research objectives

The purpose of this study was to compare the diagnostic validities of CARS and ABC for suspected ASD, as well as to obtain more updated and appropriate cutoff scores for each assessment scale. Our present findings provide insights into the usage of optimal assessment scales for suspected ASD in Chinese mental health hospitals.

Research methods

A total of 591 outpatient children from the ASD Unit at Beijing Children’s Hospital between June and November of 2019 were identified. First, the CABS was used to screen out suspected autism from these children. Then, each suspected ASD was evaluated by CARS and ABC. Receiver operating characteristic curve analysis was used to compare diagnostic validations. We also calculated the area under the curve for both CARS and ABC.

Research results

In this study, we found that the CARS is better than the ABC in terms of its diagnostic validity for suspected ASD. Furthermore, we verified that the diagnostic reliability of the CARS is better than the ABC in terms of the Cronbach alpha coefficient for suspected ASD. We also found that the cutoff scores of the CARS and ABC for suspected ASD were 34 and 67, respectively. These findings suggest that the CARS may be more suitable for diagnosing suspected ASD. However, there are three specific limitations were need to be addressed. First, the adult ASD group was not included in this study, and future studies should clarify the diagnostic validation of ABC and CARS in different age groups. Second, although a total of 474 outpatients were included in this study, the sample was still small. A large sample of ASD is needed to confirm these results in future studies. Third, CARS-2 has been well developed, but there is currently no Chinese version of CARS-2. More new tools for the assessments of ASD in China are needed, especially the original tools which designed by Chinese researcher in a Chinese setting.

Research conclusions

This study demonstrated that the CARS was superior to the ABC in terms of its diagnostic validity in assessing suspected ASD cases in children. In the clinical evaluation for suspected ASD, our findings suggest that the cutoff values of CARS and ABC were 34 and 67, respectively. Based on our results, we recommend that the CARS could be used for assessments of suspected ASD cases in Chinese hospitals.

Research perspectives

First, future studies should clarify the diagnostic validation of ABC and CARS in different age groups as the adult ASD group was not included in this study. Furthermore, CARS-2 (normalized form) is the same as the original CARS, whereas CARS-2-HF (high-functioning form) is a newly developed optional diagnostic for evaluating ASD in children over a certain age and with IQ scores above 80. We can introduce and verify the reliability and validity of CARS-2 for its further usage in diagnosing suspected ASD in China. More new tools for the assessments of ASD in China are needed, especially the original tools which designed by Chinese researcher in a Chinese setting.