Development of the Documentation and Appraisal Review Tool for systematic reviews

Table 3 Comparison of Documentation and Appraisal Review Tool to modified Overview Quality Assessment Questionnaire and Assessment of Multiple Systematic Reviews

DART questions	Corresponding AMSTAR question(s)	Corresponding modified OQAQ question(s)
(1) Did the authors develop the research question(s) and inclusion/exclusion criteria before conducting the review?	(1) Was an "a priori" design provided?	Not addressed
(2) Did the authors describe the search methods used to find evidence (original research) on the primary question(s)?	(3) Was a comprehensive literature search performed?	(1) Were the search methods used to find evidence on the primary question stated?
(2a) Are key words and/or MESH terms stated?	(3) Was a comprehensive literature search performed?	Not addressed
(3) Was the search for the evidence reasonably comprehensive?	(3) Was a comprehensive literature search performed?	(2) Was the search for evidence reasonably comprehensive?
(3a) Does the search include at least 2 databases?	(3) Was a comprehensive literature search performed?	Not addressed
(3b) Did the authors choose the most applicable electronic databases and only limit the search by date when performing an update?	Not addressed	Not addressed
(3c) Are search methods likely to capture all relevant studies and did the authors hand-search journals or reference lists to identify published studies which were not electronically available?	(3) Was a comprehensive literature search performed?	Not addressed
	(4) Was the status of publication (i.e., grey literature) used as an inclusion criterion?
(4a) Did the authors provide in the inclusion criteria: Population, intervention, outcome, and study design, when selecting studies for the review?	Not addressed	Not addressed
(4b) Did the authors state whether the selection criteria were applied by more than one person?1	(2) Was there duplicate study selection and data extraction?1	Not addressed
(4c) Did the authors state how disagreements were resolved during study selection?1	(2) Was there duplicate study selection and data extraction?1	Not addressed
(4d) Did the authors provide a flowchart or descriptive summary of the included and excluded studies?	(5) Was a list of studies (included and excluded) provided?	Not addressed
(4e) Did the authors include all study designs appropriate for the research questions posed?	Not addressed	Not addressed
(5) Were the characteristics of the included studies provided? (in an aggregated form such as a table, data from the original studies were provided on the participants, interventions and outcomes)	(6) Were the characteristics of the included studies provided?	Not addressed
(6) Did the authors make any statements about assessing for publication bias?	(10) Was the likelihood of publication bias assessed?	Not addressed
(7a) Was the quality assessment specified with adequate detail to permit replication?	(7) Was the scientific quality of the included studies assessed and documented?	(5) Were the criteria used for assessing the validity of the included studies reported?
(7b) Was the quality assessment conducted independently by more than one person?	Not addressed	Not addressed
(7c) Did the authors state how disagreements were resolved during the quality assessment?	Not addressed	Not addressed
(8) Did the authors appropriately assess for quality by appropriately examining the following sources of bias in all of the included studies: confounding, sufficient sample size, outcome reporting bias, follow-up, randomization, allocation concealment, blinding, selection bias, information bias, verification bias, and differences between the first and second study measurement point?	(7) Was the scientific quality of the included studies assessed and documented? (partial match)	(6) Was the validity of all studies referred to in the text assessed using appropriate criteria? (partial match)
(9) Did the authors use appropriate methods to extract data from the included studies?	Not addressed	Not addressed
(9a) Were standard forms developed and piloted prior to the systematic review conduct?	Not addressed	Not addressed
(9b) Did the authors ensure that data from the same study that appeared in multiple publications were counted only once in the synthesis?	Not addressed	Not addressed
(9c) Was data extraction performed by more than one person?	(2) Was there duplicate study selection and data extraction?	Not addressed
(10) Did the authors assess and account for heterogeneity (differences in participants, interventions, outcomes, and trial design, quality or treatment effects) among the studies selected for the review?	(9) Were the methods used to combine the findings of studies appropriate?	(7) Were the methods used to combine the findings of the relevant studies reported?
		(8) Were the findings of the relevant studies combined appropriately?
(11) Did the authors describe the methods they used to combine/synthesize the results of the relevant studies (to reach a conclusion) and were the methods used appropriate for the review question(s)?	(9) Were the methods used to combine the findings of studies appropriate?	(7) Were the methods used to combine the findings of the relevant studies reported?
		(8) Were the findings of the relevant studies combined appropriately?
(12) Did the authors perform sensitivity analyses on any changes in protocol, assumptions, and study selection? (For example, using sensitivity analysis to compare results from fixed effects and random effects models)	Not addressed	Not addressed
(13) Are the conclusions of the authors supported by the reported data with consideration of the overall quality of that data?	(8) Was the scientific quality of the included studies used appropriately in formulating conclusions? (partial match)	(9) Were the conclusions made by the author(s) supported by the data reported? (partial match)
(14) Were conflicts of interest stated and were individuals excluded from the review if they reported substantial financial and intellectual COIs?	(11) Was the conflict of interest stated? (partial match)	Not addressed
(15) On a scale of 1-10, how would you judge the overall quality of the paper?	Not addressed	(10) Overall quality

¹Separate questions in DART, but concepts not separated in AMSTAR. DART: Documentation and Appraisal Review Tool; OQAQ: Overview Quality Assessment Questionnaire; AMSTAR: Assessment of Multiple Systematic Reviews; COIs: Conflicts of interests.