Development of the Documentation and Appraisal Review Tool for systematic reviews

Table 2 Documentation and Appraisal Review Tool for systematic reviews

Title of Systematic Review:
Author:
Publication date:		Article tracking number:
Reviewer:		Date completed:
1 Did the authors develop the research question(s) and inclusion/exclusion criteria before conducting the review?				Use this space to document the rationale for your answer
a	It was clear the authors developed the research question(s) and inclusion criteria before conducting the review and that they stated the question(s) clearly	Yes
b	Not described or cannot tell	No
2 Did the authors describe the search methods used to find evidence (original research) on the primary question(s)?				Use this space to document the rationale for your answer
a	Key words and/or MESH terms were stated and where feasible the search strategy was provided	Yes
b	Not described or cannot tell	No
3 Was the search for the evidence reasonably comprehensive? Were the following included?				Use this space to document the rationale for your answer
a	Search included at least two electronic sources	Yes	No
b	Authors chose the most applicable electronic databases (e.g., CINAHL for nursing journals, EMBASE for pharmaceutical journals, and MEDLINE for general, comprehensive search) and only limited search by date when performing an update of a previous systematic review	Yes	No
c	Search methods are likely to capture all relevant studies (e.g., includes languages other than English; gray literature such as conference proceedings, dissertations, theses, clinical trials registries and other reports) and authors hand-searched journals or reference lists to identify published studies which were not electronically available	Yes	No
4 Did the authors do the following when selecting studies for the review?				Use this space to document the rationale for your answer
a	Provide in the inclusion criteria: population, intervention, outcome and study design?	Yes	No
b	State whether the selection criteria were applied independently by more than one person?	Yes	No
c	State how disagreements were resolved during study selection?	Yes	No
d	Provide a flowchart or descriptive summary of the included and excluded studies?	Yes	No
e	Include all study designs appropriate for the research questions posed?	Yes	No
5 Were the characteristics of the included studies provided? (in an aggregated form such as a table, data from the original studies were provided on the participants, interventions and outcomes)				Use this space to document the rationale for your answer
a	Yes
b	Partially
c	No
6 Did the authors make any statements about assessing for publication bias?				Use this space to document the rationale for your answer
a	The authors did assess for publication bias and if publication bias was detected they stated how it was handled	Yes
b	The authors did assess for publication bias but did not state how it was handled if it was detected	Partially
c	Not described or cannot tell	No
7 Did the authors do the following to assess the overall quality of the individual studies included in the review?				Use this space to document the rationale for your answer
a	Was the quality assessment specified with adequate detail to permit replication?	Yes	No
b	Was the quality assessment conducted independently by more than one person?	Yes	No
c	Did the authors state how disagreements were resolved during the quality assessment?	Yes	No
8 Did the authors appropriately assess for quality by appropriately examining the following sources of bias in all of the included studies?				Use this space to document the rationale for your answer
All studies:
a	Confounding (assessed comparability of study groups at start of study, was randomization successful?)	Yes	No
b	Sufficient sample size (only applicable to studies that summarize their results in a qualitative manner; it's not a concern for pooled results)	Yes	No
c	Outcome reporting bias (assessed for each outcome reported using a system such as the ORBIT classification system)	Yes	No
d	Follow up (assessed for completeness and any differential loss to follow-up)	Yes	No
For Randomized Controlled Trials only:
e	Randomization	Yes	No
f	Allocation concealment	Yes	No
g	Blinding	Yes	No
For Case-Control and Cohort Studies only:
h	Selection bias	Yes	No
i	Information bias--recall and completeness to follow-up	Yes	No
For Quasi-Experimental Studies only:
j	Differences between the first and second study measurement point - such as changes or improvements in other interventions, changes in measurement techniques or definitions, or aging of subjects	Yes	No
k	Selection bias	Yes	No
For Diagnostic Accuracy Studies only:
l	Selection (spectrum) bias - were subjects selected to be representative of patients to whom the test will be applied in clinical practice, and to represent the broadest spectrum of disease?	Yes	No
m	Verification bias - were all patients subjected to the same reference standard of diagnosis, and was it measured blindly and independently of the test?	Yes	No
9 Did the authors use appropriate methods to extract data from the included studies?				Use this space to document the rationale for your answer
a	Were standard forms developed and piloted prior to the systematic review conduct?	Yes	No
b	Did the authors ensure that data from the same study but that appeared in multiple publications were counted only once in the synthesis?	Yes	No
c	Was data extraction performed by more than one person?	Yes	No
10 Did the authors assess and account for heterogeneity (differences in participants, interventions, outcomes, trial design, quality or treatment effects) among the studies selected for the review?				Use this space to document the rationale for your answer
a	The authors stated the differences among the studies and how they accounted for those differences	Yes
b	The authors stated the differences but not how they accounted for them	Partially
c	Not described or cannot tell	No
11 Did the authors describe the methods they used to combine/synthesize the results of the relevant studies (to reach a conclusion) and were the methods used appropriate for the review question(s)?				Use this space to document the rationale for your answer
a	Methods were reported clearly enough to allow for replication. The overview included some assessment of the qualitative and quantitative heterogeneity of the study results and the results were appropriately combined/synthesized. For meta-analyses, an accepted pooling method (i.e., more than simple addition) was used. Or the authors state that the evidence is conflicting and that they can't combine/synthesize the results	Yes
b	The methods were reported clearly enough to allow for replication but they were not combined appropriately	Partially
c	Not described or cannot tell	No
12 Did the authors perform sensitivity analyses on any changes in protocol, assumptions, and study selection? (For example, using sensitivity analysis to compare results from fixed effects and random effects models)				Use this space to document the rationale for your answer
a	Sensitivity analyses were used when appropriate on all changes in a priori design	Yes
b	Sensitivity analyses were only used on some changes in a priori design	Partially
c	Not described or cannot tell	No
13 Are the conclusions of the authors supported by the reported data with consideration of the overall quality of that data?				Use this space to document the rationale for your answer
a	The conclusions are supported by the reported data and reflect both the scientific quality of the studies and the risk of bias in the data obtained from those studies	Yes
b	The authors failed to consider study quality and/or their conclusions were not supported by the data, or cannot tell	No
14 Were conflicts of interest stated and were individuals excluded from the review if they reported substantial financial and intellectual COIs?				Use this space to document the rationale for your answer
a	COIs were reported for each team member and individuals were excluded if they had substantial COIs	Yes
b	COIs were reported but it was not clear whether individuals were excluded based on their COIs	Partially
c	COIs were not reported and individuals were not excluded based on their COIs	No
15 On a scale of 1-10, how would you judge the overall quality of the paper?
Rating	Overall Comments
Good (8-10)
Fair (5-7)
Poor (< 5)

COIs: Conflicts of interests.