Primary author contact for systematic reviews of randomized controlled trials: A systematic review

Abstract

BACKGROUND

Systematic reviews (SRs) synthesize and evaluate data, mainly from randomized trials, which then guides the development of clinical recommendations in evidence-based medicine. However, the data and methodological information in the included papers can often be lacking or unclear, and reviewers usually need to contact the authors of included studies for clarifications. Contacting authors is recommended, but it is unclear how often SR teams do it, or what the level of response is.

AIM

To investigate how often reviewers undertake contact with the authors of included randomized controlled trials (RCTs) for clarification on data and risk of bias concerns, to explore the factors that influence whether SR authors contact or do not contact the authors, and the content and level of responses.

METHODS

We conducted a systematic electronic database search in MEDLINE using the search string “(systematic review)” AND “(RCT OR randomized OR trial)” for articles published between 1 January 2024 and 19 February 2024, without language restrictions. Screening and data extraction was done independently by two reviewers, and conflicts resolved by a senior author. Contact authors of included SRs were contacted for clarifications.

RESULTS

Of the 329 included SRs, 38% (n = 125) explicitly mentioned contact with the authors of included studies. The remaining 62% (n = 204) did not. We attempted contact with all SR teams for clarifications and received 90 responses (19.4%). Of the 50 respondents who did not explicitly mention contact in their SRs, 25 (50%) replied that they did make contact. We received a total of 64 responses on the level and content of information sought. The mean ± SD contacts SR teams made were 10 (10), replies received 5 (6.7), and response waiting time 10.1 (28.3) weeks. Resources, time, poor previous experience, perceived likelihood of poor response and bias concerns were reported as barriers to attempting contact.

CONCLUSION

The majority of SRs published in 2024 did not confirm seeking clarifying or missing information from primary study authors. However, SR teams reported that 50% of contacted primary authors respond. Additional research can clarify this rate of response and establish methods to increase the integration of this core methodological element in SRs.

Key Words: Systematic review; Methodology; Contacting authors; Interventions; Randomized control trials

Core Tip: We found that a majority of systematic review teams do not seek clarifying or missing information from primary study authors. Time and resources are seen as a barrier, however, we found that almost 50% of contacted primary authors were reported to respond. Contacting authors should be seen as a core methodological requirement for systematic reviewers, and further steps should be taken to investigate and promote it.

INTRODUCTION

In clinical medicine, evidence-based medicine has been the cornerstone of decision making for more than three decades, underpinned by systematic reviews (SRs) to develop clinical guideline recommendations[1]. Most SRs primarily include randomized controlled trials (RCTs) in their search and evidence synthesis process, as RCTs are the gold standard design for assessing the efficacy of a medical intervention[2]. This is derived from the reduced risk of bias inherent in the methods of an RCT.

There is substantial evidence that completeness and quality of methods information is still sporadically reported, even in high impact journals[3]. There is also evidence that authors often release different forms of data over time meaning that SR authors may have questions that need clarifying or data that needs confirming[4]. Missing data, due to inadequate reporting of summary statistics or overall findings when results are unfavourable or null, is also a challenge and represents a significant source of bias. This can be observed in situations where the authors may hold the notion that certain results do not add value to their publication or limit impact, or there are pressures to deliver positive results due to commercial demands[5]. Incomplete data that do not make it into SRs can result into healthcare decision makers making less-than-ideal choices for treatment protocols, resource allocation, and overall care, potentially resulting in poorer patient outcomes[6].

When conducting a SR, contacting the authors of eligible and included RCTs to clarify material or provide additional recorded data that may be missing is suggested[7]. This can be beneficial for SRs, as they are able to more clearly discern the risk of bias in each RCT and include RCTs that they would otherwise have had to discount. Their results would then be of a more representative sample and draw wider conclusions in their meta-analysis. Nonetheless, reviewers applying this guidance face challenges in identifying the authors of RCTs, frequently experience RCT authors failing to reply, and struggle with summarizing the process in their own SR. Missing info about quality appraisal can also be common.

The aim of this study was to investigate how often systematic reviewers undertake contact with the authors of included papers for clarification on data and risk of bias concerns. We explored the factors that influence whether SR authors contact or do not contact the authors of the included studies, and the content and level of response to the communication. We also explored variables including impact factor of journal and available funding on whether author contact was initiated or not.

MATERIALS AND METHODS

A protocol for this review was deposited in the online repository for the University of Central Lancashire prior to the initiation of this review[8].

This work was exempt from ethical approval as there were no ethics issues related to it. Only corresponding authors who have made their contact information publicly available for the purpose of being contacted about their publications were contacted, solely with questions about their publications.

The report of this review followed the preferred reporting items for SRs and meta-analyses reporting guidelines.

Literature search

We performed a systematic electronic database search of all the RCTs published between 1 January 2024 and 19 February 2024 from the single database MEDLINE on 19 February 2024. The search strategy was “systematic review” (Title/Abstract) AND [“RCT” (Title/Abstract) OR “randomized” (Title/Abstract) OR “trial” (Title/Abstract)]. All retrieved citations from this search were imported into Covidence and results de-duplicated.

Study selection

Two authors (Shah E and Tony-Jimmy TE) independently screened all titles and abstracts, discarding those not meeting inclusion criteria. Disagreements were resolved by discussion and consensus in the presence of a third and fourth author (Sinopoulou V and Gordon M). All SRs selected for full-text screening were downloaded and independently reviewed by two authors (Shah E and Tony-Jimmy TE) to confirm whether papers met the inclusion criteria. All differences were resolved by Gordon M and Sinopoulou V, in a similar manner.

Inclusion criteria for full-text SR manuscripts were: (1) Full-text SR manuscripts of RCTs involving human participants; (2) Full-text SR manuscripts of RCT with interventions for the management of symptoms, involving any pharmacological or non-pharmacological intervention compared to any other intervention, placebo, no treatment, or usual care; and (3) Full-text SR manuscripts with outcome measures that directly impact patient health or risk to health. SRs that included any phase RCTs were eligible. There were no limitations on language or region.

Exclusion criteria were: (1) Manuscripts which reported outcome data of non-randomized or quasi-randomized trials; (2) Manuscripts that reported on non-medical interventions such as service evaluation, delivery, safety, and education trials; (3) Manuscripts on in-vitro interventions; and (4) Manuscripts without outcome results (e.g. protocols, trial registrations).

Outcomes

The outcomes for this review were: (1) Whether contact with the included studies’ authors was initiated or not; (2) Number of contacts initiated, and number of responses received when contact was initiated; (3) Time given to primary authors to respond; (4) Type of information requested when author contact was initiated; (5) Factors that prevented reviewers from contacting authors; and (6) Whether impact factor (higher or lower than 5) and funding (industry, publicly funded or not reported) are correlated with whether author contact was initiated or not.

Data extraction

Two authors (Shah E and Tony-Jimmy TE) independently extracted data using a predesigned extraction form and disagreements were resolved by a third and fourth reviewer (Sinopoulou V and Gordon M).

The extracted data were key characteristics such as year of publication, contact author name and email, journal source, impact factor of SR publication journal, funding sources, DOI; including quotations or any comments made regarding author contact and data for the above outcomes.

Missing information

SR correspondence authors were contacted via email at their listed correspondence email when outcome information was lacking. Authors were given two weeks to reply at which point a reminder was sent if no response was received, and an additional two weeks were given for response. Emails were sent and responses received in March 2024.

Statistical analysis

We calculated descriptive statistics as absolute numbers and percentages for all binary outcomes, and mean ± SD for continuous outcomes. We conducted χ² tests to test for differences in relation to impact factor and funding source.

RESULTS

Our search identified 866 unique papers. After abstract screening, 484 SR were excluded with reasoning, leaving 382 SRs that met our inclusion criteria for full text screening. Of those included, 3 SRs were not accessible for full text screening. Thus, 379 SRs were included for full text screening, of which 329 met the inclusion criteria and were included in the review (Figure 1).

Figure 1 Prisma diagram. SR: Systematic review; RCTs: Randomized controlled trials.

No included SRs provided enough information for our data extraction in their text, and we attempted contact with the corresponding authors for all included SRs. The questions we asked were structured on the outcomes of this review as outlined in the methods section.

Was contact with the included studies’ authors initiated or not

Of the 329 full-text SRs examined, 38% (n = 125) explicitly mention contact with the primary authors of included studies as part of their planned protocol and methods. The remaining 62% (n = 204) of the examined SRs did not explicitly mention contact with the primary study authors or that this was a part of their protocol.

Our team contacted the corresponding authors of included SRs via email for further information on the nature of their primary author contact. 7% (n = 23) of the 329 included SRs could not be contacted via email, due to inaccurate, outdated or lack of contact information, or being on leave. Of the 125 SRs that mentioned primary author contact in their papers, 116 could be contacted. From the 116 emails that were successfully sent, 40 replies were received, with 1 reply deemed unclear, yielding a 34% (39/116) response rate from SRs that explicitly mention author contact with primary studies’ authors.

Of the 204 SRs that did not explicitly mention contact with primary study authors in their paper, 14 SR teams were not accessible for email contact, for similar reasons as already outlined. From the 190 emails that were successfully sent, 50 replies were received, with 2 replies deemed unclear, yielding a 25% response rate. After receiving these email responses, 25 of 48 of clear respondents (52%) confirmed they did contact study authors despite not reporting so in their papers.

The total number of SRs with confirmed initiated contact with primary RCT authors out of all included SRs was 64/329 (19.4%), with 61% (n = 39) reporting author contact in their paper, and 39% (n = 25) not reporting author contact in their paper (Figure 2).

Figure 2  Graphical illustration of included systematic reviews teams’ responses.

Number of contacts initiated, number of responses received when contact was initiated and time given to primary authors to respond

Of the 64 SRs that confirmed author contact, 37 SR authors provided details on how many primary authors they contacted and the number of replies they received. The mean ± SD contacts SR authors made were 10 (10) RCT authors, and the mean ± SD replies were received 5 (6.7). The mean ± SD response rate per contact sent was 0.49 (0.45) or 49% (45). The waiting time for responses prior to recording a non-response was a mean ± SD of 10.1 (28.3) weeks with a median (range) of 4 (2-126) weeks (Table 1).

Table 1 Summary of key results.

Summary
Total number of SRs that report author contact in their paper (n = 329), n (%)	125 (38)
Total number of SRs that confirmed author contact but didn’t report it (n = 48), n (%)	25 (52)
The mean ± SD contacts made by authors per SR	10 (10) RCT authors
The mean ± SD response rate for SRs per author contact sent	0.49 (0.45) or 49% (45)
The mean ± SD and median (range) waiting time for responses prior to recording a non-response	10.1 (28.3) weeks and 4 (2-126) weeks

SR: Systematic review; RCTs: Randomized controlled trials.

Type of information requested when author contact was initiated

Of the 64 SR authors who responded with details about the contacts they made, 64% (n = 41) reached out to clarify outcome data; 48% (n = 31) for inclusion/exclusion criteria; 39% (n = 25) for methods, design or risk of bias; and 14% (n = 9) for other reasons (full-text requests, data in corrections).

Factors that prevented reviewers from contacting authors

Of the 23 papers that did not mention RCT author contact in their SR and replied to confirm as such, 61% (n = 14) considered RCT author contact as part of their study design. When not pursuing author contact, 52% (n = 12) of respondents gave reasons related to resources and time; 43% (n = 10) poor previous experience; 26% (n = 6) perceived likelihood of poor response; 21% (n = 5) integrity or bias; and 30% (n = 7) other reasons (no missing data, or enough data was available, or data could be estimated).

Impact factor and funding source

Neither an impact factor higher or lower than five, nor whether SR funding was from industry, public or not reported, was correlated with whether SR authors confirmed they contacted primary study authors, confirmed that they did not, or did not respond to us to confirm either (all P values > 0.05).

DISCUSSION

SRs are a core form of knowledge translation for clinicians and researchers. However, their role in guideline development and international policy is perhaps more impactful. Previous research has clearly identified that large numbers of studies are unclear when it comes to methods, such as to lead to a downgrade in judgements of quality[3], that missing data can be a risk, as it may change as studies are subsequently published in different forms[4] and this in turn can have major impacts on the certainty of subsequent outputs and guidance for the clinical community[9,10].

This is the first study to investigate whether authors of SRs are currently following guidance[7] to mitigate these risks, by routinely attempting to contact randomized controlled trial authors to clarify or supply information.

This review has found that a third of SRs explicitly planned to do this and reported it in their methods, with a further 7.5% reporting they did this but did not record it in their review. This still leaves a vast majority of reviews that did not ever attempt such contact at all, despite the risks and guidance to the contrary.

Responding review teams commented on the reasons for not planning this core element of review work, with some of the reasons being experiential and related to poor response in the past. Other reasons included perceived bias, which is not supported by organizations such as Cochrane[7], and most commonly, time. This final point is interesting. We recognize the significant investment of time and resource needed for SRs as a barrier, but for those already pursuing such works we believe maximining the value, impact and ultimate verisimilitude of findings is so central that the minimal extra effort needed would seem to justify the investment.

In what is also a novel finding, 37 review teams gave details of the effectiveness of their primary author contacts, suggesting 50% of primary authors contacted responded. This is in line with our experience as SR authors and would therefore suggest barriers of resource are perceived but not likely borne out in reality if such practice is integrated.

It is worth commenting that this practice is secondary researchers accounting for primary study issues. Therefore, a better solution would be for trial authors to include such detail, as well as editors and reviewers mandating this to be the case. But as this is a well-recognized and pervasive issue that is likely to continue given the rapid growth in academic output, such mechanisms are vital to enhance the evidence base when synthesized as a whole.

A limitation of this work is the sample size and choice. A large sample has been taken but it has been limited to recent publications and trial-focused interventional reviews. It is possible that these findings are not therefore more generalisable. However, the importance of this context is such that even if it is a limited finding, we believe its importance to still be key.

There are some limitations of this work. While a large sample was obtained, it was limited to recent publications and trial-focused interventional reviews, potentially affecting the generalisability of the findings. Additionally, the responsiveness of authors we contacted for missing information may not accurately reflect whether primary author contact was established. Furthermore, some authors who responded to our initial email did not respond to the follow-up questionnaire or provided only partial responses, possibly due to response fatigue from our two-step email approach. These factors may further impact the overall generalisability of the study. Finally, the responsiveness of authors we contacted for missing information may not accurately reflect whether primary author contact was established, with non-responses further limiting the sample size.

Future research could seek to investigate in greater detail the rate of response from authors, time taken to make such contacts and possibly methods to enhance response or enhance primary publications.

CONCLUSION

Our study provides novel insights on the current state of the field regarding author responses to requests for information on published RCTs. It does highlight how common such issues occur and the need for peer reviewers and journal editors to work to ensure that publications are transparent in such reporting. This study is limited by its focus on recent publications, which may not represent the broader spectrum of SRs. Additionally, the response rate from contacted authors may have been influenced by our specific inquiry approach, and may not reflect general responsiveness. Future research should investigate strategies to enhance primary author response rates in SRs, such as the use of standardized contact protocols or incentives for participation. Additionally, studies should assess the impact of author contact on the risk of bias assessments and the quality of evidence synthesis.