The first results of the Brazilian Reproducibility Initiative are now published in preprint format

The study successfully replicated between 15% and 45% of a sample of biomedical experiments published by Brazilian research groups and mapped the challenges faced by the laboratories participating in the project. Photo: Eduardo Anizelli/Folhapress.

The Brazilian Reproducibility Initiative, a project that helped establish our network, has just published its first results in preprint format. The study, which involved 56 laboratories and 213 researchers across the country, successfully replicated between 15% and 45% of a sample of experiments published by Brazilian groups, depending on the criteria used. Funded by the Serrapilheira Institute, the project took seven years to be completed and represents the world's first national-level estimate of experiment replicability.

The initiative set out to replicate 60 randomly selected experiments from articles published by Brazilian research groups using three traditional biomedical research laboratory methods: the MTT assay, which measures the viability of cultured cells; reverse transcription polymerase chain reaction (RT-PCR), used to analyze the expression of specific genes; and the elevated plus maze, used to assess anxiety-related behavior in rodents. Each experiment was sent for replication by three different laboratories in the consortium.

Of the 180 planned replications, 143 were carried out, with 37 canceled due to difficulties in acquiring supplies or laboratories failing to complete the experiments. However, only 97 were considered valid replications of the original experiment by a validation committee created for the project. The remaining ones were excluded from the main analysis due to issues such as protocol deviations, insufficient sample size, insufficient biological variability between experimental units, or inadequate documentation.

The Initiative's main results include 97 replications of 47 experiments, with each experiment replicated between one and three times. The replication success rate was measured using five pre-established criteria, and resulted in the following results:

a) 45% of the original effect estimates fell within the 95% prediction interval of a meta-analysis of the replications.

b) 26% of the replication effect estimates fell within the 95% confidence interval of the original effect.

c) 19% of replication effect estimates showed a statistically significant aggregated effect (p < 0.05) in the same direction as the original effect.

d) 15% of experiments had at least half of their replications with a statistically significant result in the same direction as the original.

e) 40% of replications had at least half of the replications considered successful by the laboratory conducting them.

An important limitation is that the replications showed much greater variability between experimental units than the original experiments, making several of them statistically underpowered to detect the difference between groups observed in the original experiment. To address this issue, the study conducted various alternative analyses using different filters to select the experiments. When considering only the 27 experiments with adequate statistical power, even when accounting for the variability observed in the replication, the statistical significance rate obtained in the replications rises to 30%.

The relative effect sizes (e.g., ratios between measurements in the treated group and the control group) were, on median, almost 40% smaller in the replications than in the original articles. Conversely, the coefficients of variation (which measure the variability observed within each group) were 2.5 times higher, particularly in in vitro experiments using MTT and RT-PCR techniques. The differences between the original experiment and the replications were about 34% greater than those between individual replications, suggesting that part of the reasons for low replicability are specific to experiments published in the scientific literature. Possible reasons for this include issues such as publication bias and researchers' preference for positive results.

Still, considerable heterogeneity among results was observed across replications, even though the responsible laboratories were blind to the original article and had no incentive to seek specific outcomes. This heterogeneity, combined with the difficulties many laboratories faced in following their own protocols, led the Initiative to conduct a self-assessment process to identify reasons for protocol deviations and opportunities to improve experiment reproducibility.

The results indicate that, in some cases, strictly following pre-established protocols is impossible—either because the biological model behaves differently than expected or due to external conditions such as infrastructure limitations, supply chain issues, or unpredictable factors like the COVID-19 pandemic forcing changes in plans. In other cases, however, non-adherence to protocols resulted from communication failures or misinterpretations, some of which stemmed from the lack of shared terminologies among laboratories to describe aspects of experimental design, such as the experimental unit used in cell culture studies.

For Olavo Amaral, coordinator of the Initiative, this is one of the factors indicating that academic laboratories are poorly equipped to collaborate—largely because the research culture in academia is more focused on small groups conducting exploratory approaches rather than large confirmatory projects like the Initiative. 'We had the idea that bringing many laboratories together would be enough to establish reproducible protocols, but retrospectively, it's like expecting a collection of garage bands to form an orchestra.'

For Amaral, the academic community should structure itself to develop confirmatory projects, either through large consortia like the Initiative or specialized centers dedicated to such projects with a higher methodological rigor. 'We can't expect every scientific project to apply all possible reproducibility practices, as some are quite costly, and this could stifle the exploratory side of science that is also necessary. But it is crucial to increase rigor in selected cases, and our scientific workforce is not well-organized or prepared for this.'

Amaral also highlights that biomedical scientists are rarely trained to manage large projects and are not accustomed to receiving external supervision, both of which are essential for large-scale projects to thrive. Beyond researcher training, simple measures such as adopting consensus terminologies among laboratories to describe experimental units, control groups, and other aspects of experimental design also present opportunities for improving reproducibility.

Thus, the results of the Initiative reinforce one of the central premises behind the Brazilian Reproducibility Network—that improving the reliability of science requires coordinated actions among researchers, institutions, and other stakeholders to promote systemic changes that encourage and reward more rigorous science. Unsurprisingly, the Network may be the Initiative’s most important legacy, helping the project continue with future articles dedicated to deeper analyses of its findings.