The challenge of publishing preprints

Study assesses the responsibilities and shortcomings of media coverage of papers that have not been peer-reviewed

The use of preprints to disseminate research results made the news for the worst possible reasons during the COVID-19 pandemic. Before explaining why, it should be noted that preprints are scientific studies shared on open-access platforms as soon as they have been written, prior to undergoing peer review. As a result of these characteristics, poorly conducted studies defending the use of chloroquine and other drugs to treat COVID-19—as if expressing absolute truths—made headlines amid the global health crisis.

In a strictly technical setting, such papers would be quickly discarded due to their methodological flaws. However, by taking advantage of preprint platforms, the authors were able to provide ammunition to the denialist tactics adopted by many politicians and public authorities.

A study published in the journal PLOS One analyzed the use of preprints by journalists during emergencies, which the research group classifies as periods of post-normal science. “The term describes complex and high-risk situations characterized by uncertain facts, an urgent need for political decision-making, and disputes about values, rather than just about scientific evidence,” explains science and health communication expert Alice Fleerackers, a professor of interdisciplinary studies at Simon Fraser University in Canada and one of the authors of the paper.

In an interview with Science Arena, Fleerackers says the early response to COVID-19 was a good example of post-normal science. “Decision-makers needed to choose important directions on how to control the spread of a then-mysterious new virus,” says Fleerackers. “This was done based on scientific evidence being produced in the moment, which was constantly evolving, and was often only initially available in preprints.”

Such decisions, emphasizes Fleerackers, were taken in the midst of political disputes linked to values, such as discussions about whether it was morally acceptable to impose mandatory vaccinations or to close businesses for the public’s protection.

The qualitative analysis led by the researcher involved interviews with 19 health and science journalists who published articles in The New York Times, The Guardian, Wired, Popular Science, HealthDay, IFL Science, MedPage Today, and News Medical between March 1 and April 30, 2021.

According to Fleerackers, the group noticed that journalists seem to adopt new practices and norms for communicating science in “post-normal” situations, and the use of preprints is an example of this.

At the same time, the research led by Fleerackers highlights some of the obstacles faced by media professionals covering health and science based on preprints. Many lack the expertise or time needed to fully understand and assess the relevance of a study.

An unavoidable question is: does this limitation in the media apply to any and all scientific publications, including scientific articles that have undergone peer review?

“In fact, it is still unclear whether journalists are able to verify the quality of scientific research, whether it’s been peer-reviewed or not,” admits Fleerackers.

“In general, journalists trust the credibility of peer review, which is seen as a form of validation by the scientific community. They therefore believe that articles submitted to peer review have been put to the test by other scientists,” says Fleerackers. “In practice, however, we know very little about whether peer review is really an effective quality control mechanism.”

Preprints will likely continue to be covered in the media

In the researcher’s opinion, this explains why preprints are so interesting. Without this verification by external peers, journalists need to use other strategies to assess the relevance of the results and conclusions shared on preprint repositories. The fact that it is so challenging raises questions about how capable journalists are of gauging the quality of scientific papers, particularly in a media environment that puts them under pressure to produce high volumes of news reports in extremely short timeframes.

“I would love to see more support for science journalists, such as better training and more time and resources, to give them the conditions needed to write and talk about science well, independent of peer review,” says Fleerackers.

The study published by the researcher and her team suggests that preprints will likely continue to be covered in the media, even now that the end of the pandemic has been announced. Fleerackers believes this could have important implications for scientists, journalists, and the public who read their work. This is not a recommendation, stresses the researcher, but an observation.

“While some of the journalists we interviewed are hesitant to use preprints outside the context of the pandemic, others told us that they plan to continue using this type of science publication,” she says.

Because preprints make newly discovered results freely and immediately available, journalists feel like they are following science in motion, almost in real time, and this gives preprints an air of novelty—like the latest news so desired by journalists.

It can take months or even more than a year for articles submitted to journals that conduct a peer review to be published due to the lengthy assessment process.

“For some, open access and getting a glimpse of science in motion rather than finding things out months and months after the research was first conducted is very appealing.” For others, adds Fleerackers, the perception is that preprints have become part of their daily routine—the new normal.

“Whether reporting on preprints outside of emergency situations is a good or bad thing remains to be seen. It is more important, from my perspective, to ensure that journalists have the support they need to cover preprints accurately.”

You can never be too careful

The results of the analysis serve as a reminder that all science is provisional. “We tend to think of scientific discoveries as facts or absolute truths, and this is often how peer-reviewed research is reported in the media or taught at universities,” says Fleerackers. But scientific activity, she says, is a continuous and iterative process on which new studies are based and problematized, sometimes even contradicting previous findings.

To manage the risks associated with the provisional nature of science, Fleerackers recommends that journalists be encouraged to make readers aware of uncertainties and put new discoveries into a broader context, taking into account the largest possible volume of evidence, rather than reporting on isolated studies. “These aspects are signals of quality for all science journalism, but they are arguably even more important when reporting on preliminary and emerging findings.”

Before the pandemic, preprints were never a concern because they simply did not interest the external community—whether good or bad, they were not yet accepted. “The famous paper on chloroquine was methodologically flawed and contained serious problems, and if limited to the scientific community it would have been criticized or simply unnoticed, because it was really bad,” points out Peter Schulz, a physics professor at the School of Applied Sciences at the University of Campinas (UNICAMP) and a scholar of science communication.

“However, in the context of the beginning of the pandemic, when the whole world was scared, these open-access preprint repositories became an easy source of scientific findings that the press soon elevated to the status of great discoveries,” says Schulz.

* This article may be republished online under the CC-BY-NC-ND Creative Commons license.
The text must not be edited and the author(s) and source (Science Arena) must be credited.