Post-Publication Peer Review
It’s not clear to me that any of these issues allowed the Mehra et al. article to slip through the system. The New England Journal of Medicine has no problem identifying qualified peer reviewers. Additionally, it has a large internal staff that vets each article, so conventional peer review serves as only one part of the overall review process.
The hard truth is that peer review was not designed to detect fraud. When I provide peer review for an article, I am not typically repeating the author’s calculations to see if they came up with the right P value. I am fully prepared to give the authors the benefit of the doubt and assume all of their tables and graphs are supported by actual data.
In the case of the Mehra et al. article, it’s almost a fluke the fraud was detected. The dataset was supplied by Surgisphere, founded by vascular surgeon Sapan Desai. This tiny company no one had ever heard of claimed it had a database of records from approximately 700 hospitals on six continents.17 The same dataset was used in a study published in The Lancet on the impact of hydroxychloroquine on COVID-19 outcomes.18 The controversial nature of the topic led to greater-than-normal levels of scrutiny of the paper, which led to the identification of the anomalies in the dataset, which led to the open letter issued to The New England Journal of Medicine. If the Lancet article were not published, it seems quite possible the Mehra et al. article would have never been challenged.
These events highlight the importance of post-publication peer review. Our current system of peer review was birthed by the photocopier. The advent of online publication, however, means we are no longer constrained by the need to send physical copies to individual reviewers. The open letter issued to The New England Journal of Medicine is a particularly dramatic example of post-publication peer review, but a similar process takes place every day on online platforms, such as Twitter, on which articles are critiqued almost as fast as they can be published. Learning how to harness this process and make it a formal part of article review is the next challenge for academic publishing.
Peer review may not be broken, but by broadening our definitions of peer and review, we can make it work even better.
Philip Seo, MD, MHS, is an associate professor of medicine at the Johns Hopkins University School of Medicine, Baltimore. He is director of both the Johns Hopkins Vasculitis Center and the Johns Hopkins Rheumatology Fellowship Program.
