I was recently invited to work as a guest editor for a super-interesting paper. The focus of the paper closely aligned with my research background, so I accepted.

The empirical part of the manuscript seemed solid enough. The main results were displayed in a graph and the summary statistics. The authors had also decided to include several robustness checks, controlling for confounding factors while still finding the same results.

I was less certain about the theoretical foundation of the paper, however. Important interpretations were scattered across the manuscript, and I thought that important contributions from previous research were missing. So although I was convinced that the paper should be sent out for review, I thought it needed reviewers with in-depth theoretical knowledge, who knew and had contributed to the previous literature.

Two people who I judged to fit that bill agreed to review the paper and, after a few months, I had both their reports.

����Vlog

As an editor, life is easy when reviewers come to the same conclusion. In this case, both rejected the paper and declined to review a revision. But I wanted to summarise the main critique in my decision letter, so I started reading the reports.

I was surprised when going through the first. I knew the reviewer’s previous work and expected them to be an expert on the manuscript’s theoretical element. But almost all their comments focused on the methodological part. The reviewer was especially critical regarding the power analysis, proposed an alternative econometric model and suggested that the author should use an LLM to recategorise the data (as a robustness check). Finally, the reviewer expected a more extensive discussion regarding the ethical considerations of the study.

����Vlog

Had I misinterpreted the reviewer’s skills and knowledge? Intrigued, I continued with the second reviewer’s report. This was a bit more balanced, containing a few remarks and suggestions regarding the theoretical contribution. Still, its focus was on the econometrics as well – albeit that the suggestions it made contrasted with the first reviewer’s. And the review followed a remarkably similar structure – once again concluding with a suggestion regarding the ethical contributions of the paper.

Focusing on the ethical contribution was strange because the author had already dedicated a complete paragraph on this. Why would both reviewers focus on that? And while the econometric recommendations were basically all fine, the paper was already stuffed with robustness checks so these comments were unlikely to materially improve the paper. Why would reviewers with a theoretical background focus on them?

I was now leaning towards the conclusion that both reviewers had used AI to write large parts of their reports. So how was I to proceed? Should I reject the paper on the basis of these reviews? Should I ignore them and contact new reviewers? Should I feed the most recent LLMs with endless prompts to try to understand which specific comments were generated by AI so I could discount those?

I asked the chief editor. Their response: as long as the reviewers did not tick the “Did you use AI in creating this review?” box, I was to treat this as a genuine review.

I responded that I could not, in good conscience, reject the paper based on these reviews and stood down as guest editor. Yet my replacement is probably going to use the reviews and reject the paper anyway – and the author must now wait even longer for that decision.

So did I do the right thing? After all, the review and editorial process was never fail-safe. Humans are prone to making mistakes. Many great papers were initially rejected, and many terrible papers were accepted. So would handing the reviewing task over to AI really be such a bad thing for science?

����Vlog

My view is that it would. AI-generated reviewer reports that endlessly demand the same structure, theoretical framework and econometric approach will result in uniform, uncontroversial and ordinary research. And while researchers are quick to acknowledge the benefits of AI in research, this insistence on blandness will result in intellectually stagnating disciplines.

The problem, of course, is that the pressure on everyone to publish means that referees increasingly lack the time to focus extensively on other people’s papers, especially when they are asked – because of the volume of submissions journals now receive – to review many papers each month. But universities could free up that time if they focused on publication quality rather than quantity.

����Vlog

That is a challenge, because quality is harder to measure than quantity. Focusing on quantity is easy for universities. They just have to count the number of published papers a researcher produced in a certain time frame. But focusing on quality gives researchers the time to, for example, run multi-study experiments, carry out sufficient robustness checks or conduct in-depth interviews with as many participants as necessary. Each paper takes longer but there are fewer of them. This reduces the number of submissions, and reviewers and editors would have the time to focus on far fewer papers.

Second, – an approach undervalued at almost every leading journal – would give reviewers the chance to comment on the design and suggest econometric methods before the study is conducted. This approach would strengthen the collaboration between reviewers and author while reducing hindsight-related commentary, spreading the reviewing burden over time.

Third, editors could ask reviewers to evaluate specific parts of a manuscript – and nothing else. This was my mistake as an editor. I should have demanded a specific and not a generic evaluation.

I never heard from the author in this case, but that is exactly the problem. Once a paper is rejected, authors rarely get to voice complaints. Either way, it cannot be acceptable for an author to wait half a year to read reviewers’ reports that their own AI could have created in less than a minute.

If we are willing to embrace what AI offers research, we must be just as willing to confront what it can quietly take away.

is professor in the department of social studies at the University of Stavanger, Norway.

����Vlog