September 2nd, 2020

Promoting open science practices is a key ambition of Quantitative Science Studies (QSS), the official journal of the International Society for Scientometrics and Informetrics (ISSI), published by MIT Press. QSS is an open access journal and has recently been indexed in the Directory of Open Access Journals. It charges a relatively modest APC (USD 600 for ISSI members and USD 800 for others; covered by a subsidy from TIB if authors have no institutional support for APCs) and provides a transparent cost breakdown. Following the recommendations made by the Initiative for Open Citations, reference lists of articles published in QSS are openly available in Crossref. The same applies to other metadata, such as author names, ORCIDs, affiliations, abstracts, etc. Other open science practices adopted by QSS include transparency of author contributions (using the CRediT taxonomy), reporting of competing interests, and data sharing. QSS also encourages authors to post preprints of their submissions in preprint repositories.

QSS is now taking an additional step to promote open science practices. In close collaboration with MIT Press, we have begun a transparent peer review pilot project. Transparent peer review, sometimes also referred to as open peer review, is an umbrella term for a range of innovations that increase the transparency and openness of peer review, with the ultimate aim of improving quality control in science. Our transparent peer review pilot focuses on making the content of review reports openly available. For articles where authors have chosen to participate in the pilot and that are accepted for publication in QSS, the review reports, along with the responses of the authors and the decision letters of the editor, will be published in Publons under a CC BY license. The pilot also offers reviewers the option of revealing their identity, but this is not required. For articles that are rejected for publication in QSS, review reports will not be published.

We expect that publishing review reports will lead to several benefits. Review reports provide insights on the strengths and weaknesses of an article, with the comments ideally informing the final version of an article. In a traditional closed peer review process, contributions from reviewers are not shared publicly. Furthermore, unresolved disagreements among reviewers, editors, and authors are not made visible to readers. Transparent peer review has the advantage of making these publicly visible, providing helpful background information for readers of an article. Transparent peer review also demonstrates the level of scrutiny of evaluations performed by journals, enabling the scientific community to distinguish between journals that systematically provide robust evaluations and those that adopt more superficial standards, including ‘predatory’ journals, but also established journals that do not apply uniform evaluation criteria.

Transparent peer review is not without potential disadvantages. Authors may choose not to participate for a variety of reasons: concerns about criticisms of their initial submission being made public; concerns that dissenting work will not be read fairly in the context of reviews, particularly from well-established scholars; and concerns that the full context of reviews may diminish the credibility and finality of the editorial decision. Reviewers may choose not to participate fearing a greater expectation in reviewing in a transparent environment; they may modify their reviews to avoid implications of their identity; and they may provide more positive reviews in order to avoid public conflict in the case that their identity is revealed. Also, if reviewers choose to sign their reviews, this may change the dynamics between authors and reviewers, which could have undesirable side effects (e.g., quid pro quo between authors and reviewers and negative consequences of the social hierarchies in science).

Despite the uncertainty around transparent peer review, there is growing support for making peer review more transparent and open and for getting a better understanding of the various forms of peer review. For instance, as shown in a recent study by one of our editorial board members, more than 600 journals are already using some form of transparent or open peer review. Moreover, an increasing number of journals, including prominent ones such as Nature, are conducting pilot projects akin to our own. Furthermore, a recent trial by 11 Wiley journals found that 87% of the authors were willing to publish the review reports for their article. There was negligible effect on the willingness of researchers to review for the journals. This suggests that the benefits of transparency may outweigh some of the perceived disadvantages.

Recognizing that transparent peer review has benefits but also risks, this pilot will enable the QSS community to familiarize itself with this form of reviewing and to better understand the advantages and disadvantages within our community. The results of the pilot will be evaluated by the editorial board of QSS and made public. Based on this, a decision about the continued use of transparent peer review at QSS will be made. The transparent peer review trial will run for at least half a year, depending on the degree of participation of the community. Authors who do not want to participate may opt out and will still be able to publish their work in QSS. The ScholarOne submission system used by QSS has been reconfigured to make sure that authors and reviewers are properly informed about the pilot.

We invite you to join us in this pilot by submitting your work to QSS and participating in transparent peer review. We hope this will serve to improve peer review in our field, strengthen the quality of research, and provide an empirical basis for engaging in open science practices.

 

Ludo Waltman, Editor-in-Chief QSS

Vincent Larivière, Associate Editor QSS

Staša Milojević, Associate Editor QSS

Cassidy R. Sugimoto, President ISSI