3.1.1. System Architecture

ACOMS+ is a system that facilitates a flexible and efficient peer review process and journal management. The objectives of ACOMS+ can be summarized as follows: First, ACOMS+ aims to streamline the submission process for academic organizations by providing high-quality peer review features; second, ACOMS+ aims to promote and advance open access and open science by offering features reflecting the latest academic publishing trends; and third, ACOMS+ can be used free of charge to reduce the financial burden of academic organizations. The system architecture has been designed to achieve these goals, as illustrated in Fig. 1. The system architecture, focusing on the user roles and related systems, is now described.

Fig. 1

System architecture of ACOMS+. ACOMS+, AccessON Peer Review Management System Plus.

All ACOMS+ users become AccessON integrated members. AccessON provides a service to help Korean researchers access open access papers more easily and academic organizations to disclose and disseminate research results with open access. ACOMS+ users can use all services provided by AccessON, including Collab, Self-Archiving, and SAFE. Notably, they can access the submission and review systems of all journals using ACOMS+ with a single integrated account provided by AccessON.

Authors represent the most basic role in ACOMS+. Once registered to a journal’s submission and review system, a member is automatically granted permissions as an author. Authors can submit their papers to the journal of their choice by entering details regarding the paper and uploading the manuscript. Upon completion of peer review, they can further improve the manuscript based on the review comments and resubmit it. When the manuscript is finally accepted for publication, the author can print out the certificate of acceptance and proof the edited manuscript before publication.

The role of reviewers is crucial in the peer review process. The members who are registered in the system are granted rights as a reviewer and can act as reviewers from the moment they are invited to do so. After accepting an invitation to review an article, they should review the article according to the journal’s requirements. ACOMS+ stores the history of review completed, prints out the review certificate, and automatically enters the history of peer review activities in association with ORCID.

Editors or chief editors are in charge of submission management. When a manuscript is submitted, an editorial board member and a reviewer are assigned. Once the reviewer has completed the review, the chief editor will make a decision based on the review comments and notify the author of the result. The chief editor may manage the publication of papers and assign the accepted papers for publication in each issue. The authority of chief editor may be granted by the academic organization, and the authority of the editorial board may be granted by the chief editor or the academic organization to journal members registered in the system.

Publishing managers are responsible for the publication of articles. They assign articles accepted for publication to volumes/issues, set the issue type, communicate with authors, and proof the edited article iteratively. Once the manuscript has been proofed by the author, it is ready for publication as a journal article.

Academic societies handle the setup and administration of the system to be used. They enter basic information and policies of journals, establish the conditions for submission and review, and manage the issues of published papers and the rights of journal members. ACOMS+ provides features that allow journal administrators to view the history of submission and review as well as the statistics and data to be used to register in Korean and international indexing databases.

The system administrator is in charge of managing the system in general. They perform tasks such as managing system-wide announcements, approving applications to utilize the system, and creating journal systems. They also improve the system to reflect the needs of academic organizations and correct system errors. They can also view the overall statistics of journals using the system.

ACOMS+ provides linkage with external and internal systems. External systems include ORCID and KCI. ACOMS+ also provides features including notifications of the review process through mobile messengers and payment of article processing charges (APC). Internally, it is linked to various systems provided by KISTI, primarily AccessON, which is explained in more detail in Section 3.1.2 and Section 3.2.2. Also, it supports linkages with the Korea DOI Center. If a journal is issued a DOI via this DOI Center, the DOI can be automatically issued by entering the metadata and full-text of manuscripts. Finally, all actions occurring in the ACOMS+ system are stored in a database systematically.

3.1.2. Flowchart of the Peer Review Process

Due to its organic interaction with users in various roles and a diversity of internal and external systems, the submission and review system involves a complicated process. Fig. 2 shows the flowchart of the ACOMS+ process. A series of procedures in the ACOMS+ system—from application and setup to submission and review, publication, dissemination, and distribution—is now described.

Fig. 2

Process flowchart of ACOMS+. ACOMS+, AccessON Peer Review Management System Plus; KISTI, Korea Institute of Science and Technology Information; APC, article processing charges; DOAJ, Directory of Open Access Journals; DOI, Digital Object Identifier; JATS XML, Journal Article Tag Suite Extensible Markup Language.

First, the academic organization fills in and submits the application form online. For the application process, the academic organization is required to enter basic information, including the name of the journal, ISSN, date of foundation, and publication cycle. Based on the application submitted, the system administrator checks the information to ensure that the information is accurate and that the journal is not predatory. If all conditions are satisfied, the system administrator generates a system and provides the URL of the submission and review system and journal administrator page to the academic organization.

The next step is the setup of the journal information for the operation of the submission and review system. The journal administrator of the academic organization selects the journal’s open access, copyright, self-archiving, APC, and peer review policies, and creates the relevant policy guidance pages. The conditions for submission review should also be set. To provide the optimal environment for each journal, ACOMS+ provides several options such as journal language, abstract length, number of keywords, file types, and journal discipline. Lastly, templates of alert mails sent by the system during the submission and review process can be customized and saved for each journal preference to complete the setup.

When the setup is completed, users are ready to submit manuscripts and review articles. ACOMS+ provides a basic submission and review process. An author logs in and submits an article. The author checks the basic policies of the journal and enters the article type, title, abstract, keywords, author information, and field(s), and uploads the files required by the journal (e.g., manuscript, cover letter, supplementary materials, figures, tables) for submission. Then, the chief editor receives the submission and follows an established protocol to check whether the manuscript is submitted in accordance with the peer review policy and whether there is plagiarism. If a submission is not appropriate for the journal, the chief editor may reject it without review. If the manuscript requires simple corrections or enhancements, the chief editor may return the submission to the author without rejecting it. In this case, the author modifies and resubmits the manuscript based on the chief editor’s comments.

If it is judged that the manuscript is ready for review, the chief editor checks in the manuscript and assigns an appropriate editor to the article. The editor then decides whether to accept the assignment. For example, they may decline an assignment if there is a conflict of interest with the author(s) of the assigned manuscript, if the manuscript does not fit their research area, or if the manuscript is of poor quality and needs to be rejected or enhanced. If the assignment is accepted, the editor invites reviewers. Specifically, the editors determine the number of reviews for editor recommendation, find the right experts, and send them emails asking them to review the manuscript. Next, the reviewers perform the review. Reviewers can download full-text files only if they accept the invitation. For double-blind peer review journals, the system restricts information disclosure so that reviewers do not know who the authors are. Once all reviewers have completed their review, the editor integrates the reviewers’ comments to make an editor recommendation, and the chief editor will send the decision (e.g., accept, major revision, minor revision, reject) to the author. Finally, the author submits a revised version of the manuscript along with a response letter to the reviewers, or submits the final version for publication.

The manuscript accepted after peer review is published and disseminated in linkage with AccessON Journal Repository. Journal Repository is an online service platform for open access journals supporting journal publishing, distribution, and archiving. Journal publishing managers can generate new volumes/issues of the journal and assign manuscripts accepted for publication in ACOMS+ to volumes/issues. After the process of editing, they communicate with the author to review and proofread the published version. Once verified, the published version is made available immediately on Journal Repository. The published articles are indexed in AccessON and ScienceON as well as on the homepage of each journal, the Journal Repository, to facilitate dissemination via academic search engines such as Google Scholar.

3.2.1. Open Peer Review Process

ACOMS+ is Korea’s first submission and peer review system to support the open peer review process. Submission and peer review systems or publishers have different open peer review process (Tennant et al., 2017). Fig. 3 shows the open peer review process provided by ACOMS+. ACOMS+ provides the basic and hybrid options based on the transparent peer review model. In the basic option, the journal administrator can set the open peer review option to disclose review information when the paper is published after peer review, which may be single-blinded or double-blinded, so that authors do not know who the reviewers are. The information disclosed with the publication may include reviewer information, review reports, author response, manuscript version, and editor information, and the journal can select the information to be disclosed as they want.

Fig. 3

Open peer review process of ACOMS+. ACOMS+, AccessON Peer Review Management System Plus.

The hybrid option is based on the agreement of four major parties in the peer review process—journal, author, reviewer, and chief editor. First, the journal administrator decides whether to allow hybrid open peer review when setting the journal’s peer review policy. If open peer review is allowed, the next choice is passed on to the author. When submitting a new manuscript to a hybrid open peer review journal, the author can choose whether to accept open peer review or not. If the author agrees, the reviewers choose whether to allow open peer review when they accept the invitation to review. The reviewers are notified that their details and the review report will be made public with the publication. If both the author and reviewers accept the use of hybrid open peer review, the chief editor selects whether to allow open peer review when deciding the acceptance of the paper for publication. This is the final step in determining whether it is appropriate to disclose the reviewer comments and author responses that were exchanged during the peer review process. The entire peer review process is conducted as a single- or double-blind review, where the authors are blinded to the reviewers’ identity.

The settings and four-way agreements for open peer review made during the peer review process are stored and managed in a database. When open peer review is selected, journals can make the review information public with the publication of the paper while the readers are allowed to view the review process transparently.

3.2.2. Open Access Publishing and Self-Archiving

ACOMS+ supports both gold open access and green open access—two representative open access models. To this end, ACOMS+ is closely linked to AccessON. This subsection describes in detail the linkage functions to support each model.

The gold open access features are provided in conjunction with Journal Repository. ACOMS+ and Journal Repository are connected as a single system to support the entire process of open access academic publishing from submission and review to publication and dissemination. An open access journal can integrate ACOMS+ and Journal Repository and use the integrated administrator page to manage both systems at once. This integration saves journal administrators the effort of managing the same information into two different systems because the journal’s basic information and journal policies are linked. Additionally, as described in Section 3.1.2, submission and review data of ACOMS+ can be linked to the publication process. For papers that have undergone open peer review, the reviewers’ information and review reports can be disclosed with the publication without the need to enter any information or upload any files separately in Journal Repository.

ACOMS+ and Journal Repository provide open access declaration and policy templates. Fig. 4 presents the concept of auto-populating policy templates based on journal settings. For example, if a journal declares open access and selects CC BY-NC as its license policy, the settings are stored in the database. The system will retrieve the license policy wording corresponding to the set value from the database and automatically populate it on the user page. Journal administrators can modify the wording as needed and disclose it. ACOMS+ and Journal Repository offer policy templates for CCL, copyright, self-archiving, APC, and journal archiving, which can be used to evaluate the publishing environment of an open access journal (Kim et al., 2023). In terms of journal operation, this feature is expected to eliminate barriers for journals concerned about the difficulty of setting policies with regard to transition to open access publishing, and to help open access journals to be listed in prestigious journal indexes such as Web of Science or DOAJ.

Fig. 4

Concept of rule-based open access policy generation.

The features for green open access are provided through a connection with AccessON’s Self-Archiving. The journal administrator page provides a feature to set author self-archiving conditions. Authors can use this linkage if the journal allows pre-print or post-print self-archiving of accepted manuscripts to any website. In this case, when submitting the final article after being accepted for publication, authors are asked if they wish to self-archive their manuscripts to AccessON. If the author agrees, self-archiving happens automatically without the need to review the journal’s policies or upload files. If an article is under embargo, the system will calculate the embargo period and automatically release the pre-print and post-print after the embargo ends. To sum up, ACOMS+ has built an optimal environment for green open access by minimizing the author’s work for self-archiving.

3.2.3. Online Quantitative/Qualitative Evaluation

ACOMS+ ensures objective and fair review and improves the quality of peer review by providing a structured set of quantitative and qualitative metrics. A case study, survey, and focus group interview that were conducted to build the metrics and the results reflected in the system are outlined in this subsection.

A case study of journals, publishers, and systems in Korea and abroad to comprehensively review items widely used to evaluate academic articles was firstly conducted. For Korea, 177 items from 20 journals, including the Journal of Korean Academic Society of Nursing Education and journals in the field of library and records management were collected. For the international context, 149 items were collected from 12 systems, publishers, and journals, including article submission and review systems such as Editorial Manager and ScholarOne, and large publishers such as Elsevier and Emerald Publishing, and journals such as F1000Research and PLOS ONE. Based on the collected items, items and contents to initially classify the major themes and detailed evaluation items were analyzed. Then, a survey was conducted to evaluate the appropriateness and consistency of the initial evaluation items among journals using ACOMS. The questionnaire comprised questions on whether the division of major themes is appropriate, whether the detailed evaluation items are appropriate for each major theme, and what detailed evaluation items should be included in major themes. It was conducted online between June 3 and June 18, 2022, and 102 of the 113 journals responded. During the survey period, a focus group interview was conducted simultaneously with the editorial board members of five journals to get opinions on the organization of the major themes, detailed evaluation items, and evaluation methods. Finally, the results of the survey and interview were analyzed to integrate similar themes and move detailed evaluation items under appropriate themes. Consequently, five major themes and 32 detailed evaluation items were finalized (Appendix A).

ACOMS+ supports all major themes and evaluation items by default and provides a feature to customize evaluation methods to suit the needs of each journal. Each item can be modified or removed depending on the subject area or evaluation perspective of journals. For quantitative evaluation, journals can choose between a 5-point Likert scale, a 7-point scale, and a scoring system. They can also choose whether to use quantitative or qualitative evaluation. Reviewers perform quantitative and/or qualitative evaluations based on the settings selected by the journal administrator (Fig. 5); the editors, chief editor, and authors can view the evaluation results in detail. Also, ACOMS+ provides a feature for authors to submit a revised manuscript with responses to each of the reviewers’ qualitative evaluations.

Fig. 5

Snippets from the qualitative/quantitative evaluation table.

3.2.4. Peer Reviewer Invitation

One of the key challenges in the operation of a journal’s peer review system is to secure a pool of reviewers and allocate appropriate reviewers to an article. This subsection briefly describes the features for editors to use to invite reviewers.

First, ACOMS+ provides a function of searching reviewers through diverse reviewer pools. Total five reviewer search pools are provided: existing reviewers, corresponding authors, ACOMS+ members, ORCID referrals, and journals’ own reviewers. Existing reviewer pools, author pools, and ACOMS+ member pools can be searched against users of other journals or ACOMS+ as a whole as well as against users within a journal. It is planned to introduce a data-driven algorithm that recommends appropriate reviewers based on each user’s publication and review history. Second, ACOMS+ enables editors to invite reviewers simply by entering the name and e-mail address. When an expert is considered a good fit even though not in a pool of reviewers, this feature can be used to invite the expert as a reviewer. Third, ACOMS+ enables non-members to review an article. Reviewers are allowed access to reviewing without the need to register as ACOMS+ users. If a reviewer signs up for membership after completing a review, the review history accumulated as a non-member is consolidated, making it easier to manage review history. Finally, ACOMS+ also offers a consolidated view of the review history for a reviewer (Fig. 6). Editors can learn about a reviewer’s reviewing behavior through their history and use it to make decisions about reviewer invitations. It is also planned to enhance this feature by providing a list of published papers and keywords of the research conducted by the reviewer.

Fig. 6

Sample of review history page.