a comprehensive survey of the reviewer assignment problem

🇺🇦    make metadata, not war

A COMPREHENSIVE SURVEY OF THE REVIEWER ASSIGNMENT PROBLEM

Similar works.

Research Papers in Economics

This paper was published in Research Papers in Economics .

Having an issue?

Is data on this page outdated, violates copyrights or anything else? Report the problem now and we will take corresponding actions after reviewing your request.

Browse Econ Literature

• Working papers
• Software components
• Book chapters
• JEL classification

More features

• Subscribe to new research

RePEc Biblio

Author registration.

• Economics Virtual Seminar Calendar NEW!

A Comprehensive Survey Of The Reviewer Assignment Problem

• Author & abstract
• 3 Citations
• Related works & more

Corrections

(School of Business, Sun Yat-sen University, Guangzhou, China)

Suggested Citation

Download full text from publisher.

Follow serials, authors, keywords & more

Public profiles for Economics researchers

Various research rankings in Economics

RePEc Genealogy

Who was a student of whom, using RePEc

Curated articles & papers on economics topics

Upload your paper to be listed on RePEc and IDEAS

New papers by email

Subscribe to new additions to RePEc

EconAcademics

Blog aggregator for economics research

Cases of plagiarism in Economics

About RePEc

Initiative for open bibliographies in Economics

News about RePEc

Questions about IDEAS and RePEc

RePEc volunteers

Participating archives

Publishers indexing in RePEc

Privacy statement

Found an error or omission?

Opportunities to help RePEc

Get papers listed

Have your research listed on RePEc

Open a RePEc archive

Have your institution's/publisher's output listed on RePEc

Get RePEc data

Use data assembled by RePEc

Help | Advanced Search

Computer Science > Digital Libraries

Title: reviewer assignment problem: a systematic review of the literature.

Abstract: Appropriate reviewer assignment significantly impacts the quality of proposal evaluation, as accurate and fair reviews are contingent on their assignment to relevant reviewers. The crucial task of assigning reviewers to submitted proposals is the starting point of the review process and is also known as the reviewer assignment problem (RAP). Due to the obvious restrictions of manual assignment, journal editors, conference organizers, and grant managers demand automatic reviewer assignment approaches. Many studies have proposed assignment solutions in response to the demand for automated procedures since 1992. The primary objective of this survey paper is to provide scholars and practitioners with a comprehensive overview of available research on the RAP. To achieve this goal, this article presents an in-depth systematic review of 103 publications in the field of reviewer assignment published in the past three decades and available in the Web of Science, Scopus, ScienceDirect, Google Scholar, and Semantic Scholar databases. This review paper classified and discussed the RAP approaches into two broad categories and numerous subcategories based on their underlying techniques. Furthermore, potential future research directions for each category are presented. This survey shows that the research on the RAP is becoming more significant and that more effort is required to develop new approaches and a framework.

Submission history

Access paper:.

• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

A Survey on Reviewer Assignment Problem

• Conference paper
• Cite this conference paper

• Fan Wang 1 ,
• Ben Chen 1 &
• Zhaowei Miao 2  

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 5027))

Included in the following conference series:

• International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems

3040 Accesses

19 Citations

Research into Reviewer Assignment Problem (RAP) is still in its early stage but there is great world-wide interest, as the foregoing process of peer-review which is the brickwork of science authentication. The RAP approach can be divided into three phases: identifying assignment procedure, computing the matching degree between manuscripts and reviewers, and optimizing the assignment so as to achieve the given objectives. Methodologies for addressing the above three phases have been developed from a variety of research disciplines, including information retrieval, artificial intelligent, operations research, etc. This survey is not only to cover variations of RAP that have appeared in the literature, but also to identify the practical challenge and current progress for developing intelligent RAP systems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Unable to display preview.  Download preview PDF.

Basu, C., Hirsh, H., Cohen, W.: Recommendation as Classification: Using Social and Content-based Information in Recommendation. In: 15th national/tenth conference on Artificial intelligence/Innovative applications of artificial intelligence, pp. 714–720. AAAI, USA (1998)

Google Scholar  

Basu, C., Hirsh, H., Cohen, W., Nevill-Manning, C.: Technical paper recommendation: A study in combining multiple information sources. Journal of Artificial Intelligence Research 14, 231–252 (2001)

MATH   Google Scholar  

Bausch, D.O., Brown, G.G., Hundley, D.R., Rapp, S.H., Rosenthal, R.E.: Mobilizing Marine Corps Officers. Interfaces 21(4), 26–38 (1991)

Benferhat, S., Lang, J.: Conference Paper Assignment. International Journal of Intelligent Systems 16, 1183–1192 (2001)

Article   MATH   Google Scholar  

Biswas, H.K., Hasan, M.M.: Using Publications and Domain Knowledge to Build Research Profiles: an Application in Automatic Reviewer Assignment. In: 2007 International Conference on Information and Communication Technology, pp. 82–86 (2007)

Cameron, D., Aleman-Meza, B., Arpinar, B.: Collecting Expertise of Researchers for Finding Relevant Experts in a Peer-Review Setting. In: First International ExpertFinder Workshop (2007)

Caron, G., Hansen, P., Jaumard, B.: The assignment problem with seniority and job priority constraints. Operations Research 47(3), 449–454 (1999)

Article   MATH   MathSciNet   Google Scholar  

Carter, M.W., Tovey, C.A.: When is the classroom assignment problem hard? Operations Research 40(1), S28–S39 (1992)

Casati, F., Giunchiglia, F., Marchese, M.: Publish and perish: Why the Current Publication and Review Model is Killing Research and Wasting Your Money, http://www.acm.org/ubiquity/views/v8i03_fabio.html

Cohen, P.R., Kjeldsen, R.: Information retrieval by constrained spreading activation in semantic network. Information Processing & Management 23(4), 255–268 (1987)

Article   Google Scholar  

Cohen, W., Fan, W.: Web-collaborative Filtering: Recommending Music by Crawling the Web. Computer Networks 33(1-6), 685–698 (2000)

Cook, W.D., Golany, B., Kress, M., Penn, M., Raviv, T.: Optimal Allocation of Proposals to Reviewers to Facilitate Effective Ranking. Management Science 51(4), 655–661 (2005)

Dell’Amico, M., Martello, S.: The k-cardinality assignment problem. Discrete Applied Mathematics 76(1-3), 103–121 (1997)

Dumais, S., Nielsen, J.: Automating the assignment of submitted manuscripts to reviewers. Research and Development in information Retrieval, 233–244 (1992)

Geller, J.: How IJCAI 1999 can prove the value of AI by using AI. In: 15th International Joint Conference on Artificial Intelligence, pp. 55–58 (1997)

Geller, J., Scherl, R.: Challenge: Technology for Automated Reviewer Selection (1997), http://njit.edu/~geller/finalchall.ps1997

Goldsmith, J., Solan, R.H.: The AI Conference Paper Assignment Problem, http://www.cs.uic.edu/~sloan/my-papers/GodlsmithSloanPaperAssignment.pdf

Gupta, D., Digiovanni, M., Narita, H., Goldberg, K.: Jester 2.0: A new Lineartime Collaborative Filtering Algorithm Applied to Jokes. In: Workshop on Recommender Systems at ACM SIGIR 1999 (1999)

Hansen, P., Wendell, R.E.: A note on airline commuting. Interfaces 11(12), 85–87 (1982)

Hartvigsen, D., Wei, J.C., Czuchlewski, R.: The Conference Paper-Reviewer Assignment Problem. Decision Sciences 30(3), 865–876 (1999)

Hettich, S., Pazzani, M.J.: Mining for Proposal Reviewers: Lessons Learned at the National Science Foundation. In: 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 862–871. ACM Press, New York (2006)

Chapter   Google Scholar  

Hill, W., Stead, L., Rosenstein, M., Furnas, G.: Recommending and Evaluating Choices in a Virtual Community of Use. In: SIGCHI conference on Human factors in computing systems, pp. 195–201. ACM Press/Addison-Wesley Publishing Co., New York (1995)

Hofmann, T.: Probabilistic Latent Semantic Analysis. In: 15th Conference on Uncertainty in Artificial Intelligence, pp. 289–296 (1999)

Janak, S.L., Taylor, M.S., Floudas, C.A., Burka, M., Mountzizris, T.J.: Novel and Effective Integer Optimization Approach for the NSF Panel-Assignment Problem: A Multiresource and Preference-Constrained Generalized Assignment Problem. Ind. Eng. Chem. Res. 45(1), 258–265 (2006)

Klingman, D., Phillips, N.: Topological and computational aspects of preemptive multicriteria military personnel assignment problems. Manage. Sci. 30(1), 1362–1375 (1984)

MathSciNet   Google Scholar  

Konstan, J., Miller, B., Maltz, D., Herlocker, L., Gordon, L., Riedl, J.: Grouplens: Applying Collaborative Filtering to Usenet News. Communications of the ACM 40(3), 77–87 (1997)

LeBlanc, L.J., Randels, D., Swann, T.K.: Heery International’s Spreadsheet Optimization Model for Assigning Managers to Construction Projects. INTERFACE 30(6), 95–106 (2000)

Merelo-Guervós, J.J., Castillo-Valdivieso, P.: Conference Paper Assignment Using a Combined Greedy/ Evolutionary Algorithm. In: Yao, X., Burke, E.K., Lozano, J.A., Smith, J., Merelo-Guervós, J.J., Bullinaria, J.A., Rowe, J.E., Tiňo, P., Kabán, A., Schwefel, H.-P. (eds.) PPSN 2004. LNCS, vol. 3242, pp. 602–611. Springer, Heidelberg (2004)

Merelo-Guervós, J.J., García-Castellano, F.J., Castillo, P.A., Arenas, M.G.: How Evolutionary Computation and Perl saved my conference. In: Sánchez, L. (ed.) Segundo Congreso Español sobre Metaheurísticas, Algoritmos Evolutivos y Bioinspirados, pp. 93–99 (2003)

Popescul, A., Ungar, L.H., Pennock, D.M., Lawrence, S.: Probabilistic Models for Unified Collaborative and Content-base Recommendation in Sparse-Data Environments. In: 17th Conference on Uncertainty in Artificial Intelligence, pp. 437–444. Morgan Kaufmann Publishers Inc., San Francisco (2001)

Rodriguez, M.A., Bollen, J., Van de Sompel, H.: Mapping the Bid Behavior of Conference Referees. Journal of Informetrics 1(1), 62–82 (2007)

Rodriguez, M.A., Bollen, J.: An Algorithm to Determine Peer-Reviewers. Arxiv preprint cs.DL/0605112 (2006)

Schirrer, A., Doerner, K.F., Hartl, R.F.: Reviewer Assignment for Scientific Articles using Memetic Algorithms. OR/CS Interfaces Series 39, 113–134 (2007)

Scott, A.: Peer review and the relevance of science. Futures 39, 827–845 (2007)

Shardanand, U., Maes, P.: Social Information Filtering: Algorithms for Automating “Word of Mouth”. In: SIGCHI conference on Human factors in computing systems, pp. 210–217. ACM Press/Addison-Wesley Publishing Co., New York (1995)

Sun, Y.H., Ma, J., Fan, Z.P., Wang, J.: A Hybrid Knowledge and Model Approach for Reviewer Assignment. In: 40th Annual Hawaii International Conference on System Sciences, p. 47. IEEE Computer Society, Washington (2007)

Tian, Q., Ma, J., Liu, O.: A Hybrid Knowledge and Model System for R&D Project Selection. Expert Systems with Applications 23(3), 265–271 (2002)

Veronika, A., Riantini, L.S., Trigunarsyah, B.: Corrective Action Recommendation For Project Cost Variance in Construction Material Management. In: Kanok-Nukulchai, W., Munasinghe, S., Anwar, N. (eds.) 10th East Asia-Pacific Conference on Structural Engineering and Construction 2005, pp. 23–28 (2006)

Watanabe, S., Ito, T., Ozono, T., Shintani, T.: A Paper Recommendation Mechanism for the Research Support System Papits. In: International Workshop on Data Engineering Issues in E-Commerce, pp. 71–80 (2005)

Weber, R.: The Journal Review Process: a Manifesto for Change. Communications of the Association for Information Systems 2(2-3) (1999)

Yarowsky, D., Florian, R.: Taking the load off the conference chairs: towards a digital paper-routing assistant. In: 1999 Joint SIGDAT Conference on Empirical Methods in NLP and Very-Large Corpora (1999)

Download references

Author information

Authors and affiliations.

School of Business, Sun Yat-Sen University, Guangzhou, P.R. China

Fan Wang & Ben Chen

Management School, Xiamen University, Xiamen, P.R. China

Zhaowei Miao

You can also search for this author in PubMed   Google Scholar

Editor information

Rights and permissions.

Reprints and permissions

Copyright information

© 2008 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper.

Wang, F., Chen, B., Miao, Z. (2008). A Survey on Reviewer Assignment Problem. In: Nguyen, N.T., Borzemski, L., Grzech, A., Ali, M. (eds) New Frontiers in Applied Artificial Intelligence. IEA/AIE 2008. Lecture Notes in Computer Science(), vol 5027. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-69052-8_75

Download citation

DOI : https://doi.org/10.1007/978-3-540-69052-8_75

Publisher Name : Springer, Berlin, Heidelberg

Print ISBN : 978-3-540-69045-0

Online ISBN : 978-3-540-69052-8

eBook Packages : Computer Science Computer Science (R0)

Share this paper

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Reviewer Assignment Problem: A Systematic Review of the Literature

• Department of Industrial Engineering
• Istanbul Technical University
• Yildiz Technical University

Research output : Contribution to journal › Review article › peer-review

Appropriate reviewer assignment significantly impacts the quality of proposal evaluation, as accurate and fair reviews are contingent on their assignment to relevant reviewers. The crucial task of assigning reviewers to submitted proposals is the starting point of the review process and is also known as the reviewer assignment problem (RAP). Due to the obvious restrictions of manual assignment, journal editors, conference organizers, and grant managers demand automatic reviewer assignment approaches. Many studies have proposed assignment solutions in response to the demand for automated procedures since 1992. The primary objective of this survey paper is to provide scholars and practitioners with a comprehensive overview of available research on the RAP. To achieve this goal, this article presents an in-depth systematic review of 103 publications in the field of reviewer assignment published in the past three decades and available in the Web of Science, Scopus, ScienceDirect, Google Scholar, and Semantic Scholar databases. This review paper classified and discussed the RAP approaches into two broad categories and numerous subcategories based on their underlying techniques. Furthermore, potential future research directions for each category are presented. This survey shows that the research on the RAP is becoming more significant and that more effort is required to develop new approaches and a framework.

Bibliographical note

Access to document.

• 10.1613/JAIR.1.14318

Other files and links

• Link to publication in Scopus

Fingerprint

• Systematic Literature Review Keyphrases 100%
• Reviewer Assignment Problem Keyphrases 100%
• reviews INIS 100%
• assignments INIS 100%
• Assignment Problem Computer Science 100%
• Reviewer Engineering 100%
• Systematic Review Psychology 100%
• Guides Social Sciences 100%

T1 - Reviewer Assignment Problem

T2 - A Systematic Review of the Literature

AU - Aksoy, Meltem

AU - Yanik, Seda

AU - Amasyali, Mehmet Fatih

N1 - Publisher Copyright: © 2023 AI Access Foundation. All rights reserved.

N2 - Appropriate reviewer assignment significantly impacts the quality of proposal evaluation, as accurate and fair reviews are contingent on their assignment to relevant reviewers. The crucial task of assigning reviewers to submitted proposals is the starting point of the review process and is also known as the reviewer assignment problem (RAP). Due to the obvious restrictions of manual assignment, journal editors, conference organizers, and grant managers demand automatic reviewer assignment approaches. Many studies have proposed assignment solutions in response to the demand for automated procedures since 1992. The primary objective of this survey paper is to provide scholars and practitioners with a comprehensive overview of available research on the RAP. To achieve this goal, this article presents an in-depth systematic review of 103 publications in the field of reviewer assignment published in the past three decades and available in the Web of Science, Scopus, ScienceDirect, Google Scholar, and Semantic Scholar databases. This review paper classified and discussed the RAP approaches into two broad categories and numerous subcategories based on their underlying techniques. Furthermore, potential future research directions for each category are presented. This survey shows that the research on the RAP is becoming more significant and that more effort is required to develop new approaches and a framework.

AB - Appropriate reviewer assignment significantly impacts the quality of proposal evaluation, as accurate and fair reviews are contingent on their assignment to relevant reviewers. The crucial task of assigning reviewers to submitted proposals is the starting point of the review process and is also known as the reviewer assignment problem (RAP). Due to the obvious restrictions of manual assignment, journal editors, conference organizers, and grant managers demand automatic reviewer assignment approaches. Many studies have proposed assignment solutions in response to the demand for automated procedures since 1992. The primary objective of this survey paper is to provide scholars and practitioners with a comprehensive overview of available research on the RAP. To achieve this goal, this article presents an in-depth systematic review of 103 publications in the field of reviewer assignment published in the past three decades and available in the Web of Science, Scopus, ScienceDirect, Google Scholar, and Semantic Scholar databases. This review paper classified and discussed the RAP approaches into two broad categories and numerous subcategories based on their underlying techniques. Furthermore, potential future research directions for each category are presented. This survey shows that the research on the RAP is becoming more significant and that more effort is required to develop new approaches and a framework.

UR - http://www.scopus.com/inward/record.url?scp=85162077444&partnerID=8YFLogxK

U2 - 10.1613/JAIR.1.14318

DO - 10.1613/JAIR.1.14318

M3 - Review article

AN - SCOPUS:85162077444

SN - 1076-9757

JO - Journal of Artificial Intelligence Research

JF - Journal of Artificial Intelligence Research

Reviewer Assignment Problem: A Systematic Review of the Literature

Article sidebar, main article content.

Appropriate reviewer assignment significantly impacts the quality of proposal evaluation, as accurate and fair reviews are contingent on their assignment to relevant reviewers. The crucial task of assigning reviewers to submitted proposals is the starting point of the review process and is also known as the reviewer assignment problem (RAP). Due to the obvious restrictions of manual assignment, journal editors, conference organizers, and grant managers demand automatic reviewer assignment approaches. Many studies have proposed assignment solutions in response to the demand for automated procedures since 1992. The primary objective of this survey paper is to provide scholars and practitioners with a comprehensive overview of available research on the RAP. To achieve this goal, this article presents an in-depth systematic review of 103 publications in the field of reviewer assignment published in the past three decades and available in the Web of Science, Scopus, ScienceDirect, Google Scholar, and Semantic Scholar databases. This review paper classified and discussed the RAP approaches into two broad categories and numerous subcategories based on their underlying techniques. Furthermore, potential future research directions for each category are presented. This survey shows that the research on the RAP is becoming more significant and that more effort is required to develop new approaches and a framework.

Article Details