literature review on digital library system

• Architecture and Design
• Asian and Pacific Studies
• Business and Economics
• Classical and Ancient Near Eastern Studies
• Computer Sciences
• Cultural Studies
• Engineering
• General Interest
• Geosciences
• Industrial Chemistry
• Islamic and Middle Eastern Studies
• Jewish Studies
• Library and Information Science, Book Studies
• Life Sciences
• Linguistics and Semiotics
• Literary Studies
• Materials Sciences
• Mathematics
• Social Sciences
• Sports and Recreation
• Theology and Religion
• Publish your article
• The role of authors
• Promoting your article
• Abstracting & indexing
• Publishing Ethics
• Why publish with De Gruyter
• How to publish with De Gruyter
• Our book series
• Our subject areas
• Your digital product at De Gruyter
• Contribute to our reference works
• Product information
• Tools & resources
• Product Information
• Promotional Materials
• Orders and Inquiries
• FAQ for Library Suppliers and Book Sellers
• Repository Policy
• Free access policy
• Open Access agreements
• Database portals
• For Authors
• Customer service
• People + Culture
• Journal Management
• How to join us
• Working at De Gruyter
• Mission & Vision
• De Gruyter Foundation
• De Gruyter Ebound
• Our Responsibility
• Partner publishers

Your purchase has been completed. Your documents are now available to view.

A Literature Review on Digitization in Libraries and Digital Libraries

The purpose of this work was due to critical literature reviews often being used as a crucial tool for the research outputs of different ideologies to access and develop the knowledge base. The main aim of this paper is to review current literature on digitization in libraries considering different facets like process of selection, acquisition, conversion, creation of metadata, storage, preparing for long term access and issues and challenges for managing these through a review of literature. The current study follows a literature review process where there is analysis of 56 full papers on different aspects of digitization and digital libraries after a filtration from 110 research papers from 2010 to 2019. A number of major issues and challenges like high budget, diversity in records, efficiency and hard labor, document security, quality control, information retrieval, and manipulation of metadata associated with digitization and digital library management with correspondence solutions to them were identified. The study provides valuable information to researchers and librarians working in the field of digitization by illustrating the process, challenge, and implication in libraries as were outlined by various scholars from different corners of the world.

Adekannbi, J. O. 2015. “Comparative Analysis of the Preservation and Conservation Techniques of Selected Special and Academic Libraries in Nigeria.” Library Philosophy and Practice 1328, https://digitalcommons.unl.edu/libphilprac/1328/ . Search in Google Scholar

Adjei, E., and M. Mensah. 2019. “The Story So Far-Digital Preservation in Institutional Repositories.” Digital Library Perspectives 35: 80–96, https://doi.org/10.1108/dlp-12-2018-0039 . Search in Google Scholar

Akter, T. 2013. “Digital Preservation of Library Materials.” International Research: Journal of Library and Information Science 3 (1): 20–29. Search in Google Scholar

Alhaji, I. U. 2015. “Digitization of Library Resources and the Formation of Digital Libraries: A Practical Approach.” University News 53 (47): 22–8. Search in Google Scholar

Atram, P. N. 2017. “Digital Library Services in the Digital Age.” International Journal of Library and Information Science 6 (1): 79–82. Search in Google Scholar

Bakshi, S. I. 2016. “Digitization and Digital Preservation of Cultural Heritage in India with Special Reference to IGNCA, New Delhi.” Asian Journal of Information Science and Technology 6 (2): 1–7. 10.51983/ajist-2016.6.2.134 Search in Google Scholar

Balogun, T. 2018. “The Nexus between Digitization, Preservation and Access in the Context of Selection of Materials for Archives.” Library Philosophy and Practice , https://digitalcommons.unl.edu/libphilprac/1893/ . Search in Google Scholar

Bandi, A., and S, Bandi. 2011. “Open Access to Knowledge: Initiatives in India.” In National Conference on Beyond Librarianship:Creativity, Innovation and Discovery . New Delhi: Vedams Books. Search in Google Scholar

Baro, E. E. 2013. “Digitization Projects in University Libraries in Nigeria: The Journey So Far.” Library Hi Tech News 9: 21–6, https://doi.org/10.1108/lhtn-09-2013-0049 . Search in Google Scholar

Bishoff, L. 2010. “Digital Preservation Plan.” Information Standards Quarterly 22 (2): 20–25, https://doi.org/10.3789/isqv22n2.2010.03 . Search in Google Scholar

Chauhan, S. 2012. Digitization of Resources in University Libraries in India: Problems and Perspectives (Thesis) . Amritsar: Department of Library and Information Science, Guru Nanak Dev University. Search in Google Scholar

Cherukodan, S., G. S. Kumar, and S. H. Kabir. 2011. “Using Open Source Software for Digital Libraries: A Case Study of CUSAT.” The Electronic Library 31 (2): 217–25. 10.1108/02640471311312393 Search in Google Scholar

Dappert, A., and M. Enders. 2010. “Digital Preservation Metadata Standards.” Information Standards Quarterly 22 (2): 4–13. 10.3789/isqv22n2.2010.01 Search in Google Scholar

Dhamdhere, S. N. 2012. “Digital Library Services and Practices: An Online Survey.” International Journal of Library Science 6 (2): 65–75. Search in Google Scholar

Fox, R. 2011. “Digital Libraries: The Systems Analysis Perspective Forensics of Digital Librarianship.” OCLC Systems and Services International Digital Library Perspectives 27 (4): 264–71, https://doi.org/10.1108/10650751111182560 . Search in Google Scholar

Gbaje, E. S. 2013. “Digital Preservation Policy in National Information Centres in Nigeria.” The Electronics Library 31 (4): 483–92, https://doi.org/10.1108/el-01-2012-0011 . Search in Google Scholar

Goswami, M. 2018. Preservation of Library Documents in Digital Environment with Reference to College Libraries of Assam: An Empirical Study (Thesis) . Guwahati: Department of Library and Information Science, Guwahati University. Search in Google Scholar

Hariri, N., and Y. Norouzi. 2011. “Determining Evaluation Criteria for Digital Libraries’ User Interface: A Review.” The Electronic Library 29 (5): 698–722, https://doi.org/10.1108/02640471111177116 . Search in Google Scholar

Hirwade, M. A. 2011. “A Study of Metadata Standards.” Library Hi Tech News 28 (7): 18–25, https://doi.org/10.1108/07419051111184052 . Search in Google Scholar

Islam, S. 2011. “Towards Digitization: Problems and Prospects for the Libraries of Bangladesh.” World Congress of Muslim Librarians and Information Scientists 2011, IIUM , https://www.academia.edu/3342256 . Search in Google Scholar

Katre, D. 2012. “Need of Legisalation and Digital Preservation Policy Framework in Indian Context.” DESIDOC Journal of Library and Information Technology 32 (4): 321–6, https://doi.org/10.14429/djlit.32.4.2526 . Search in Google Scholar

Krtalic, M. 2012. “Exploring a Framework for Comprehensive and Successful Preservation Management in Libraries.” Journal of Documentation 68 (3): 353–77. 10.1108/00220411211225584 Search in Google Scholar

Kumar, K. 2014. “Analytical Survey on Digital Preservation and Techniques Among Engineering Education Institutional Libraries in Rayalaseema Region of Andhra Pradesh.” Library Philosophy and Practice , https://digitalcommons.unl.edu/libphilprac/1045/ . Search in Google Scholar

Kumar, N. 2016. “E-resources preservation: Challenge with libraries of twenty first century.” International Journal of Education & Management 6 (2): 229–232. Search in Google Scholar

Kumar, D. V., and P. Padma. 2015. “Need and Opportunities of Digital Preservation in Academic Libraries: An Overview.” International Journal of Educational Research and Technology 6 (4): 13–7. Search in Google Scholar

Londhe, N. L. 2011. “Development of a Digital Library of Manuscripts: A Case Study at the University of Pune, India.” Program: Electronic Library and Information Systems 45 (2): 135–48, https://doi.org/10.1108/00330331111129697 . Search in Google Scholar

Mapulanga, P. 2012. “Digitising Library Resources and Building Digital Repositories in the University of Malawi Libraries.” Electronic Library 31 (5): 635–47, https://doi.org/10.1108/EL-02-2012-0019 . Search in Google Scholar

Maurya, R. N. 2011. “Digital Library and Digitization.” International Journal of Information Dissemination and Technology 1 (4): 228–31. Search in Google Scholar

Moulaison, H. L. 2015. “Open DOAR Repositories and Metadata Practices.” D-Lib Magazine 21: 3–4, https://doi.org/10.1045/march2015-moulaison . Search in Google Scholar

Musa, A. U., and S. Musa. 2012. “Digitization of Library Resources in Kashim Ibrahim Library: Processes, Challenges and the Impacts on the Services of the Library.” In Kaduna State Chapter of NLA/AGM Conference Held at Kaduna State . University Kaduna. Search in Google Scholar

Nneji, K. O. 2018. “Digitization of Academic Library Resources: A Case Study of Donald E.U. Ekong Library.” Library Philosophy and Practice , https://digitalcommons.unl.edu/libphilprac/1990/ . Search in Google Scholar

Nnenna, O. B., and E. L. Ume. 2015. “Digitization of Library Resources in Academic Libraries: Challenges and Implication.” IOSR Journal of Mobile Computing and Application 2 (2): 35–40, 10.9790/0050-0223540 . Search in Google Scholar

Oni, O., L. Abu, and I. A. Ekeniyere. 2018. “Prospects and Challenges of Digitization of Academic Libraries Resources for Effective Services Delivery in Universities in Edo State.” International Researchers: Journal of Library and Information Science 8 (2): 278–92. Search in Google Scholar

Perez, I. J., E. Herrera-Viedma, J. López-Gijón, and F. J. Cabrerizo. 2010. “A New Application of a Fuzzy Linguistic Quality Evaluation System in Digital Libraries.” In Proceedings of the 10th International Conference on Intelligent Systems Design and Applications, IEEE, 639–44. Cairo, Egypt. 10.1109/ISDA.2010.5687193 Search in Google Scholar

Rafiq, M., and K. Ameen. 2013. “Digitization in University Libraries of Pakistan.” OCLC Systems and Services: International Digital Library Perspectives 29 (1): 37–46, https://doi.org/10.1108/10650751311294546 . Search in Google Scholar

Rafiq, M., and K. Ameen. 2017. “Barriers to Digitization in University Libraries of Pakistan: A Developing Country’s Perspective.” The Electronic Library 36 (3): 457–70. 10.1108/EL-01-2017-0012 Search in Google Scholar

Rahman, A., M. J. Rahman, and M. H. H. Chowdhury. 2015. “Digital Resource Management in Libraries: Step Towards Digital Bangladesh.” National Seminar on Cross Talk of Digital Resource Management : 1–24. Search in Google Scholar

Rathour, P. R., and A. K. Sahu. 2011. “Comparison of Open Source Software for Digital Libraries.” World Digital Libraries 4 (1): 19–31. Search in Google Scholar

Reddy, T. R., and K. Kumar. 2013. “Open Source Software’s and Their Impact on Library and Information Centre: An Overview.” International Journal of Library and Information Science 5 (4): 90–6. Search in Google Scholar

Roopa, E., and M. Krishnamurthy. 2015. “Perspectives of Digital Library Services: A Review.” International Journal of Next Generation Library and Technology 1 (1): 1–19, http://www.ijnglt.com/files/Perspective%20of%20digital%20library%20services.pdf . Search in Google Scholar

Sadiku, M. N. O. 2017. “Digital Preservation.” International Journal of Advanced Research in Computer Engineering 7 (4): 5–6, https://doi.org/10.23956/ijarcsse.v7i8.25 . Search in Google Scholar

Sathyanarayana, N. V. 2013. “Collection Development in the E-Content World: Challenges of Procurement, Access and Preservation.” DESIDOC Journal of Library and Information Technology 33 (62): 109–13, https://doi.org/10.14429/djlit.33.2.4198 . Search in Google Scholar

Shaikh, T. A. A. M. 2015. “Digitization of Library: Services, Challenges and Preservation of Digitized Information.” E-Library Science Research Journal 3 (11): 1–9, https://www.researchgate.net/publication/283571905 . Search in Google Scholar

Singh, S. 2018. “Digitization of Library Resources and the Formation of Digital Libraries: Special Reference in Green Stone Digital Library Software.” IP Indian Journal of Library Science and Information Technology 3 (1): 44–8. 10.18231/2456-9623.2018.0010 Search in Google Scholar

Stiller, J., and V. Petras. 2018. “Learning from Digital Library Evaluations: The Europeana Case.” ABI Technik 38 (1): 37–45, https://doi.org/10.1515/abitech-2018-0006 . Search in Google Scholar

Thomas, B. 2015. “The Issues and Challenges Facing a Digital Library with a Special Focus on the University of Calgary.” Library Review 64 (8–9): 533–51, https://doi.org/10.1108/lr-11-2014-0125 . Search in Google Scholar

Toyo, O. D. 2017. “Library Resources Digitization and its Impact on the Services of Academic Libraries: The Case of John Harris Library, University of Benin.” International Journal of Education and Evaluation 3 (7): 33–48. Search in Google Scholar

Tramboo, S., Humma, S. M. Shafi, and S. Gul. 2012. “A Study on the Open Source Digital Library Software’s: Special Reference to DSpace, Eprints and Greenstone.” International Journal of Computer Applications 59 (16): 1–9, https://doi.org/10.5120/9629-4272 . Search in Google Scholar

Trivedi, M. 2010. “Digital Libraries: Functionality, Usability, and Accessibility.” Library Philosophy and Practice 381, https://digitalcommons.unl.edu/libphilprac/381 . Search in Google Scholar

Tsakonas, G., and C. Papatheodorou. 2011. “An Ontological Representation of the Digital Library Evaluation Domain.” Journal of the American Society for Information Science and Technology 62 (8): 1577–93, https://doi.org/10.1002/asi.21559 . Search in Google Scholar

Velmurugan, C. 2013. “Research on Digital Preservation (2013). Issues and Challenges on Libraries and Information Resource Centres in India.” e-Library Science Research Journal 1 (7): 1–8, https://www.researchgate.net/publication/270214770 . Search in Google Scholar

Venkadesan, S. 2010. “Digital Preservation of Electronic Resources.” INDEST (January). Search in Google Scholar

Vullo, G. 2010. “A Global Approach to Digital Library Evaluation.” LIBER Quarterly 20 (2): 169–178, https://doi.org/10.18352/lq.7985 . Search in Google Scholar

Walters, T. O. 2010. “Economics, Sustainability, and the Cooperative Model in Digital Preservation.” Library Hi Tech 28 (2): 259–72, https://doi.org/10.1108/07378831011047668 . Search in Google Scholar

Wei, Z. 2011. “Research on the Application of Open Source Software in Digital Library.” Procedia Engineering 15: 1662–7, https://doi.org/10.1016/j.proeng.2011.08.310 . Search in Google Scholar

Xie, I. 2014. Digital Library Evaluation Criteria: What Do Users Want? Also available at https://www.researchgate.net/publication/264191174_Digital_Library_Evaluation_Criteria_What_do_Users_Want . 10.14699/kbiblia.2014.25.1.005 Search in Google Scholar

Zaveri, P. 2015. “Digital Disaster Management in Libraries in India.” Library Hi Tech 33 (2): 230–44, https://doi.org/10.1108/lht-09-2014-0090 . Search in Google Scholar

Zhang, Y. 2010. “Developing a Holistic Model for Digital Library Evaluation.” Journal of the American Society for Information Science and Technology 61 (1): 88–110, https://doi.org/10.1002/asi.21220 . Search in Google Scholar

© 2022 Walter de Gruyter GmbH, Berlin/Boston

• X / Twitter

Supplementary Materials

Please login or register with De Gruyter to order this product.

Journal and Issue

Articles in the same issue.

Current Research on Theory and Practice of Digital Libraries: Best Papers from TPDL 2019 & 2020

• Open access
• Published: 28 February 2022
• Volume 23 , pages 1–3, ( 2022 )

Cite this article

You have full access to this open access article

• Trond Aalberg 1 ,
• Fabien Duchateau 2 ,
• Mark Hall 3 ,
• Tanja Merc̆un 4 &
• Thomas Risse   ORCID: orcid.org/0000-0001-6248-1709 5  

2550 Accesses

Explore all metrics

This volume presents a special issue on selected papers from the 2019 & 2020 editions of the International Conference on Theory and Practice of Digital Libraries (TPDL). They cover different research areas within Digital Libraries, from Ontology and Linked Data to quality in Web Archives and Topic Detection. We first provide a brief overview of both TPDL editions, and we introduce the selected papers.

Explore related subjects

• Artificial Intelligence

Avoid common mistakes on your manuscript.

1 Overview of TPDL 2019 & 2020 editions

The TPDL conference constitutes a leading scientific forum that brings together researchers, developers, content providers, and practitioners in the field of Digital Libraries. Digital libraries and repositories store, manage, represent, and disseminate rich and heterogeneous data that are often of enormous cultural, scientific, educational, artistic, and social value. Serving as digital ecosystems for empowering researchers and practitioners they provide unparalleled opportunities for novel knowledge extraction and discovery. New applications raise novel challenges that can only be addressed in an interdisciplinary community of researchers and practitioners from various disciplines including the Digital Humanities, Information Sciences, and others.

The 23 \(^{\mathrm{rd}}\) edition of TPDL took place at the Oslo Metropolitan University, Norway, on September 9–12, 2019. The general theme was connecting with communities to reflect the need for digital libraries to interact with researchers and practitioners for effective data utilization, management, and exploitation. The proceedings contain 17 long research papers, 11 short research papers, and 18 poster and demonstration papers [ 2 ]. The 24 \(^{\mathrm{th}}\) edition of TPDL was special for two reasons. First, it should have taken place at the Université de Lyon, France, from August 25 to 28, 2020, but due to the COVID-19 pandemic, the event was organized online from August 25 to 27, 2020 [ 5 ]. Secondly, it was held jointly with two conferences in Information Systems, namely the 24 \(^{\mathrm{th}}\) European Conference on Advances in Databases and Information Systems (ADBIS) and the 16 \(^{\mathrm{th}}\) French EDA days on Business Intelligence & Big Data. The proceedings feature 14 long research papers and 4 short papers, whose topics span from knowledge graphs, linked data and ontology design to user studies, digital cultural heritage, and research data management [ 4 ].

Following the tradition of previous TPDL editions [ 3 , 6 ], the Program and General chairs decided to prepare a volume consisting of extended versions of the best papers from TPDL 2019 & 2020. This gives invited authors an opportunity to add details or new contributions and to describe additional experiments or studies, but also to consider perspectives suggested by reviewers. For the Digital Library community, such a volume is useful to establish the progress made so far in our field through mature works as well as to identify future research directions.

2 Presentation of Selected Papers

Papers accepted in both editions of TPDL had already been thoroughly reviewed by three reviewers and one senior meta-reviewer. Additionally, all papers were discussed during a meeting of the Program and General chairs for producing the list of TPDL accepted papers. From the pool of papers accepted in TPDL 2019 & 2020, the General and the Program Committee Chairs nominated ten papers based on their reviewing scores: three from TPDL 2019 and seven from TPDL 2020. Authors of these papers were invited to extend their original paper by at least 30% in a four-months period. Nine new versions were submitted. As required by IJDL’s policy, these extended papers went through another round of reviews by at least three reviewers, supported by a senior reviewer. Finally, 6 papers were accepted for publication in this special issue. Below is a short description of these papers.

The paper An Extended Analysis of the Persistence of Persistent Identifiers of the Scholarly Web by Martin Klein and Lyudmila Balakireva investigates the notion of persistence of DOIs by analyzing their resolution on the web. Digital Object Identifiers (DOIs) are common standard to persistently identify resources. The DOI concept is based on the assumption that mappings between the resources and the DOIs are updated when the resource location is changing. The authors show that persistence is in reality not guaranteed. Instead content providers respond differently to varying request methods and network environments and even change their response to requests against the same DOI. The paper presents quantitative analysis results and aims informing the scholarly communication community about this disconcerting lack of consistency.

Brenda Reyes Ayala presents Correspondence as the Primary Measure of Information Quality for Web Archives: A Human-Centered Grounded Theory Study , a grounded theory of quality specifically for web archives. She analyzed support tickets submitted by clients of the Internet Archive’s Archive-It (AIT). Therefore, the theory is human-centered and grounded on the perception of users and creators on the quality of web archives. The resulting theory compromises the dimensions correspondence, relevance, and archivability. Web archivists and cultural heritage institutions will benefit from the clarified notion of quality in a web archive.

The next paper by Arthur Brack, Anett Hoppe, Markus Stocker, Sören Auer, and Ralph Ewerth on Analysing the Requirements for an Open Research Knowledge Graph: Use Cases, Quality Requirements and Construction Strategies tackles the issue of exploring and comparing scientific research literature in a semantic way using an Open Research Knowledge Graph (ORKG). They first describe the typical tasks of a researcher (use cases) that can be supported by an ORKG, such as finding related work or obtaining a deep understanding of a paper. Minimal requirements (in terms of granularity or completeness for instance) are defined for each use case. Next, manual, semi-automatic, and automatic approaches for building an ORKG are surveyed, both for designing ontologies and for populating the graph. This article thus provides a broad overview of the different requirements for an ORKG, and it fosters future research on this emerging topic.

The VeTo+: improved expert set expansion in academia article by Serafeim Chatzopoulos, Thanasis Vergoulis, Theodore Dalamagas, and Christos Tryfonopoulos extends their work on identifying experts in a given field, for example, in order to identify new reviewers for a conference. Rather than relying on data containing explicit links between experts and topics, the authors identify potential experts by considering the venues the experts publish in and the topics attached to the publications. The extended algorithm presented here builds on this by enabling dynamic weighting of the two factors and filtering of the considered publication venues. The second is of particular interest to interdisciplinary venues, such as TPDL or JCDL, where except for the interdisciplinary venue, there is often little publishing overlap between experts.

The following paper Multi-Label Classification of Legislative Contents with Hierarchical Label Attention Networks by Danielle Caled, Mário J. Silva, Bruno Martins, and Miguel Won deals with the issue of annotating legislative documents using the EuroVoc hierarchical thesaurus. Contrary to existing annotation methods, authors propose a deep learning model to take into account the three levels of the thesaurus rather than predicting for a single level. In order to evaluate this approach, more than 200,000 legal documents in Portuguese, which have been classified according to EuroVoc, are integrated into the dataset EUR-Lex PT. This article finally highlights recurring challenges for automatic annotation of large collection of documents, such as the long tail label distribution which degrades accuracy.

Angelo Salatino, Francesco Osborne, and Enrico Motta’s paper CSO Classifier 3.0: A Scalable Unsupervised Method for Classifying Documents in Terms of Research Topics extends their work presented at TPDL 2019 by introducing an improved design of the Computer Science Ontology (CSO) Classifier for automatic classification of research papers according to the CSO and presenting novel mechanisms for detecting outlier topics. To measure the performance of their solutions, the authors created a gold standard using 70 documents, manually annotated by domain experts. The paper also describes how the classifier has been adopted by other researchers since 2019 and how the classifier could be adapted for classifying with other knowledge organization systems as the code is freely available to the wider research community.

To conclude, we believe that this volume of selected Digital Libraries papers reflects the ongoing work and topic trends in our field. We hope that our readers will find them insightful, and we invite them to take note about the next editions of TPDL (online in 2021 [ 1 ] and in Padova in 2022).

Berget, G., Hall, M.M., Brenn, D., Kumpulainen, S. (eds.): Linking Theory and Practice of Digital Libraries - 25th International Conference on Theory and Practice of Digital Libraries, TPDL 2021, Virtual Event, September 13-17, 2021, Proceedings, Lecture Notes in Computer Science , vol. 12866. Springer (2021). https://doi.org/10.1007/978-3-030-86324-1

Doucet, A., Isaac, A., Golub, K., Aalberg, T., Jatowt, A. (eds.): Digital Libraries for Open Knowledge - 23rd International Conference on Theory and Practice of Digital Libraries, TPDL 2019, Oslo, Norway, September 9-12, 2019, Proceedings, Lecture Notes in Computer Science , vol. 11799. Springer (2019). https://doi.org/10.1007/978-3-030-30760-8

Frommholz, I., Rauber, A., Sebastiani, F.: Guest editors’ introduction to the focussed issue on the 14th european conference on digital libraries (ECDL 2010). Int. J. Digit. Libr. 11 (2), 57–58 (2010). https://doi.org/10.1007/s00799-011-0068-6

Article   Google Scholar  

Hall, M.M., Mercun, T., Risse, T., Duchateau, F. (eds.): Digital Libraries for Open Knowledge - 24th International Conference on Theory and Practice of Digital Libraries, TPDL 2020, Lyon, France, August 25-27, 2020, Proceedings, Lecture Notes in Computer Science , vol. 12246. Springer (2020). https://doi.org/10.1007/978-3-030-54956-5

Sawadogo, P., Darmont, J., Duchateau, F.: Feedback from the participants of the adbis, TPDL and EDA 2020 joint conferences. CoRR arXiv:2012.01184 (2020)

Tsakonas, G., Kamps, J.: Current research on theory and practice of digital libraries: best papers from TPDL 2017. Int. J. Digit. Libr. 21 (1), 1–3 (2020). https://doi.org/10.1007/s00799-020-00278-4

Download references

Acknowledgements

This special issue would not have been possible without the contributions from many colleagues. We would like to thank the organizers of TPDL 2019 & 2020, the IJDL editorial team, in particular Ingo Frommholz and Nabil Adam, the reviewers for their insightful feedback, and the authors for submitting high-quality extended papers.

Open Access funding enabled and organized by Projekt DEAL.

Author information

Authors and affiliations.

Oslomet, Oslo, Norway and NTNU, Trondheim, Norway

Trond Aalberg

University Lyon, UCBL, CNRS, INSA Lyon, LIRIS, UMR5205, F-69622, Villeurbanne, France

Fabien Duchateau

The Open University, Milton Keynes, UK

University of Ljubljana, Ljubljana, Slovenia

Tanja Merc̆un

Goethe University Frankfurt, University Library J. C. Senckenberg, Frankfurt, Germany

Thomas Risse

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Thomas Risse .

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Aalberg, T., Duchateau, F., Hall, M. et al. Current Research on Theory and Practice of Digital Libraries: Best Papers from TPDL 2019 & 2020. Int J Digit Libr 23 , 1–3 (2022). https://doi.org/10.1007/s00799-022-00322-5

Download citation

Published : 28 February 2022

Issue Date : March 2022

DOI : https://doi.org/10.1007/s00799-022-00322-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Digital libraries
• TPDL conference
• Special issue
• Find a journal
• Publish with us
• Track your research

Open Source Digital Library Software: A Literature Review

Proceedings of the National Seminar on 'Preservation and Conservation of Information Resources in Knowledge Society: Issues, Challenges and Trends', pp. 238-258, Manipur University, Canchipur, Imphal, March 3-4, 2009

18 Pages Posted: 13 Dec 2019

Hanadashisha Warr

Affiliation not provided to ssrn, prof p hangsing.

North-Eastern Hill University (NEHU) - Department of Library and Information Science

Date Written: 2009

Digital Library initiatives are gaining momentum and many open source digital library software emerged. These developments necessitate universal standard for digital library software. Conforming the software to universal standard means compromising local requirements. No matter how general the software may be, suitability of the digital library software to specific requirements of digital library project will differ from software to software. These variations in digital library software allow ample room for librarians to assess the digital library softwares with the local requirement criterion. To ease assessment/selection process, this paper attempts to evaluate four most popular open source digital library software. Various features of the softwares are comparatively tabled for ready reference.

Keywords: Digital Library, Digital Library Software, Open Source Digital Library Software, Library Digitization, Digital Archive

Suggested Citation: Suggested Citation

Paokholun Hangsing (Contact Author)

North-eastern hill university (nehu) - department of library and information science ( email ).

North-Eastern Hill University Shillong, ME India

Do you have a job opening that you would like to promote on SSRN?

Paper statistics, related ejournals, io: productivity, innovation & technology ejournal.

Subscribe to this fee journal for more curated articles on this topic

Writing Technologies eJournal

Subscribe to this free journal for more curated articles on this topic

Legal Information, Technology & Law Librarianship eJournal

Economic & social impacts of innovation ejournal, diffusion of innovation ejournal, information technology & systems ejournal, information systems: behavioral & social methods ejournal, international political economy: globalization ejournal, information curation, management & organization ejournal, library services & librarianship ejournal.

• Other Journals

An important role for librarians over the next five to ten years is to provide access to online library resources—free, open-access, or purchased, all valuable resources—in an intuitive, easy-to-use one-stop shop and not to be afraid of running a continual beta test in which new services and functions can be added when necessary. To fill this role, librarians and electronic resources managers need flexible, interoperable resource-discovery systems based on open-source software. In addition, we must continue to assess users’ needs and reach out by adapting our systems to fit their requirements, rather than expecting them to come to us; indeed, our very future depends on it. 1

Two decades after the advent of electronic journals and databases, librarians are still grappling with ways to best manage e-resources in conjunction with traditional print resources and at the same time explore new purchasing initiatives and practices, such as demand-driven acquisition of electronic books. In addition, these times of economic austerity are creating budgetary pressures at many institutions of higher education, resulting in librarians having to justify their spending on collections and resource management more than ever.

Techniques for Electronic Resource Management (TERMS) began in 2008 after a discussion about electronic resource management (ERM), current ERM tools, and what was lacking both in current practice and with the systems available. TERMS expands on Pesch’s electronic resources life cycle (see figure 1.1 ) and seeks to become a reference point for those who are new to ERM, those who have suddenly shifted job functions to oversee ERM, and those who may want to implement its recommendations of best practice.

TERMS Tumblr blog http://6terms.tumblr.com TERMS Facebook group https://www.facebook.com/groups/174086169332439 6TERMS on Twitter https://twitter.com/6terms

• TERMS 1: Investigating New Content for purchase/addition, assigned to Ann Kucera (Baker College, Michigan)
• TERMS 2: Acquiring New Content, assigned to Nathan Hosburgh (Montana State University, Montana)
• TERMS 3: Implementation, assigned to Stephen Buck (Dublin City University, Ireland)
• TERMS 4: Ongoing Evaluation and Access, assigned to Anita Wilcox (University College Cork, Ireland)
• TERMS 5: Annual Review, assigned to Anna Franca (King’s College, London, United Kingdom)
• TERMS 6: Cancellation and Replacement Review, assigned to Eugenia Beh (Texas A&M University, Texas)

TERMS Wiki: Main Page http://library.hud.ac.uk/wikiterms/Main_Page

In 2012, TERMS was also endorsed by the Knowledge Base + project in the United Kingdom, which has a project deliverable of providing “workflow management tools related to the selection, review, renewal and cancellation of publications” 2 and has also received interest in the United States from GoKB from Kuali OLE (open library environment), “a community of nine research libraries working together to build the first open-source system designed by and for academic and research libraries for managing and delivering intellectual information.” 3

Electronic Resources and Libraries website www.electroniclibrarian.com

One of the first things to note when performing a literature review on ERM is that there are no independent literature reviews solely on this area. Instead, ERM has now become an integral part of standard literature reviews for acquisitions processing, collection development and management, cataloging and classification, and serials management. At the same time, there are areas of ERM that sit outside of these traditional functioning areas in libraries. This makes performing a literature review on ERM more of a challenge.

From the field of library acquisitions, the issues most readily identified in recent years have been the switch from print processing to ERM and the continued struggle to find management tools that work within the local library context. 6

“Simply put, collection management is the systemic, efficient and economic stewardship of library resources.” 7 The term collection development has been with us since the 1960s. 8 However, it is a constantly evolving area, and as the library collection moves from one dominated by print to one dominated by electronic resources, collection development policies may have been patched rather than redesigned to reflect the different emphasis on delivery. In a 2012 study, Mangrum and Pozzebon found that “over half of the libraries tried to address ER [electronic resources] in some way. However, most policies contain traditional language with a section on library ER inserted into the latter portion of the document.” 9

In regard to collection development and management trends, the two biggest growth areas are e-book purchasing and purchase-on-demand or patron-driven acquisition models. There has been an explosion in collection management literature on these two topics over the past three to four years. 10 A single place for best practices, or from which a local library can create its own localized best practices, is definitely needed: “Bleiler and Livingston stressed that a lack of established policies and procedures for assessment puts a library at risk for financial loss and recommended that libraries create selection policies and standardized methods for assessment, train staff for contract negotiation, and share strategies, policies, and best practices.” 11

In addition to changes to the format of delivery of library resources, libraries must also contend with the impact of today’s economic environment. Hazen suggests that libraries need to rethink their collection development in light of these issues and move from collection to collection and content, where content is “a category that encompasses everything to which a library enjoys ready physical or digital access regardless of ownership status [and] is central to all that we do.” 12

In 2001, Jewell reported on the selection, licensing, and support of online materials by research libraries and concluded that several libraries had developed local systems for acquiring, managing, and supporting electronic resources. 13 Jewell’s report was followed in 2004 by a report from the Digital Library Federation’s (DLF) Electronic Resource Management Initiative (ERMI), which “was organized to support the rapid development of such systems by producing a series of interrelated documents to define needs and to help establish data standards.” 14 The report went on to provide a road map for ERM.

On the back end, we continue to rely on methods developed when we had 250 rather than 25,000 eresources. Information on our electronic resources is currently kept in paper files (license agreements), Excel spreadsheets (vendor contact information and administrative passwords), staff web pages (usage statistics), small databases (trial and decision tracking, divisional library resources, technical problem reports), SFX (ejournal holdings), and our ILS (acquisitions and payment data). Few of these systems are connected to each other; in some cases, information is readily accessible only to one or a few individuals, not by intent, but by the limitations of the storage mechanism. Many procedures are not documented and rely on informal channels of communication. 15

More recently there have been a number of open-source and community ERM systems, such as CORAL 16 and CUFTS, developed by Simon Fraser University (SFU) and implemented by SFU and the University of Prince Edward Island, which view this “technology not necessarily as a way of spending less money, but spending money more wisely.” 17

Another growing area of ERM is the work being performed to develop a suite of standards to support the vast amount of access and management knowledge and myriad of tools needed to maintain adequate access to electronic resources. 18 According to Sarah Glasser, “KBART and IOTA are both working to decrease OpenURL link failures that are caused by metadata deficiencies.” In addition, “PIE-J differs from KBART and IOTA because it is not focused on link resolver errors. Formed by NISO in 2010, PIE-J addresses access barriers that arise from the manner in which electronic journals are presented on provider websites.” 19

There has been a lot of discussion about the implementation of ERM systems in recent years. 20 However, use of these systems is still far from ubiquitous, and many academic libraries have yet to implement or even purchase a system. “A risk of ERMS implementations, more talked about than written about … was that the costs (in added work) to maintain a new system would outweigh the value of the added functionality.” 21 Despite early expectations, Collins and Grogg see the current crop of ERM systems as “less like a silver bullet and more like a round of buckshot.” 22

One of the most time-consuming parts of an ERM implementation is analyzing licenses and inputting them into the relevant fields of an ERM system in order for them to be meaningful to librarians and patrons. The University of Northern Colorado has developed an in-house system to perform license mapping that “makes information that is often deeply embedded within a license readily available to library personnel who could use such information in the daily operations of the library. This information is useful to any library that maintains license agreements for electronic resources.” 23

A panel session at the 2010 NASIG conference concluded that the “ERM system at UC has not solved all their problems, but some improvements have been realized. Budget tracking and staffing continue to be challenges. A final determination of the effectiveness will not be evident until the system becomes a part of the general staff workflow and not considered as something extra.” 24

Collins and Grogg cited workflow management as number one in librarians’ top six ERM priorities. They found that “over a third of librarians surveyed prioritized workflow or communications management, and they called it one of the biggest deficiencies (and disappointments) of ERMS functionality.” 25 This area has also been highlighted by the National Information Standards Organization (NISO), which has created a working group, ERM Data Standards and Best Practices Review, to undertake a gap analysis regarding ERM. 26

In the United Kingdom, the Managing Electronic Resource Issues (MERI) project at the University of Salford aimed “to produce a use case of ERM systems and a preliminary set of requirements for an electronic resource management system, for use by the University of Salford and other HE institutions and system suppliers.” 27 The requirements document from this project went on to inform the SCONUL shared ERM requirements project. An output of these projects was a set of workflows that describe the various processes involved in managing electronic resources. 28 The University of Huddersfield was one of the sixteen UK universities to take part, and like others, had never actually recorded these workflows until asked to do so by the project. All project members found that by recording workflows, they were able to take advantage of efficiencies discovered as part of documenting the process.

One of the objectives of the TERMS blog and wiki was to collect a number of e-resource workflows from a variety of different types of libraries. Both the University of Huddersfield and Portland State University shared their workflows as part of TERMS. The release of the six TERMS via the blog also encouraged other universities to share their workflows and discuss efficiencies; indeed, “rethinking e-resources workflows and developing practical tools to streamline and enhance various inelegant processes have become the priorities.” 29

Since the launch of the first draft of TERMS, the project has now attracted interest in various workflows from different libraries around the world, including the University of Cork, Duke University, Florida Gulf Coast University, and Texas A&M University.

A recent press release by Jisc in the United Kingdom suggested that international collaboration is needed to transform ERM in libraries—“Many of the concerns libraries have in the management of electronic resources are the same across the world”—and that projects such as GoKB and the Knowledge Base + service in the United Kingdom “are exploring community-based solutions.” 30

TERMS Wiki Investigating New Content for Purchase/Addition http://library.hud.ac.uk/wikiterms/Investigating_New_Content_for_purchase/addition Acquiring New Content http://library.hud.ac.uk/wikiterms/Acquiring_New_Content Implementation http://library.hud.ac.uk/wikiterms/Implementation Ongoing Evaluation and Access http://library.hud.ac.uk/wikiterms/Ongoing_Evaluation_and_Access Annual Review http://library.hud.ac.uk/wikiterms/Annual_Review Cancellation and Replacement Review http://library.hud.ac.uk/wikiterms/Cancellation_and_Replacement_Review

• There are currently no refbacks.

• Corpus ID: 14487517

Digital Libraries – Literature Review. 1. Section a –digital Libraries 2.2 Standard Techniques System Contains the Attribute Groups User, Information Access, System Structure and Document, Which Considers User Interaction from a System Perspective. Collection Contains the Attribute Groups Content, M

• Stacey Greenaway
• Published 2007
• Computer Science

One Citation

The application of workflows to digital heritage systems, 38 references, a survey of user studies for digital libraries, digital library attributes: framing usability research, evaluating digital libraries, analytical usability evaluation for digital libraries: a case study, principles for digital library development, topic-based browsing within a digital library using keyphrases, evaluation of digital libraries: an overview, understanding users' experiences: evaluation of digital libraries, tailoring claims analysis to the design and deployment of digital libraries : a case study, an experimental framework for comparative digital library evaluation: the logging scheme, related papers.

Showing 1 through 3 of 0 Related Papers

Intelligent libraries: a review on expert systems, artificial intelligence, and robot

Library Hi Tech

ISSN : 0737-8831

Article publication date: 30 June 2020

Issue publication date: 21 June 2021

This paper reviews literature on the application of intelligent systems in the libraries with a special issue on the ES/AI and Robot. Also, it introduces the potential of libraries to use intelligent systems, especially ES/AI and robots.

Design/methodology/approach

Descriptive and content review methods are applied, and the researchers critically reviewed the articles related to library ESs and robots from Web of Science as a general database and Emerald as a specific database in library and information science from 2007–2017. Four scopes considered to classify the articles as technology, service, user and resource. It is found that published researches on the intelligent systems have contributed to many librarian purposes like library technical services like the organization of information resources, storage and retrieval of information resources, library public services as reference services, information desk and other purposes.

A review of the previous studies shows that ESs are a useable intelligent system in library and information science that mimic librarian expert’s behaviors to support decision making and management. Also, it is shown that the current information systems have a high potential to be improved by integration with AI technologies. In this researches, librarian robots mostly designed for detection and replacing books on the shelf. Improving the technology of gripping, localizing and human-robot interaction are the main concern in recent librarian robot research. Our conclusion is that we need to develop research in the area of smart resources.

Originality/value

This study has a new approach to the literature review in this area. We compared the published papers in the field of ES/AI and robot and library from two databases, general and specific.

• Library system
• Intelligent systems
• Artificial Intelligent (AI)
• Intelligent library
• Smart library
• Expert System (ES)

Asemi, A. , Ko, A. and Nowkarizi, M. (2021), "Intelligent libraries: a review on expert systems, artificial intelligence, and robot", Library Hi Tech , Vol. 39 No. 2, pp. 412-434. https://doi.org/10.1108/LHT-02-2020-0038

Emerald Publishing Limited

Copyright © 2020, Asefeh Asemi, Andrea Ko and Mohsen Nowkarizi

Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial & non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at: http://creativecommons.org/licences/by/4.0/legalcode .

1. Introduction

Understanding the nature of the information needs and defining this need for the system,

Identifying information resources that are relevant to information needs,

Evaluation of existing information resources, evaluation of retrieved information,

Organizing existing information resources, organizing selected information from items retrieved,

Managing existing information resources, managing retrieved information,

Using existing information resources, using retrieved information,

Information and knowledge analysis,

Converting information to knowledge,

Dissemination and transfer of information and knowledge,

Interaction and exchange of information and knowledge.

In WoS, which articles did authors write about ES/AI and robots’ application in the library?

In Emerald Insight, which articles did authors write about ES/AI and robots’ application in the library?

2. Artificial intelligence

AI applies to different sciences. We can say in the library and information science, it more uses in scientific databases and library systems. Such as behavioral science, social sciences, psychology, management and library science and information science. It is related to some of the systems that apply different forms of intelligence such as learner systems, inferior systems, systems with natural language understanding or natural language interpretation, systems with visual scene perception and systems that perform other types of feat that require human types of intelligence ( Bavakutty and Salih, 2006 ). In this branch of the science that involves machines, solutions are utilized to solve complex problems of human behavior. We can present computer-based algorithms based on human behavior and knowledge in using systems. “It is an interdisciplinary field making use of concepts from various fields like cybernetics, information theory, psychology, linguistics, logic, etc. it can use to simulate human behavior and for computer ailed instruction, ES, robots and for NLP. It can also use for Intelligent Retrieval from databases” ( Bavakutty and Salih, 2006 ). In this way, computer software and the use of various computer-based products help in the operation of various types of libraries and their public services and the generation of output products. Automation implies the degree of mechanization where the routines and receptive jobs or operations are left to be performed by machines with little or no intervention by human beings. Lesser the degree of human intervention, greater the degree of automation; this does not mean that automation does away with human beings. On the contrary, human beings are relieved of routine chores, giving them more time for tasks, which require their intelligence. In view of the various features of a modern computer system, we find that it has been applied in several areas of library work. Book acquisitions, cataloging, serials control, and circulation, information retrieval and dissemination, interlibrary loan, cooperative acquisition and cataloging have been automated in the library ( Lakshmikant and Vishnu, 2008 ).

3. Intelligent systems

Intelligent systems (ISs) are defined as any formal or informal system that is able to obtain and process data, to interpret the data by applying technologies of artificial intelligence and business intelligence and to provide reasoned judgments based on that to decision makers as a basis for action ( Sharda et al. , 2017 ).

ISs are computer-based systems that help in the task of subject indexing can be thought of as an ES ( Lancaster, 1997 ). Lancaster has a clear statement relating to the scope of AI: “Computer programs have been developed, which exhibit human-like reasoning, which may be able to learn from their mistakes and which quickly and cleverly perform tasks normally done by scarce and expensive human experts.” AI has a wide application area. Figure 1 gives a good idea of this coverage.

Technologies that are frequently used in intelligent systems: machine learning, case-based reasoning, genetic algorithms, fuzzy logic and natural language processing (NLP).

NLP is another facility of an intelligent system that can use to retrieve information needs from different scientific databases. In the information retrieval process, the user can state his information requirement in natural language, making the searching more easily and fruitful this allows users to state complex retrieval languages ( Bavakutty and Salih, 2006 ).

Business intelligence (BI) as is the set of techniques and tools for the transformation of raw data into meaningful and useful information for business analysis/decision support purposes ( Sharda et al , 2017 ). BI solutions include data access, storage, data analysis and visualization technologies to support better decision making.

4. Expert systems

Expert systems (ESs) are computer-based systems that simulate human decision making. They can integrate with information systems to improve their accuracy and performance ( Singh et al. , 1996 ). Various librarian ES has been developed. Waters (1986) designed the National Agricultural Library’s microcomputer-based ES to help users obtain answers to simple reference questions. In general, they ask questions from the user and take the user’s answer as input, then explain the rationale for decision results. In general, these systems consist of two main elements: A knowledge base and inference engine. The knowledge base encompasses all the information needs that human/librarian experts are using to decide. This information is present in the knowledge base as facts and rules. ESs can make much better decisions than librarian decision makers because their knowledge base can involve the experiences of a team of the best experts. The manner of librarian experts to make decisions is emulated for the design rules of the knowledge base. The rules are consisting of two main phases: “if phase” and “then phase.” The “if phase” is consisting of conditions, and the “then phase” is consisting of results. ESs are distinguished from other computer systems with the application of reasoning through the inference engine. The inference engine simulates human decision makings based on the knowledge base and a rule base ( Figure 2 ).

Knowledge-based indexing ( Amin and Razmi, 2009 );

Natural Language Processing and abstracting ( Albayrak and Erensal, 2004 );

Reference work ( Amin and Razmi, 2009 );

Cataloging ( Weiss, 1994 ) and ( Amin and Razmi, 2009 );

Online information retrieval ( Bellman and Zadeh, 1970 ), ( Sacchanand and Jaroenpuntaruk, 2006 ) and ( Bavakutty and Salih, 2006 );

Using intelligent interfaces in online information storage and retrieval systems;

Information needs analysis and representation, including different services, such as classification, indexing and abstracting;

Reference services;

Development of collection;

Hypertext and hypermedia ( Bavakutty and Salih, 2006 ).

5. Methodology

Descriptive and content review methods are applied to the study. The researchers critically reviewed the articles related to ES/AI and robots in the library. According to this review, the application of ES/AI and robots classified as the following:

Technology : The articles surveyed and evaluated the information management systems in the libraries belongs to this group. These articles relate to usability and implementation. They do not propose or propose an information system or model.

Resource : These articles related to information resources. This category may include the selection, acquisition and use of information resources.

User / End-user : Existing information and knowledge systems/models are usually working based on the opinion of experts/users and end-user behavior. Therefore, applying ES technologies such as inference engine and fact/rule base will improve the performance and accuracy of considered systems.

Service : The articles in this group have proposed an ES or related technology and methods that can be connected and included in ESs to present public or technical services. The public services present to end-users to fulfill their information needs and technical services present to the librarians or any professional user in library activities.

6. Findings

The purpose of the study is to review the articles on intelligent libraries and the use of ES/AI and robots in the libraries. Based on the research questions, the findings presented in two sections. The first section is related to the review of the articles in WoS as a general database in different subjects. The second section is related to the review of the articles in Emerald as a professional database in the Library and Information Science. According to this review, the application of ES/AI and robots classified into four classes such as technology, resource, user/end-user and service.

6.1 ES/ AI and robots’ application in the library (WoS)

The topics of “expert system” and “library” were searched in the WoS database on 10th Oct 2017. We found 1,208 documents related to this topic. Then we have refined the topics through “Research Area,” “Document Type.” In the research area, we selected the area of “Information Science, Library Science.” We chose “article” for “document type” and excluded unrelated articles. Finally, found 14 articles as a result, which are shown in Table 1 .

The review of papers shows the fading of the ES/AI in recent studies. It is found that the majority (46%) of the paper worked on the experts’/users’ behavior. This is even though no research has been done on the use of intelligent resources using ESs between the years 2007–2017 on the WoS ( Figure 3 ). However, the studies that are related to information systems have a closed relation with the knowledge and opinion of experts. Using ES technologies such as inference engine and fuzzy rule base may increase the accuracy of them. Therefore, the current information systems can be improved by integration with ES technologies. ESs use in intelligent libraries. In general, the information provided to users in a library leads to a change in the behavior of the user’s knowledge and creates learning. The intelligent library uses an appropriate protocol for the exchange of information. This protocol is unique, and it is designed to help, confirm or perform the terms of the agreement. The terms of the agreement include a series of guidelines that will be carried out automatically. These guidelines relate to information sources, services, and technology for distributing and exchanging information. For operating a smart library, resources and services must be available under the agreement. All users must use the digital signature and agree to the terms of the agreement. Smart libraries can exchange information based on the internet of Things (IoT).

Recently the researchers tried to increase the ability of librarian robots by applying the new methods. We searched for the topic of “Librarian robot” using WoS on 10th Oct 2017. Then we limited the results to the duration of 2007–2017. We excluded unrelated articles and finally found 15 articles and proceeding papers as a result, which is shown in Table 2 . In this table, we determine the research area related to applied methodologies of papers in the “Research area of publication source.” A summary of the applied method is explained in “Method,” and the main contribution of papers is mentioned in “contribution.”

The most recent papers that are related to librarian robots are in the area of service ( Figure 4 ). Improving the technology of gripping, localizing and human-robot interaction are the most discussed issues in librarian robots. Librarian robots can be used in large libraries. This robot reduces a lot of common and duplicate activities in different places of the library, especially at the library’s repository. For example, this robot can be helpful in shelf-reading activity. There are some imaginations that the use of librarian robot creates a gap between information and people. Smart libraries and librarian robots are always faced with this challenge. But not a way out of using new technologies, because the development of information does not coincide with the development of expert human resources. In many libraries, librarian robots can be helpful in solving library problems. Only the small number of the studies are related to resources. It is shown that we need to develop our research in this area.

The library should take special care of every aspect related to the man-machine interface: favoring systems standardization, avoiding the accumulation of different equipment, using a clear, brief and direct language, including images and sound, representing reality and reflecting the human mental patterns ( De Prado, 2000 ). AI techniques such as genetic algorithms, artificial neural networks, ESs, and fuzzy logic or hybrid methods can improve librarian robots to reflect human mental patterns.

6.2 ES and robot’s application in the library using Emerald Insight [1]

Table 3 shows the review of the papers in the field of ES/AI and robots, and library exported from the Emerald Insight as a specific database in the library and information science.

Figure 5 shows the most recent papers exported from Emerald Insight, which are related to ES/AI and robot in the library are in the area of service. The finding is the same as the exported papers from the WoS database.

The following trends show a line graph of the relative frequencies across the main category in the abstracts of the articles ( Figure 6 ). The thematic interaction was observed in the main categories of the articles based on their keywords. Most common categories in the abstracts are digital, information, library, search, and user.

Figure 7 shows a line graph of the relative frequencies across the main category of the keywords of the articles. The thematic interaction was observed in the main categories of the articles based on their keywords. Most common categories in the abstracts are digital, information, Internet, library, and systems.

7. Discussion

The ES should be considered only when development is: “possible,” “appropriate,” and “justified” ( Lancaster, 1997 ). This question must be answered before we initialed an ES project. Waters (1986) gives some good guidelines on when we should consider using ESs. An ES has received a lot of attention from the research community in the 1980s. Unfortunately, much of the writing sensationalized the field expectations dramatically ( Lakshmikant and Vishnu, 2008 ) fueled by public expectations began to over-promise misconceptions about what AI can and cannot do arise and they persist today. Many rushed into the field in search of quick answers and quick profits. Several Al researchers saw what was happing and feared a backlash. Once all the excitement wore off during 1988–90, things did begin to change some of the realities and limitations of the AI techniques became evident. An AI backlash has resulted in ascertaining to an extent, but fortunately, it has not been wide-scaled instead. The optimism remains with a better sense of realism than before, and both the benefits and limitations are better appreciated.

An expert of the problem available;

Experts have the time for the ES development project;

Experts can articulate their knowledge and methods;

The problem is not too complex, but knowledge intensive;

The problem is not poorly understood;

The problem requires cognitive skills only.

Reliable visual localization;

Robust and fault-tolerant force-guided extraction;

Performance adequate for books of different sizes and thicknesses;

Active book searching;

Combine navigation and active vision;

Fault-tolerant probabilistic strategy.

In the context of robot librarians and AI has been investigated by limited number of studies. In the database of Emerald, only one article was found ( Yao et al. , 2015 ). They introduced a collaborative library service based on artificial intelligence. They developed an intelligent robot called Xiaotu (female). The task of this robot is to provide online reference services. Four factors are important in the success of this robot: artificial intelligence, self-learning, vivid logo and language, and modular architecture ( Yao et al. , 2011 ). Yuehu and Yanqing (2012) studied using the internet technology of objects. They have tried to look at smart sets along with the robot librarian. Then compare the smart library with other libraries. Kyrarini et al. (2017) presented a framework called “Skill Robot Library” (SRL). This framework has the authority to store key points of the route. In fact, this robot can store user’s behavior in information retrieval, and it will work based on this stored behavior. Behan and O'Keeffe (2009) designed as a mobile robotic assistant, called “LUCAS” for the University of Limerick. This assistant is a help system that supports users intellectually. Kim and Kohtaro (2009) tried to provide robots based on the structured data. This study introduces a conventional and intelligent environment for a librarian robot. This environment is based on RFID technology for these systems. In another study, reference services were investigated using the instant messaging (IM) smart robot. The Shanghai Jiaotong University Library is presented for example. This library provides the IM robot’s intelligent library service using BotPlatform ( Yi et al. , 2011 ).

8. Conclusion

A review of the articles shows that we can use expert and intelligence systems in different library activities and information services. The main goal is to provide specialized services with the help of librarians and information resources specialists. Library services include technical and public services. Both categories use intelligent systems and ESs. These activities include the provision of information resources, the organization of information resources such as classification, indexing, and abstracting, the storage and retrieval of information from library systems, reference services, and circulation desk. We classified the scopes of the researches into four classes “technology,” “user,” “service,” and “resource.” A review of the articles shows that users’ information behavior is a very good way to design intelligent systems. The storing information in cloud and non-cloud spaces allow for the development of these systems. In big data and social networks where scientific information resources are exchanged, intelligent agents can play an important role. User profiles can be a good source for designing ES algorithms based on user knowledge. ESs are the most useable intelligent system in library and information science, which mimic librarian expert’s behaviors to support decision and management. However, individually using this technology is reduced in recent studies. Most information systems have a closed relation with the knowledge and opinion of experts. Using ES technologies such as inference engine and fuzzy rule base may increase the accuracy of them. Therefore, the current information systems can be improved by integration with ES technologies. The librarian robot can reduce the usual and repetitive activities on library shelves. Almost the third of the articles in Emerald Insight in ES have related to the “user” scope, and in librarian robot (18% in WoS), most of the articles have related to the "service" in Emerald Insight and in WoS as well. One of our conclusion is that we need to further research in the area of smart resources.

AI Coverage ( Lancaster, 1997 )

ES elements

The scope of the articles in the field of ES/AI and library (WoS)

The scope of the articles in the field of library and robot (WoS)

The scope of the articles in field ES/AI and Robot in the library (Emerald Insight)

Relative frequencies across the main categories in the abstracts of the articles and thematic interaction between them in the field of ES/AI and robot and library (Emerald Insight)

Relative frequencies across the main categories in the keywords of the articles and thematic interaction between them in the field of ES/AI and robot and library (Emerald Insight)

Articles related to ES/AI and library (WoS)

Articles related to library and robot (WoS)

Articles related to ES/AI and Robot in the library (Emerald Insight)

https://www.emerald.com/insight

Albayrak , E. and Erensal , Y.C. ( 2004 ), “ Using analytic hierarchy process (AHP) to improve human performance: an application of multiple criteria decision making problem ”, Journal of Intelligent Manufacturing , Vol. 15 No. 4 , pp. 491 - 503 , doi: 10.1023/B:JIMS.0000034112.00652.4c .

Amin , S.H. and Razmi , J. ( 2009 ), “ An integrated fuzzy model for supplier management: a case study of ISP selection and evaluation ”, Expert Systems with Applications , Vol. 36 No. 4 , pp. 8639 - 8648 , available at: http://isiarticles.com/bundles/Article/pre/pdf/19174.pdf .

Asemi , A. , Akbari , A. , Kheradmandnia , M. and Farazi , A. ( 2012 ), “ Investigating the level of desirability of information in management information systems in libraries in Iran ”, The Electronic Library , Vol. 30 No. 6 , pp. 833 - 843 , doi: 10.1108/02640471211282136 .

Ataman , B.K. ( 2009 ), “ Requirements for information professionals in a digital environment: some thoughts ”, Program Electronic Library and Information Systems , Vol. 43 No. 2 , pp. 215 - 228 , doi: 10.1108/00330330910954415 .

Bailey , C.W. Jr. ( 1991 ), “ Intelligent library systems: artificial intelligence technology and library automation systems ”, in Hewitt, J.A. (Ed.) Advances in Library Automation and Networking , Vol. 4, JAI Press, Greenwich, CT, available at: http://www.digital-scholarship.org/cwb/intlibs.pdf ( accessed 20 June 2018 ).

Bailey , C.W. ( 1992 ), “ The intelligent reference information system project. A merger of CD-ROM LAN and expert system technologies ”, Information Technology and Libraries , Vol. 11 No. 3 , p. 237 , available at: https://www.learntechlib.org/p/146739/ ( accessed 21 April 2019 ).

Bavakutty , M. and Salih , M. ( 2006 ), Research on Library Computerisation , 1st ed. , Ess Ess Publications .

Behan , J. and O'Keeffe , D.T. ( 2009 ), “ The development of an intelligent library assistant robot (Ireland) ”, Journal of Korea Robotics Society , Vol. 4 No. 2 , pp. 147 - 154 .

Bellman , R.E. and Zadeh , L.A. ( 1970 ), “ Decision-making in a fuzzy environment ”, Management Science , Vol. 17 No. 4 , pp. B141 - B164 , doi: 10.1287/mnsc.17.4.B141 .

Bi , Z. , Wang , G. , Xu , L.D. , Thompson , M. , Mir , R. , Nyikos , J. , Mane , A. , Witte , C. and Sidwell , C. ( 2017 ), “ IoT-based system for communication and coordination of football robot team ”, Internet Research , Vol. 27 No. 2 , pp. 162 - 181 , doi: 10.1108/IntR-02-2016-0056 .

Bilandzic , M. and Foth , M. ( 2013 ), “ Libraries as coworking spaces: understanding user motivations and perceived barriers to social learning ”, Library Hi Tech , Vol. 31 No. 2, pp. 254-273 , doi: 10.1108/07378831311329040 .

Black , E.L. ( 2011 ), “ Selecting a web content management system for an academic library website ”, Information Technology and Libraries , Vol. 30 No. 4 , pp. 185 - 189 , doi: 10.6017/ital.v30i4.1869 .

Borgman , C.L. ( 2000 ), “ Digital libraries and the continuum of scholarly communication ”, Journal of Documentation , Vol. 56 No. 4 , pp. 412 - 430 , doi: 10.1108/EUM0000000007121 .

Cao , G. , Liang , M. and Li , X. ( 2018 ), “ How to make the library smart? The conceptualization of the smart library ”, The Electronic Library , Vol. 36 No. 5, pp. 811-825 , doi: 10.1108/EL-11-2017-0248 .

Carneiro , A. ( 2001 ), “ The role of intelligent resources in knowledge management ”, Journal of Knowledge Management , Vol. 5 No. 4 , pp. 358 - 367 , doi: 10.1108/eum0000000006533 .

Chen , K. and Hsiang , J. ( 2009 ), “ The unique approach to institutional repository, practice of National University ”, The Electronic Library , Vol. 27 No. 2, pp. 204-221 , doi: 10.1108/02640470910947566 .

Chowdhury , G.G. ( 1999 ), “ The Internet and information retrieval research: a brief review ”, Journal of Documentation , Vol. 55 No. 2 , pp. 209 - 225 , doi: 10.1108/EUM0000000007144 .

Chu , H.C. , Hwang , G.J. and Tseng , J.C.R. ( 2010 ), “ An innovative approach for developing and employing electronic libraries to support context‐aware ubiquitous learning ”, The Electronic Library , Vol. 28 No. 6 , pp. 873 - 890 , doi: 10.1108/02640471011093552 .

Comsa , A. , Maniu , I. , Modler , N. , Lovasz , E.C. and Ciupe , V. ( 2014 ), “ Automated book manipulator in libraries ”, in Pisla , D. , Bleuler , H. , Rodic , A. , Vaida , C. and Pisla , A. (Eds), New Trends in Medical and Service Robots. Mechanisms and Machine Science , Vol. 16 , Springer , Cham , doi: 10.1007/978-3-319-01592-7_6 .

Connolly , C. ( 2008 ), “ Artificial intelligence and robotic hand-eye coordination ”, Industrial Robot: International Journal , Vol. 35 No. 6, pp. 496-503 , doi: 10.1108/01439910810909484 .

Cox , A.M. , Pinfield , S. and Rutter , S. ( 2018 ), “ The intelligent library: thought leaders' views on the likely impact of artificial intelligence on academic libraries ”, Library Hi Tech , Vol. 37 No. 3, pp. 418-435 , doi: 10.1108/LHT-08-2018-0105 .

Davenport , E. , Procter , R. and Goldenberg , A. ( 1997 ), “ Distributed expertise: remote reference service on a metropolitan area network ”, The Electronic Library , Vol. 15 No. 4 , pp. 271 - 278 , doi: 10.1108/eb045567de .

De Prado , R.L. ( 2000 ), “ Do users dream of electronic libraries? ”, The Electronic Library , Vol. 18 No. 3 , pp. 202 - 209 , doi: 10.1108/02640470010337508 .

Denning , R. and Smith , P.J. ( 1994 ), “ Interface design concepts in the development of Elsa, an intelligent electronic library search assistant ”, Information Technology and Libraries , Vol. 13 No. 2 , pp. 133 - 147 .

Devadason , F.J. and Vespry , H.A. ( 1996 ), “ LISPA (library and information center staff planning advisor): a microcomputer-based system ”, Information Technology and Libraries , Vol. 15 No. 2 , pp. 105 - 112 .

Dimić , B. , Milosavljević , B. and Surla , D. ( 2010 ), “ XML Schema for UNIMARC and MARC 21 ”, The Electronic Library , Vol. 37 No. 3, pp. 418-435 , doi: 10.1108/02640471011033611 .

Ding , H. and Sølvberg , I. ( 2007 ), “ Rule-based metadata interoperation in heterogeneous digital libraries ”, The Electronic Library , Vol. 25 No. 2 , pp. 193 - 206 , doi: 10.1108/02640470710741322 .

Du , M. , Fang , J. and Wang , L. ( 2011 ), “ A parameter self-tuning fuzzy-PID control system for pneumatic manipulator of library robot ”, Paper presented at the International Conference on Electronics, Communications and Control (ICECC) , 9-11 Sep , Ningbo, China , doi: 10.1109/ICECC.2011.6067700 .

Dutton , W.H. ( 2014 ), “ Putting things to work: social and policy challenges for the internet of things ”, Info , Vol. 16 No. 3, pp. 1-21 , doi: 10.1108/info-09-2013-0047 .

Dwivedi , Y.K. , Kapoor , K.K. , Williams , M.D. and Williams , J. ( 2013 ), “ RFID systems in libraries: an empirical examination of factors affecting system use and user satisfaction ”, International Journal of Information Management , Vol. 33 No. 2 , pp. 367 - 377 , doi: 10.1016/j.ijinfomgt.2012.10.008 .

Edelman , H. ( 2006 ), “ Intelligent design and the evolution of American research library collections ”, Library Resources and Technical Services , Vol. 50 No. 4 , pp. 234 - 238 , doi: 10.5860/lrts.50n4.234 .

Fourie , I. ( 2003 ), “ How can current awareness services (CAS) be used in the world of library acquisitions? ”, Online Information Review , Vol. 27 No. 3 , pp. 183 - 195 , doi: 10.1108/14684520310481409 .

Francisco , R. and Arsenio , A.M. ( 2015 ), “ Switching brains: cloud-based intelligent resources management for the internet of cognitive things ”, EAI Endorsed Transactions on Cognitive Communications , Vol. 1 No. 2 , pp. 1 - 13 , doi: 10.4108/cogcom.1.2.e4 .

Frederick , D. E. ( 2016 ), “ Libraries, data and the fourth industrial revolution (Data Deluge Column) ”, Library Hi Tech News , Vol. 33 No. 5 , pp. 9 - 12 , doi: 10.1108/LHTN-05-2016-0025 .

Gelfand , J. ( 1998 ), “ Teaching and exposing grey literature: what the information profession needs to know-examples from the sciences ”, Collection Building , Vol. 17 No. 4, pp. 159-166 , doi: 10.1108/01604959810238301 .

Giannattasio , I. and Bruckmann , D. ( 1992 ), “ An update of audiovisual and new technology in French libraries ”, IFLA Journal-International Federation of Library Associations , Vol. 18 No. 3 , pp. 252 - 257 , doi: 10.1177/034003529201800313 .

Golub , K. , Tudhope , D. , Zeng , M.L. and Zumer , M. ( 2014 ), “ Terminology registries for knowledge organization systems: functionality, use, and attributes ”, Journal of the Association for Information Science and Technology , Vol. 65 No. 9 , pp. 1901 - 1916 , doi: 10.1002/asi.23090 .

Grigorescu , S.M. , Natarajan , S.K. , Mronga , D. and Graser , A. ( 2010 ), “ Robust feature extraction for 3D reconstruction of boundary segmented objects in a robotic library scenario ”, Paper presented at the International Conference on Intelligent Robots and Systems (IROS), IEEE/RSJ , 18-22 Oct . Taipei, Taiwan .

Hahn , J. ( 2012 ), “ Mobile augmented reality applications for library services ”, New Library World , Vol. 113 Nos 9/10 , pp. 429 - 438 , doi: 10.1108/03074801211273902 .

Heyer , S. , Enjarini , B. , Fragkopoulos , C. and Graeser , A. ( 2012 ), “ Book detection and grasping in library scenario ”, Paper presented at the ROBOTIK 2012; 7th German Conference on Robotics , 21-22 May , Munich, Germany .

Hsieh , C.C. and Hall , W. ( 1989 ), “ Survey of artificial-intelligence and expert systems in library and information-science literature ”, Information Technology and Libraries , Vol. 8 No. 2 , pp. 209 - 214 , available at: https://www.learntechlib.org/p/170528/ ( accessed 21 April 2019 ).

Huili , F. and Bo , Sh. ( 2017 ), “ A review on researches and practices of library robots at home and abroad ”, Library Journal , Vol. 36 No. 6 , pp. 88 - 94 , available at: http://www.libraryjournal.com.cn/EN/Y2017/V36/I6/88 ( accessed 21 April 2019 ).

Hwang , G.J. , Chu , H.C. , Lin , Y.S. and Tsai , C.C. ( 2011 ), “ A knowledge acquisition approach to developing Mindtools for organizing and sharing differentiating knowledge in a ubiquitous learning environment ”, Computers and Education , Vol. 57 No. 1 , pp. 1368 - 1377 , doi: 10.1016/j.compedu.2010.12.013 .

Iglesias , E. ( 2013 ), Robots in Academic Libraries: Advancements in Library Automation , Central Connecticut State University, USA, IGI Global , doi: 10.4018/978-1-4666-3938-6 .

Islam , M.A. and Ikeda , M. ( 2014 ), “ Convergence issues of knowledge management in digital libraries: steps towards state-of-the-art digital libraries ”, Vine: The Journal of Information , Vol. 44 No. 1, pp. 140-159 , doi: 10.1108/VINE-05-2013-0029 .

Ismail , M.A. and Kareem , S.A. ( 2011 ), “ Identifying how novice researchers search, locate, choose and use web resources at the early stage of research ”, Malaysian Journal of Library and Information Science , Vol. 16 No. 3 , pp. 67 - 85 .

Joint , N. ( 2009 ), “ The Web 2.0 challenge to libraries ”, Library Review , Vol. 58 No. 3 , pp. 167 - 175 , doi: 10.1108/00242530910942027 .

Kao , S.C. and Wu , C. ( 2012 ), “ PIKIPDL A personalized information and knowledge integration platform for DL service ”, Library Hi Tech , Vol. 30 No. 3 , pp. 490 - 512 , doi: 10.1108/07378831211266627 .

Kapoor , K. , Dwivedi , Y. , Piercy , N.C. , Lal , B. and Weerakkody , V. ( 2014 ), “ RFID integrated systems in libraries: extending TAM model for empirically examining the use ”, Journal of Enterprise Information Management , Vol. 27 No. 6 , pp. 731 - 758 , doi: 10.1108/JEIM-10-2013-0079 .

Kesselman , M. ( 2017 ), “ Conference report: 50th consumer electronics show ”, Library Hi Tech News , Vol. 34 No. 3, pp. 1-8 , doi: 10.1108/LHTN-03-2017-0014 .

Kim , B.K. and Kohtaro , O. ( 2009 ), “ Information structured space and ambient intelligent systems for a librarian robot ”, The Journal of Korea Robotics Society, Vol. 4, pp. 147-154 , available at: http://www.koreascience.or.kr/article/JAKO200917161877604 .

Kim , B.K. , Ohara , K. , Kitagaki , K. , Ohba , K. and Sugawara , T. ( 2008 ), “ Design of ubiquitous space for the robotic library system and its application ”, IFAC Proceedings Volumes , Vol. 41 No. 2 , pp. 8221 - 8225 , doi: 10.3182/20080706-5-KR-1001.01391 .

Kim , B.K. , Ohara , K. , Kitagaki , K. and Ohba , K. ( 2009 ), “ Design and control of librarian robot system in information structured environments ”, Journal of Robotics and Mechatronics , Vol. 21 No. 4 , pp. 507 - 514 .

Kim , K.J. , Park , E. and Shyam Sundar , S. ( 2013 ), “ Caregiving role in human–robot interaction: a study of the mediating effects of perceived benefit and social presence ”, Computers in Human Behavior , Vol. 29 No. 4 , pp. 1799 - 1806 , doi: 10.1016/j.chb.2013.02.009 .

Kruk , S.R. and Krawczyk , H. ( 2004 ), “ Intelligent resources search in virtual libraries ”, in Intelligent Information Processing and Web Mining, Advances in Soft Computing , Vol 25, Springer, Berlin, Heidelberg, pp. 439-443 , doi: 10.1007/978-3-540-39985-8_49 .

Kyrarini , M. , Naeem , S. , Wang , X. and Gräser , A. ( 2017 ), “ Skill robot library: intelligent path planning framework for object manipulation ”, 25th European Signal Processing Conference (EUSIPCO) , Kos , 2017 , pp. 2398 - 2402 , doi: 10.23919/EUSIPCO.2017.8081640 .

Lakshmikant , M. and Vishnu , S. ( 2008 ), Automation and Networking of Libraries (Electronic Source): A Manual of Library Management Software and Applications of Computer Technology in Libraries , New Age International, India .

Lam , K.T. and Chan , D.L.H. ( 2007 ), “ Building an institutional repository: sharing experiences at the HKUST Library ”, OCLC Systems and Services: International Digital Library Perspectives , Vol. 23 No. 3 , pp. 310 - 323 , doi: 10.1108/10650750710776440 .

Lancaster , F.W. and Smith , L.C. ( 1990 ), Artificial Intelligence and Expert Systems: Will They Change the Library? , 27th Clinic on Library Applications of Data Processing March 25-27, at the University of Illinois at Urbana- Champaign. From IDEALS, the Illinois Digital Environment for Access to Learning and Scholarship, available at: https://www.ideals.illinois.edu/handle/2142/1209 .

Lancaster , F.W. ( 1997 ), Artificial Intelligence and Expert System Technologies: Prospects. Libraries for the New Millennium: Implications for Managers , Library Association , London , pp. 19 - 38 .

Leslie , D. ( 1999 ), “ Industry experts gather to map future of libraries ”, The Electronic Library , Vol. 17 No. 3 , pp. 149 - 153 , doi: 10.1108/02640479910329734 .

Lin , W. , Yueh , H.P. , Wu , H.Y. and Fu , L.C. ( 2013 ), “ Developing a service robot for a children's library: a design-based research approach ”, Journal of the Association for Information Science and Technology , Vol. 65 No. 2 , pp. 290 - 301 , doi: 10.1002/asi.22975 .

Liu , G. ( 2011 ), “ The application of intelligent agents in libraries: a survey ”, Program , Vol. 45 No. 1 , pp. 78 - 97 , doi: 10.1108/00330331111107411 .

McDonnell , M. and Shiri , A. ( 2011 ), “ Social search: a taxonomy of, and a user-centred approach to, social web search ”, Program , Vol. 45 No. 1 , pp. 6 - 28 , doi: 10.1108/00330331111107376 .

Meador , J.M. and Cline , L. ( 1992 ), “ Displaying and utilizing selection tools in a user-friendly electronic environment ”, Library Acquisitions: Practice & Theory , Vol. 16 No. 3 , pp. 289 - 294 .

Mehtab Alam Ansari , A. ( 2008 ), “ Awareness and use of OPACs in five Delhi libraries ”, The Electronic Library , Vol. 26 No. 1 , pp. 111 - 129 , doi: 10.1108/02640470810851789 .

Mei , Zh. , Chen , Y. , Jiang , M. , Wu , H. and Cheng , L. ( 2017 ), “ Mobile robots path planning based on dynamic movement primitives library ”, Control Conference (CCC) , 2017 36th Chinese , 26-28 July , doi: 10.23919/ChiCC.2017.8028446 .

Miglino , O. , Gigliotta , O. , Ponticorvo , M. and Nolfi , S. ( 2008 ), “ Breedbot: an evolutionary robotics application in digital content ”, The Electronic Library , Vol. 26 No. 3 , pp. 363 - 373 , doi: 10.1108/02640470810879509 .

Mikawa , M. ( 2010 ), A Practicum Track Using Librarian Robot in Support Program for Contemporary Educational Needs , University of Tsukuba, Japan, available at: http://www.iiis.org/CDs2010/CD2010SCI/SCI_2010/PapersPdf/SA431IB.pdf ( accessed 21 June 2018 ).

Milella , A. , Cicirelli , G. and Distante , A. ( 2008 ), “ RFID-assisted mobile robot system for mapping and surveillance of indoor environments ”, Industrial Robot , Vol. 35 No. 2 , pp. 143 - 152 , doi: 10.1108/01439910810854638 .

Mishra , L. and Srivastava , V. ( 2008 ), Automation and Networking of Libraries [Electronic Resource]: A Manual of Library Management Software and Applications of Computer Technology in Libraries , New Age International .

Modler , N. , Hufenbach , W. , Comsa , A. , Maniu , I. , Zichner , M. and Friedrich , J. ( 2012 ), “ Compliant structures in book handling applications ”, Applied Mechanics and Materials , Vol. 162 , pp. 543 - 548 .

Modler , N. , Comsa , A. , Maniu , I. , Lovasz , E.C. and Ciupe , V. ( 2014 ), “ Dedicated gripper for books handling in a library ”, in Visa , I. (Eds), The 11th IFToMM International Symposium on Science of Mechanisms and Machines. Mechanisms and Machine Science , Vol. 18 , Springer , Cham , doi: 10.1007/978-3-319-01845-4_51 .

Noh , Y. ( 2013 ), “ A study on next-generation digital library using context-awareness technology ”, Library Hi Tech , Vol. 31 No. 2 , pp. 236 - 253 , doi: 10.1108/07378831311329031 .

O'Neill , M. and Morris , A. ( 1989 ), “ The contribution of library and information science to expert system development ”, The Electronic Library , Vol. 7 No. 5 , pp. 295 - 300 , doi: 10.1108/eb044908 .

Oyelude , A.A. ( 2017 ), “ What's trending in libraries from the internet cybersphere–artificial intelligence and other emerging technologies ”, Library Hi Tech News , Vol 34 No. 2, pp. 11-12 , doi: 10.1108/LHTN-02-2017-0008 .

Park , E. and Kim , K.J. ( 2013 ), “ User acceptance of long-term evolution (LTE) services: an application of extended technology acceptance model ”, Program , Vol. 47 No. 2 , pp. 188 - 205 , doi: 10.1108/00330331311313762 .

Park , E. , Sung , J. and Cho , K. ( 2015 ), “ Reading experiences influencing the acceptance of E-book devices ”, The Electronic Library , Vol. 33 No. 1, pp. 120-135 , doi: 10.1108/EL-05-2012-0045 .

Payne , G.F. and Bradbury , D. ( 2002 ), “ An automated approach to online digital reference: the Open University Library OPAL Project ”, Program-Electronic Library and Information Systems , Vol. 36 No. 1 , pp. 5 - 12 , doi: 10.1108/00330330210426076 .

Phillips , D. ( 2017 ), “ Robots in the Library: gauging attitudes towards developments in robotics and ai, and the potential implications for library services ”, Dissertation , University of London , doi: 10.17613/M6535M .

Prats , M. , Sanz , P.J. , Del Pobil , A.P. , Martínez , E. and Marín , R. ( 2007 ), “ Towards multipurpose autonomous manipulation with the UJI service robot ”, Robotica , Vol. 25 No. 2 , pp. 245 - 256 , doi: 10.1017/S0263574706003304 .

Prats , M. , Martínez , E. , Sanz , P.J. and Del Pobil , A.P. ( 2008 ), “ The UJI librarian robot ”, Intel Serv Robotics , Vol. 1 No. 4 , pp. 321 - 335 , doi: 10.1007/s11370-008-0028-1 .

Ramos-Garijo , R. , Prats , M. , Sanz , P.J. and Del Pobil , A.P. ( 2003 ), “ An autonomous assistant robot for book manipulation in a library ”, Paper presented at the IEEE International Conference on Systems, Man and Cybernetics , 8-8 October , USA: Washington DC , doi: 10.1109/ICSMC.2003.1244499 .

Rudall , B.H. ( 2006 ), “ Contemporary systems and cybernetics ”, Kybernetes , Vol. 35 Nos 1/2 , pp. 209 - 216 , doi: 10.1108/03684920610640326 .

Sacchanand , C. and Jaroenpuntaruk , V. ( 2006 ), “ Development of a web-based self-training package for information retrieval using the distance education approach ”, The Electronic Library , Vol. 24 No. 4 , pp. 501 - 516 , doi: 10.1108/02640470610689197 .

Sharda , S. , Delen , D. , Turban , T. and King , K. ( 2017 ), Business Intelligence: A Managerial Approach , Global Edition, Pearson , available at: https://www.pearson.com/uk/educators/higher-education-educators/product/Sharda-Business-Intelligence-A-Managerial-Approach-Global-Edition-4th-Edition/9781292220543.html (accessed April 13, 2020) .

Singh , D.K. , Singh , B.K. and Dubey , Y.P. ( 1996 ), “ Expert systems and their application in library and information systems ”, DESIDOC Bulletin of Information Technology , Vol. 16 No. 4 , pp. 9 - 12 , available at: http://publications.drdo.gov.in/ojs/index.php/djlit/article/download/3272/1727/ ( accessed 21 April 2019 ).

Sowell , S.L. ( 1989 ), “ Expanding horizons in collection development with expert systems: development and testing of a demonstration prototype ”, Special Libraries , Vol. 80 No. 4 , pp. 45 - 50 .

Taha , A. ( 2012 ), “ Networked library services in a research-intensive university ”, The Electronic Library , Vol. 30 No. 6 , pp. 844 - 856 , doi: 10.1108/02640471211282145 .

Vincze , J. ( 2017 ), “ Virtual reference librarians (Chatbots) ”, Library Hi Tech News , Vol. 34 No. 4 , pp. 5 - 8 , doi: 10.1108/LHTN-03-2017-0016 .

Waters , S.T. ( 1986 ), “ Answerman, the expert information specialist - an expert system for retrieval of information from library reference-books ”, Information Technology and Libraries , Vol. 5 No. 3 , pp. 204 - 212 .

Weiss , P.J. ( 1994 ), “ The expert cataloging assistant project at the national-library-of-medicine ”, Information Technology and Libraries , Vol. 13 No. 4 , pp. 267 - 271 .

Wu , H.C. , Chou , C. , Ke , H.R. and Wang , M.H. ( 2010 ), “ College students' misunderstandings about copyright laws for digital library resources ”, The Electronic Library , Vol. 28 No. 2 , pp. 197 - 209 , doi: 10.1108/02640471011033576 .

Xiaobin , L. and Jing , G. ( 2009 ), “ Innovation community: constructing a new service mode for academic libraries ”, The Electronic Library , Vol. 27 No. 2 , pp. 258 - 270 , doi: 10.1108/02640470910947601 .

Yao , F. , Lei , J. , Chengyu , Z. and Wu , Ch. ( 2011 ), “ New attempt on real-time virtual reference service: the smart chat robot of tsinghua university library ”, New Technology of Library and Information Service , Vol. 27 No. 4 , pp. 77 - 81 , available at: http://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/10.11925/infotech.1003-3513.2011.04.13 .

Yao , F. , Zhang , Ch. and Chen , W. ( 2015 ), “ Smart talking robot Xiaotu: participatory library service based on artificial intelligence ”, Library Hi Tech , Vol. 33 No. 2 , pp. 245 - 260 , doi: 10.1108/LHT-02-2015-0010 .

Yi , S. , Bao , L. and Jianfeng , Q. ( 2011 ), “ Design and implementation of library intelligent IM reference robot ”, New Technology of Library and Information Service , Vol. 27 No. 5 , pp. 88 - 92 , available at: http://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/10.11925/infotech.1003-3513.2011.05.14 .

Yuehu , W. and Yanqing , L. ( 2012 ), “ An intelligent library based on the IOT and the robot technology ”, Library Work and Study , Vol. 3 No. 5 , pp. 250-276, available at: http://en.cnki.com.cn/Article_en/CJFDTOTAL-TSGG201203006.htm .

Zimerman , M. ( 2012 ), “ Digital natives, searching behavior and the library ”, New Library World , Vol. 113 Nos 3/4 , pp. 174 - 201 doi: 10.1108/03074801211218552 .

Further reading

Berube , L. ( 2004 ), “ Collaborative digital reference: an ask a librarian (UK) overview ”, Program , doi: 10.1108/00330330410519189?fullSc=1 .

Clyde , L.A. ( 2000 ), “ A strategic planning approach to Web site management ”, The Electronic Library , Vol. 18 No. 2 , pp. 97 - 108 , doi: 10.1108/02640470010325637 .

Detlor , B. and Arsenault , C. ( 2002 ), “ Web information seeking and retrieval in digital library contexts: towards an intelligent agent solution ”, Online Information Review , Vol. 26 No. 6 , pp. 404 - 412 , doi: 10.1108/14684520210452736 .

El-Sherbini , M. and Klim , G. ( 2004 ), “ Metadata and cataloging practices ”, The Electronic Library , Vol. 22 No. 3, pp. 238-248 , doi: 10.1108/02640470410541633 .

MacDougall , J. , Brittain , J.M. and Gann , R. ( 1996 ), “ Health informatics: an overview ”, Journal of Documentation , Vol. 52 No. 4 , pp. 421 - 448 , doi: 10.1108/eb026974 .

Mardikian , J. ( 1995 ), “ Self-service charge systems: current technological applications and their implications for the future library ”, Reference Services Review , Vol. 23 No. 4, pp. 19-38 , doi: 10.1108/eb049262 .

Morales , E.R. ( 1999 ), “ Generis: the ec-jrc generalised software control system for industrial robots ”, Industrial Robot: International Journal , Vol. 26 No. 1 , pp. 26 - 32 , doi: 10.1108/01439919910250197 .

Omekwu , C.O. ( 2006 ), “ African culture and libraries: the information technology challenge ”, The Electronic Library , Vol. 24 No. 2 , pp. 243 - 264 , doi: 10.1108/02640470610649218 .

Raitt , D.I. ( 1985 ), “ Look—no paper! the library of tomorrow ”, The Electronic Library , Vol. 3 No. 4 , pp. 276 - 289 , doi: 10.1108/eb044666 .

Reneker , M.H. and Buntzen , J.L. ( 2000 ), “ Enterprise knowledge portals: two projects in the United States Department of the Navy ”, The Electronic Library , Vol. 18 No. 6 , pp. 392 - 403 , doi: 10.1108/EUM0000000005386 .

Tegenbos , J. and Nieuwenhuysen , P. ( 1997 ), “ My kingdom for an agent? evaluation of autonomy, an intelligent search agent for the internet ”, Online and CD-Rom Review , Vol. 21 No. 3, pp. 139-148 , doi: 10.1108/eb024616 .

Acknowledgements

This research has been supported by the “Project no. NKFIH-869-4/2019 has been implemented with the support provided from the National Research, Development and Innovation Fund of Hungary, financed under the Tématerületi Kiválósági Program 2019 funding scheme.”

Corresponding author

Related articles, all feedback is valuable.

Please share your general feedback

Report an issue or find answers to frequently asked questions

Contact Customer Support

Unfortunately we don't fully support your browser. If you have the option to, please upgrade to a newer version or use Mozilla Firefox , Microsoft Edge , Google Chrome , or Safari 14 or newer. If you are unable to, and need support, please send us your feedback .

We'd appreciate your feedback. Tell us what you think! opens in new tab/window

How libraries have transformed through 25 years of digital innovation

August 26, 2024

By Susan Jenkins

The early years 

When ScienceDirect.com opens in new tab/window launched in 1999, its consolidation of discovery and access to a wide body of digitized research arrived on a wave of internet-enabled digital innovation. This wave has continued to transform the spaces, capabilities, and role of research libraries ever since.  

Imagine a librarian at work in a university library during the 1990s, surrounded by stacks filled with scholarly journals, books, maps, and other printed materials. There are also microfilm machines, CD-ROM collections, and a few computer terminals. 

Anita Laamanen opens in new tab/window , eLibrary and Tools team leader at VTT Technical Research Centre of Finland, joined the library team in 1989 and remembers those days well. “We had NTIS [National Technical Information Service] reports from the US in microfilm. Those microfilm reader devices were quite large and it was not very convenient because you needed help from library staff to use it. It was also one user at a time.”

Carol Shannon opens in new tab/window , Informationist at the University of Michigan Taubman Health Science Library, remembers working as a shelver during the extra-late hours of exams week – she recalled, “we were going around at 2:00am when the library closed putting back piles of books.”  

Electronic resources emerge 

Experimental “electronic libraries” of scanned articles appeared around the same time as the internet first promised open connection to a global network of information.  While at the time they lacked interactive features, subsequent innovations between 1993 and 1997 – among them internet browser software, ethernet connection protocols, more powerful computing chips, the Portable Document Format (PDF) standard, and the development of algorithmic search – soon enabled the launch of robust research platforms like ScienceDirect.  

In place of disconnected library collections came a new digital interface that gave anyone with institutional ethernet connection the ability to search for and read full articles from across many scholarly journals. Anita recalls, “the first digital content we subscribed to was Academic Press in 1998 - that was the start. After that came ScienceDirect and all the major publishers - IEEE, American Chemical Society and Institute of Physics, for example.” 

Now researchers could read new issues of a journal without waiting for circulation – or having to visit the library. They could download a PDF copy of an article for reading offline, or for annotating and citing in their own research writing. And with a new algorithmic “full-text search” could discover a broad range of connected research topics without needing to select a specific journal first.   

When it officially launched, ScienceDirect had already incorporated 3 years of digitized full-length research articles in HTML and PDF format from approximately 1000 Elsevier journals, a years-long extension of an experimental project opens in new tab/window from the early 1990s.  

Organized by subject, it also included five years of abstracts from over 2300 journals, including several leading scientific publishers in addition to Elsevier. The site features included a demo for guests or licensed users to see the possibilities and learn how to navigate to articles within a favorite journal and how to use the full-text keyword search.  

ScienceDirect home page in January 1999 (retrieved from The Internet Archive)

Libraries (and publishers) adapt 

Right away, library services pivoted as researchers and faculties adapted to this new accelerating digital information landscape. Librarians quickly focused on training – for each other and the research communities they served. Anita says, “I found an advertisement for a ScienceDirect Day here on the 10 th of November 2000, where we familiarized our researchers with navigation, how to search, and how to create a search alert.” 

The search alert - an email notification when a new journal volume or new article on a topic was published – was a new concept at the time. For Anita’s library, it meant that “journal alerts eventually replaced our physical printed journal circulation.” 

The year 2007 was a tipping point in the digital transformation. After years of varied adaptation, journals now presented all the same content digitally as in print, standardizing the digital editions of their publications as the version of record. 

Meanwhile, research platforms had grown and evolved, adding books and reference works, and digitizing entire back catalogs of journals to offer access to all their publications’ literature in every field. ScienceDirect had grown from 300,000 articles at its launch to over 8 million articles from over 2000 journals, covering 1823 to the present day.  

Image from 2004 brochure for librarians about the backfiles initiative (retrieved from the Internet Archive)

These were key developments that led most journals and libraries to let go of print and fully move into the digital age. Though some faculty still preferred print, there was growing desire among researchers and students for the flexibility and reach that digital offered. “In the mid-2000s we got laptops and personal devices so it was more convenient to read papers with your own laptop” recalled Anita, who added “here in the wintertime it’s not so convenient to walk to another building because it’s dark, cold, and snowing heavily, so I think people were relieved that they don’t need to go into the library anymore to get their papers.” 

Librarian expertise becomes more essential, and more diverse 

The digital transformation dissolved many of the operational tasks connected with managing paper-based knowledge, but created many new skills and roles. Teaching and research became more significant in a research librarian’s role – creating training modules for workshops, contributing to online LibGuides resources, and researching best practices for maintaining integrity in the digitized research process.  

By this time, an expansion of discovery capabilities, including improved full-text search functions and related article suggestions increased the need for librarian expertise even while it accelerated what a researcher could do on their own. Knowledge about search strategies and conducting efficient literature as well as systematic reviews became part of librarians’ standard training offerings. They also began developing information literacy curriculums for students to build awareness of sound research methods and how to evaluate quality resources, based on the new features.  

Specializations have, if anything, expanded, as has collaboration between librarians with different skill sets. Carol describes how “the main library is now adding people on digital scholarship - using digital tools to do research in different ways. Not just citation management tools, which are great, but doing text mining and bibliometric research.” But she adds, “You still need someone with subject expertise to support the technical expertise of the student, because there can be differences in areas and how things are published.”  

She offers a recent example: “I had a medical student who was interested in creating a [software] program to make it easier for doctors to diagnose conditions at the bedside. I referred them to one of our engineering librarians because they first needed to look at what patents and technology already exist. We have somebody who specializes in patents and we also have a biomedical engineering librarian. The student might come back to me for something else, but different librarians have different strengths, so we are reliant on each other.” 

ScienceDirect turns 25

ScienceDirect is celebrating its 25 th anniversary this year as a leading research platform helping further the advancement of knowledge for the benefit of society. Users today have access to: 

634,000 new articles published in 2023 

190,000 open access articles published in 2023, a 23% increase over the prior year 

21 million articles, 46,000 books, 3.3 million open access articles 

Embedded topics pages to build foundational knowledge and learn new terminology 

Highest cited content of all publishers 

One of the Top 200 internet platforms (per Ahrefs 2024 ranking) 

New roles for data analysis emerged in the library as standards and systems began to arise around the organization and dissemination of digitized research. The Digital Object Identifier (DOI) opens in new tab/window introduced in 1997 unified the identification of discrete research articles across different research platforms. Project COUNTER opens in new tab/window , (Counting Online Usage of Networked Electronic Resources, launched in 2003) was another standard that supported collections management and gave insights into library users needs as well, feeding back into the training curriculums and services a library offered.  

Anita added, “we can take a closer look at how they make their searches, because we have the data from the publishers’ platforms – what are the most used search terms? And how they search, what devices they use. If we monitor the usage, we’ve got the tools to analyze our usage and perhaps meet their needs more precisely.” 

Other roles that emerged for librarians included developing and managing their institution’s institutional repository (IR) and communicating with various faculties about how to use it. Contracts and licensing, finance and budget management, and outreach roles grew as well. 

Anita adds that “on our table are so many things now, and many of us know each other’s work, we can replace each other in the summertime for example. We have become more effective.”  

Yet for her, much of the day-to-day interaction with library users has stayed the same. “Only the way of contact has changed. Back then we met physically with the library desk between and nowadays it’s easy to contact us by chat, email perhaps, or some will still call on the phone. If they contact us by chat it’s convenient because we can share the screen and explain how to do something; or by email we can make a short video using these lovely video capturing services to show how to solve a problem.”

ScienceDirect home page in August 2024

New waves of empowerment 

In recent years, a growing awareness of Diversity, Equity and Inclusion issues has led to new international digital design standards opens in new tab/window making websites and their content accessible for people with disabilities. The reliance of research publishing on visual interaction, whether digital or paper, has presented many barriers.  

For example, where once someone with a visual impairment would need to approach their library with a special accessibility request for research sites or articles, now that information –text and illustrations – can be navigated by speech or described with audio captions, thanks to innovations in the coding of websites that allows them to interact effectively with assistive applications - from screen readers to language translation tools. ScienceDirect has improved access for nearly every aspect of its content thanks to an ongoing, comprehensive approach opens in new tab/window to accessibility, with the ScienceDirect homepage achieving the #1 spot in the 2023 WebAIM Million opens in new tab/window study that evaluates the top 1 million websites around the world for accessible content.  

As Carol confirms, these changes have made this aspect of the library’s work no longer necessary - “We say ‘let us know ahead of time if we need to provide anything for you.’ In general, we don't get any response because the tools are now so much more accessible than they used to be. It's wonderful that people don't have to come in and request something of us because we're not automatically providing it.” 

Of course, the newest innovation to sweep through libraries and institutions is generative AI. Librarian’s core skills – sustaining the integrity around evaluating and disseminating knowledge - are already integrating into new roles as this latest wave begins to transform the capabilities of research platforms for the years to come. 

Contributor

Susan Jenkins

Freelance writer and translator

S. Tyler Jenkins

• Liberty University
• Jerry Falwell Library
• Special Collections
• < Previous

Home > ETD > Doctoral > 5930

Doctoral Dissertations and Projects

The effectiveness of an interdisciplinary care team: an integrative review.

Lindsey Marie Ellingford , Liberty University Follow

School of Nursing

Doctor of Nursing Practice (DNP)

Tonia Kennedy

multidisciplinary team, integrative team, interdisciplinary team, interprofessional team, or healthcare team

Disciplines

Recommended citation.

Ellingford, Lindsey Marie, "The Effectiveness of an Interdisciplinary Care Team: An Integrative Review" (2024). Doctoral Dissertations and Projects . 5930. https://digitalcommons.liberty.edu/doctoral/5930

As health care is ever growing and changing, demands for interdisciplinary care team collaboration are crucial for collective competence and team performance across settings. In healthcare facilities across the county, interdisciplinary care teams are made up of multiple disciplines. The following is an integrative review with a purpose to determine if the existing literature supports the implementation of an interdisciplinary care team in the healthcare environment. After completion of this integrative review, the author concluded there should be a standardized tool in the healthcare system to guide the creation and implementation of an interdisciplinary care team. The Jerry Falwell Library at Liberty University was searched, and databases included: Consumer Health Database, PubMed, Cochran Library, EBSCO, and CINAHL. Parameters of the search included peer-reviewed articles published in the English language within the past five years. A total of 1,116 results were identified; 16 articles were used in the literature review. The articles were leveled using Melnyk’s level of evidence and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) was used as a guide to support the reporting of this integrative review.

Since August 29, 2024

Included in

Nursing Commons

• Collections
• Faculty Expert Gallery
• Theses and Dissertations
• Conferences and Events
• Open Educational Resources (OER)
• Explore Disciplines

Advanced Search

• Notify me via email or RSS .

Faculty Authors

• Submit Research
• Expert Gallery Login

Student Authors

• Undergraduate Submissions
• Graduate Submissions
• Honors Submissions

Home | About | FAQ | My Account | Accessibility Statement

Privacy Copyright

This paper is in the following e-collection/theme issue:

Published on 30.8.2024 in Vol 11 (2024)

Evaluation of Digital Mental Health Technologies in the United States: Systematic Literature Review and Framework Synthesis

Authors of this article:

• Julianna Catania 1 , MPH   ; 
• Steph Beaver 1 , MChem   ; 
• Rakshitha S Kamath 1 , MS, MSL   ; 
• Emma Worthington 2 , MPH   ; 
• Minyi Lu 3 , PhD   ; 
• Hema Gandhi 3 , PhD   ; 
• Heidi C Waters 3 , PhD   ; 
• Daniel C Malone 4 , PhD  

1 Costello Medical, Boston, MA, United States

2 Costello Medical, Cambridge, United Kingdom

3 Otsuka Pharmaceutical Development & Commercialization Inc, Princeton, NJ, United States

4 Department of Pharmacotherapy, Skaggs College of Pharmacy, University of Utah, Salt Lake City, UT, United States

Corresponding Author:

Daniel C Malone, PhD

Department of Pharmacotherapy

Skaggs College of Pharmacy

University of Utah

30 S 2000 East

Salt Lake City, UT, 84112

United States

Phone: 1 801 581 6257

Email: [email protected]

Background: Digital mental health technologies (DMHTs) have the potential to enhance mental health care delivery. However, there is little information on how DMHTs are evaluated and what factors influence their use.

Objective: A systematic literature review was conducted to understand how DMHTs are valued in the United States from user, payer, and employer perspectives.

Methods: Articles published after 2017 were identified from MEDLINE, Embase, PsycINFO, Cochrane Library, the Health Technology Assessment Database, and digital and mental health congresses. Each article was evaluated by 2 independent reviewers to identify US studies reporting on factors considered in the evaluation of DMHTs targeting mental health, Alzheimer disease, epilepsy, autism spectrum disorder, or attention-deficit/hyperactivity disorder. Study quality was assessed using the Critical Appraisal Skills Program Qualitative and Cohort Studies Checklists. Studies were coded and indexed using the American Psychiatric Association’s Mental Health App Evaluation Framework to extract and synthesize relevant information, and novel themes were added iteratively as identified.

Results: Of the 4353 articles screened, data from 26 unique studies from patient, caregiver, and health care provider perspectives were included. Engagement style was the most reported theme (23/26, 88%), with users valuing DMHT usability, particularly alignment with therapeutic goals through features including anxiety management tools. Key barriers to DMHT use included limited internet access, poor technical literacy, and privacy concerns. Novel findings included the discreetness of DMHTs to avoid stigma.

Conclusions: Usability, cost, accessibility, technical considerations, and alignment with therapeutic goals are important to users, although DMHT valuation varies across individuals. DMHT apps should be developed and selected with specific user needs in mind.

Introduction

Digital health comprises a broad range of technologies, including mobile health, health information technology, wearable devices, and personalized medicine, which serve as tools to enhance health care delivery. Recently, several digital mental health (MH) therapeutics, a category of digital MH technologies (DMHTs), have received US Food and Drug Administration (FDA) approval to prevent, manage, or treat a medical disorder or disease based on evidence from superiority trials and compliance with technical guidelines [ 1 , 2 ]. However, most DMHTs, particularly apps, fall outside FDA jurisdiction because they are not intended to diagnose, treat, or prevent disease and because they are “low risk” in that they would not cause harm in the event of malfunction [ 3 ]. Due to this lack of regulatory framework, few DMHTs are supported by published efficacy studies. One study found that only 16% of MH apps recommended by college counseling centers were supported by efficacy studies published in peer-reviewed journals [ 4 ].

Nonetheless, many health care providers (HCPs) use MH apps in clinical practice. Up to 83% of behavioral health providers in a small study covering the Greater Boston area reported using apps as part of their clinical care, particularly mindfulness apps for patient anxiety management [ 5 ]. As many DMHTs are currently widely used in clinical practice without undergoing any formal assessment for quality or relevance, understanding how DMHTs should be assessed based on factors impacting their value from the perspective of key stakeholders, such as patients, caregivers, providers, payers, and employers, could improve the selection of DMHTs for use by patients, thereby increasing care quality and outcomes for those seeking MH support.

To address identified gaps, a systematic literature review (SLR) was conducted using a published framework to synthesize emerging themes from mixed methods evidence in order to understand how digital health solutions, encompassing both digital therapeutics and direct-to-consumer digital health technologies, are valued, with a focus on MH disorders, Alzheimer disease, epilepsy, autism spectrum disorder (ASD), and attention-deficit/hyperactivity disorder (ADHD) in the United States.

The SLR was performed in accordance with a prespecified protocol and reported in line with the Enhancing Transparency in Reporting the Synthesis of Qualitative Research and PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [ 6 , 7 ]. The protocol was not registered.

Search Strategy

Electronic databases, encompassing MEDLINE (including MEDLINE In-Process, MEDLINE Daily, and MEDLINE Epub Ahead of Print); Embase; the Cochrane Library (including Cochrane Database of Systematic Reviews and Cochrane Central Register of Controlled Trials); PsycINFO; and the Health Technology Assessment Database, were selected in alignment with this SLR’s target indications and were searched on June 17, 2022. The search terms included combinations of free-text and Medical Subject Heading or Emtree terms related to indications of interest, DMHTs, and relevant outcomes or assessment types (eg, technology assessments and cost; Tables S1-S5 in Multimedia Appendix 1 ). Searches were limited to studies performed in the United States and to those published from 2017 onward.

Manual hand searches of gray literature, namely, the bibliographies of relevant SLRs identified from the electronic database searches and key conference proceedings (2019-2022), were performed to identify additional studies of relevance (Table S6 in Multimedia Appendix 1 ). The FDA website was also searched to identify factors involved in the FDA’s appraisal of relevant MH apps, which could supplement the factors identified in this SLR (Table S7 in Multimedia Appendix 1 ).

Study Selection

Studies were included in the SLR if they met prespecified criteria defined using the SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type) framework, which is appropriate for mixed methods research questions. Eligible studies were published in the English language, were set in the United States, and reported quantitative or qualitative outcomes relating to the factors considered in the evaluation of DMHTs. Only studies published in 2017 or later were included because of the rapidly evolving research area. Eligible studies reported on MH, Alzheimer disease, epilepsy, ASD, or ADHD from user, payer, or employer perspectives (Table S8 in Multimedia Appendix 1 ). While the primary focus of the SLR was MH, neurological conditions were also of interest because their pathologies, symptoms, and treatment strategies can overlap with those of mental illnesses. Alzheimer disease, epilepsy, ASD, and ADHD were selected because they are highly researched and represent diverse types of neurological conditions.

The titles and abstracts of records were assessed for inclusion against these eligibility criteria by 2 independent reviewers, and discrepancies were resolved by consensus, with arbitration by a third reviewer if necessary. Full texts of potentially relevant articles were acquired and screened using the same methodology.

Study Prioritization

Due to the large volume of the evidence identified, additional eligibility criteria were applied to prioritize primary research on theoretical DMHT valuation factors. In line with the thematic framework synthesis objective, theoretical valuation factors were defined as user or DMHT attributes that impact interaction with or perception of DMHTs; therefore, studies that reported only efficacy outcomes, such as mental illness symptom improvement, were deprioritized for full-text review. Secondary research was also deprioritized for full-text review. Studies that reviewed a select app against a framework and studies that reported only the outcomes specific to a select app were deprioritized for data extraction. For example, a study reporting the usability of a specific app’s features would have been deprioritized, while a study reporting what types of features increase MH app usability in general would not.

Data Extraction

All relevant data were extracted into a prespecified Microsoft Excel grid, and a quality assessment was performed for each study. Studies that reported only qualitative data were assessed with the Critical Appraisal Skills Program Qualitative Studies Checklist. Studies that reported only quantitative data were evaluated with the Critical Appraisal Skills Program Cohort Study Checklist, and studies reporting both qualitative and quantitative data were evaluated with both checklists [ 8 ]. Data extractions and quality assessments were performed by a single individual for each study, with the information verified by a second independent individual. Discrepancies were resolved by consensus, with arbitration by a third individual if necessary.

Framework Synthesis

A framework synthesis approach was undertaken to synthesize qualitative and quantitative data identified from the SLR. In line with the “best fit” framework synthesis approach, data were indexed deductively against an existing framework where possible, and novel themes were added inductively as needed [ 9 , 10 ]. The American Psychiatric Association (APA) Mental Health App Evaluation framework was considered the most appropriate framework to address the research objectives of this SLR because its key valuation themes were developed using psychiatrist and patient input, are broadly shared by other evaluation frameworks, are widely acknowledged in the literature, and have been described as durable and adaptable [ 11 - 13 ].

The APA model follows a hierarchical and chronological order whereby the evaluator moves through the framework using prompting questions (eg, “Does the app work offline?”). For this SLR, these questions were either thematically grouped into subthemes or left as prompting questions, as appropriate. The framework was therefore ultimately adapted into 3 levels: themes, subthemes, and more granular valuation criteria. It should be emphasized that this SLR did not aim to formally develop an updated framework to be used in practice by HCPs and their patients but rather was used to form a theoretical basis for understanding DMHT valuation factors, for which novel themes were expected to emerge.

A data-based convergent approach was used to synthesize quantitative and qualitative data [ 14 ]. Data were initially indexed deductively against the prespecified themes within the data collection instrument and then further synthesized within Docear [ 15 ], a mind-map software used to organize and connect data and concepts. Indexing was performed by 1 reviewer and checked by a second independent reviewer. New themes and subthemes that emerged from the literature through inductive coding were added post hoc to the thematic framework, with all extracted data then considered against both the prespecified and novel themes. The evidence identified for each theme was synthesized narratively, taking into consideration the context and design of each study.

Included Studies

A total of 4974 records were retrieved from the electronic databases. Of the 3374 (67.83%) unique records identified following deduplication across the databases, 2891 (85.68%) were excluded based on the eligibility criteria, and an additional 456 (13.52%) were deprioritized because they were not directly related to the topic of interest for this SLR. Excluded and deprioritized full texts are listed in Tables S9 and S10 in Multimedia Appendix 1 , respectively. Therefore, 27 (0.54%) articles were included from the electronic database searches. In addition, 1 article reporting on the same study as an already-included conference abstract was identified during supporting targeted searches and included as a supplementary record, resulting in a total of 28 articles reporting on 26 unique studies (Figure S1 in Multimedia Appendix 1 ). No relevant FDA appraisals were identified in the supplementary search.

Of the 26 included studies, 8 (31%) were quantitative, 12 (46%) were qualitative, and 6 (23%) used a mixed methods approach. While 5 (19%) studies assessed prospective cohorts, 22 (85%) used a cross-sectional approach, including 1 (4%) study that contained both a prospective cohort and a cross-sectional cohort ( Table 1 ). All studies (26/26, 100%) investigated a user perspective, with none specifically investigating payer or employer perspectives. Only 1 (4%) study, which examined ingestible sensor pills and smart pill dispensers to track adherence, investigated a DMHT that was not an app [ 16 ].

a Only information relevant to this systematic literature review is reported in this table.

b MDD: major depressive disorder.

c BD: bipolar disorder.

d OCD: obsessive-compulsive disorder.

e MH: mental health.

f General users are participants who were not necessarily diagnosed with indications of interest.

g ASD: autism spectrum disorder.

h HCP: Health care provider.

i mHealth: mobile health.

j PTSD: posttraumatic stress disorder.

k VA: Veterans Affairs.

l ADHD: attention-deficit/hyperactivity disorder.

m PHQ-9: Personal Health Questionnaire-9.

n GAD-7: Generalized Anxiety Disorder-7.

o DMHT: digital mental health technology.

Most frequently, studies focused on indications for mood, anxiety, or psychotic disorders (15/26, 58%), with other indications of focus including ADHD (2/26, 8%), ASD (1/26, 4%), and epilepsy (1/26, 4%). No relevant studies focused on Alzheimer disease were identified.

A total of 8 (31%) studies assessed the perspectives toward DMHTs of general population participants who were not necessarily diagnosed with relevant conditions [ 19 , 28 , 29 , 33 - 37 ]. Of these populations, several were identified as having an increased risk of MH conditions, such as patients with cancer [ 28 ], college students [ 29 , 34 ], deaf or hard-of-hearing individuals [ 19 ], and people who were unemployed or furloughed during the COVID-19 pandemic [ 33 ]. In addition, 1 (4%) study included a mix of patients who were above and below the referral criteria for psychotherapy for depression [ 37 ].

Thematic Analysis

Evidence was identified for all 5 themes included in the APA framework: engagement style (23/26, 88%), background and accessibility (16/26, 62%), privacy and security (13/26, 50%), therapeutic goal (12/26, 46%), and clinical foundation (8/26, 31%; Table 2 ). Five novel criteria were identified and added to the framework post hoc, 1 each under engagement style (forgetting or feeling unmotivated to use DMHTs) and privacy and security (personal image and stigma) and 3 under background and accessibility (willingness to pay, insurance restrictions, and cost savings compared with professional care).

a Novel findings that emerged from this systematic literature review.

b These subthemes and criteria were included in the American Psychiatric Association’s framework but were not reported on by studies included in this systematic literature review.

c HCP: health care provider.

Theme 1: Engagement Style

Engagement style was the most reported theme, with evidence identified from 23 (88%) of the 26 studies. Engagement style encompasses how and why users do or do not interact with DMHTs. The long-term usability subtheme was reported by 96% (22/23) of studies, short-term usability by 12 (52%) studies, and customizability by 7 (30%) studies. Findings from short- and long-term usability subthemes were highly interconnected.

A total of 4 studies reported that ease of use promoted short-term DMHT engagement. In the study by Schueller et al [ 35 ], 89.6% of a general population of smartphone users reported ease of use for MH apps as “important” or “very important,” and users qualitatively reported dislike of “overwhelming,” difficult-to-navigate apps. In addition, users valued apps that were “simplistic” [ 34 ], fit into their daily schedules, and were available when needed (eg, during acute symptom experiences) [ 5 , 25 ]. Select supporting qualitative data are presented in Table 3 .

a ASD: autism spectrum disorder.

b MH: mental health.

c ADHD: attention-deficit/hyperactivity disorder

d BD: bipolar disorder.

e Novel criteria identified by this systematic literature review.

f CHA: Cambridge Health Alliance.

g EMR: electronic medical record.

h HCP: health care provider.

Users valued DMHT features that aligned with their needs and priorities, as reflected by findings within the long-term usability subtheme. Across 9 studies, quantitative and qualitative findings demonstrated high interest in anxiety management features such as relaxation tools, breathing exercises, and mindfulness or meditation activities, and 10 studies identified interest in mood, symptom, or sleep tracking ( Tables 3 and 4 ). While most studies (24/26, 92%) focused on MH, patients with epilepsy also reported high interest in features to record seizure dates and types [ 17 ]. Importantly, users in 2 studies emphasized the need for developers to tailor DMHTs to the needs and priorities of the target population ( Table 3 ) [ 28 , 31 ]. Relatedly, mixed attitudes were reported toward positive affirmations and words of encouragement, with many users expressing interest but others emphasizing the value of a human component to DMHTs or cautioning against blanket encouragement and automated messages that could feel insincere [ 19 , 25 , 31 ].

a A 5-point Likert scale (0-4) was used.

b Not available.

c A 5-point Likert scale (1-5) was used.

d MH: mental health.

e DMHT: digital mental health technology.

f MDD: major depressive disorder.

g PTSD: posttraumatic stress disorder.

Both patients and caregivers expressed interest in psychoeducational content that aligned with their needs and priorities. When surveyed, >60% of veterans with anxiety or major depressive disorder (MDD), patients with epilepsy, young adults with psychosis, and essential and furloughed workers during the COVID-19 pandemic expressed interest in relevant psychoeducational content [ 17 , 22 , 32 , 33 ]. In contrast, only 4% of college students in another study reported using an MH app for information about MH, although an MH diagnosis was not required for study participation [ 29 ].

Caregivers of young adults with psychosis, caregivers of children with ADHD, and spouses and partners of people with bipolar disorder (BD) were all interested in information related to caring for the individual with the given disorder, such as information on psychological and pharmacological treatments, symptoms and symptom changes, and the MH system [ 21 , 24 , 26 ]. Comparatively smaller, but still notable, proportions of caregivers of patients with psychosis were interested in caregiver-focused information; for instance, 62% to 69% were interested in relaxation exercises, stress and mood management, and community events for caregivers, while 85% to 90% were interested in the aforementioned patient-focused information [ 21 ].

Information delivery–style preference was captured under the short-term usability subtheme. One study in young adults with psychosis and another study with their caregivers revealed that delivering information in a variety of formats was important; when presented with nonmutually exclusive options, >50% of both populations were interested in text content, video content, audio content, and discussion boards [ 21 , 22 ].

Social interaction promoted long-term engagement. Qualitatively, 3 studies found that users valued learning about similar experiences from others via social media–like features, which normalized their experiences and could provide new symptom management strategies ( Table 3 ) [ 28 , 31 , 36 ]. Similarly, 67% of both young adults with psychosis and deaf or hard-of-hearing survey participants (N=9) reported interest in peer support via chat features [ 19 , 22 ]. However, a comparatively smaller proportion of veterans with anxiety or MDD (48.3% of the full cohort and 51.3% of the smartphone user subgroup) were interested in peer support [ 32 ].

Overall, users endorsed social features to support their MH. In the study by Casarez et al [ 24 ], spouses and partners of people with BD likewise desired features to communicate with other caregivers and also emphasized that DMHTs could facilitate conversation and understanding with patients, a sentiment echoed by peer support specialists by Storm et al [ 38 ] ( Table 3 ). However, one oncology HCP cautioned that similar to support groups, “very strict guidelines of what is said” should be implemented to manage potential risks from shared social media–like content, although little additional context was provided [ 28 ].

Spouses and partners of people with BD also suggested both in-app information on accessing professional resources and direct counseling for the patient at times when other support might be inaccessible [ 24 ]. More than half of all workers, employed or unemployed during the COVID-19 pandemic, likewise endorsed links to resources, counseling, and crisis support as DMHT features, and 81.6% of young adults with psychosis endorsed a feature to communicate with professional experts [ 22 , 33 ]. Importantly, compared with patients attending public clinics, patients attending private psychiatric clinics expressed a higher comfort level for in-app communication with HCPs, suggesting demographic differences in the valuation of access to professional support through DMHTs [ 39 ].

A total of 9 studies reported an interest in DMHT reminders and notifications. Across 3 studies, >70% of patients or caregivers were interested in appointment reminders [ 17 , 21 , 22 ]. In addition, 73% and 68% of patients with epilepsy reported interest in reminders for medication refills and adherence, respectively [ 17 ]. Beyond apps, caregivers of patients with MDD, BD, and schizophrenia preferred an ingestible pill sensor that tracked medication adherence, physical activity, mood, and rest 9.79 (95% CI 4.81-19.9), 7.47 (95% CI 3.81-14.65), and 6.71 (95% CI 3.29-13.69) times more than a nondigital pill organizer, respectively [ 16 ]. Qualitatively, patients and caregivers also appreciated reminders, especially if reasonably timed or delivered via text messages [ 27 , 31 ].

Short-term DMHT engagement was also supported by games and graphics, which could communicate information in an accessible way [ 24 ], provide tools for stress management [ 17 , 33 ], and be used therapeutically with children [ 20 , 30 ]. However, some HCPs and caregivers expressed concerns that graphics and games may be distracting for certain children ( Table 3 ) [ 20 ].

In a novel finding, 3 studies reported forgetfulness or lack of motivation as an influence on DMHT engagement. In some cases, disuse was related to stress, other MH symptoms, or poor technical literacy ( Table 3 ) [ 5 , 25 , 31 ]. In contrast, “forgetting to use” DMHTs and “lack of motivation” were perceived as relatively small barriers to use in the study by Stiles-Shields et al [ 37 ].

The third subtheme under engagement style was customizability, which was generally valued by users; 70.9% of a general population of smartphone users noted customization was an important factor [ 35 ]. Similarly, 9.4% of all surveyed veterans and 10.9% of those with smartphones reported disliking a prior DMHT due to a lack of personalization [ 32 ]. Users specifically wanted to be able to opt out of irrelevant features, customize audiovisual and design elements, add personal notes to tracked mood data, and provide ongoing feedback to facilitate personalization [ 20 , 24 , 28 , 31 , 34 ].

Theme 2: Background and Accessibility

A total of 16 (62%) studies reported findings related to DMHT background and accessibility, which considers the developer of the DMHT, as well as functionality and accessibility. Of these, 12 (75%) studies reported on the technical considerations subtheme, 9 (56%) on costs, and 2 (13%) on stability.

Under technical considerations, 9 studies assessed diverse accessibility concerns. Broadly, Storm et al [ 38 ] emphasized that DMHTs should be developed in consideration of patients’ social, cognitive, and environmental needs to avoid overwhelming users. Specifically, 2 studies reported language as a barrier. Deaf or hard-of-hearing participants recommended visual content presentation, such as videos and icons, alongside text and American Sign Language translations where possible [ 19 ]. Similarly, when discussing English-only apps, 1 provider stated as follows: “language is a barrier for some [patients]” [ 5 ]. Mobility issues related to MH symptoms or other conditions and technical literacy, such as difficulties remembering passwords and navigating smartphones or apps, created accessibility barriers as well ( Table 3 ) [ 5 , 25 , 27 , 28 ]. Additional concerns included apps that restricted use based on geographic location [ 19 ], user difficulty in finding relevant, useful apps [ 32 ], and limited mobile device memory for downloading apps [ 5 , 19 ].

Offline functionality, reported by 6 studies, was also captured under the technical considerations subtheme. A majority (5/9, 56%) of participants included in the study by Borghouts et al [ 19 ] expressed concern about their mobile data plans when using their devices. Correspondingly, “availability of Wi-Fi” was noted as a top barrier to the use of apps for depression by Stiles-Shields et al [ 37 ], and several veterans in another study reported that home Wi-Fi connectivity facilitated app use by eliminating cellular data fees [ 25 , 37 ]. Quotes from patients and HCPs echoed the concern about apps without offline functionality ( Table 3 ) [ 23 , 30 ].

Data fees were also captured under the costs subtheme, with hidden or additional costs described as a barrier to app use by 2 studies [ 26 , 37 ]. Parents of children with ADHD reported that difficulty paying phone bills could result in their phones being shut off, limiting DMHT use; one MH clinic administrator stated as follows: “We often encounter parents’ phones being shut off because they haven’t paid their bill...If the app were free or low cost, I imagine it could be very helpful” [ 26 ]. In addition to hidden costs, this quote identifies up-front app costs as a barrier. Quantitatively, more than half of a general population of surveyed college students expressed that cost was a top concern for the use of MH apps [ 34 ]. Qualitative findings from 2 additional studies likewise identified cost as a barrier to DMHT use [ 25 , 27 ].

Three novel cost attributes were identified by this SLR: willingness to pay, insurance restrictions, and cost savings compared with professional care. Four studies, 3 of which focused on apps, explored willingness to pay for DMHTs from a user perspective. Willingness to pay varied based on user preference; some surveyed college students and smartphone users among general populations valued free apps due to financial restrictions or uncertainty around app effectiveness, although 1 student commented that the quality of free trials might be inferior [ 34 , 35 ]. Some smartphone users also voiced a limit on how much they would be willing to spend for an app subscription ( Table 3 ) [ 35 ]. Forma et al [ 16 ] found that caregivers were willing to pay US $255.04 (95% CI US $123.21-US $386.86) more per month for a pill with an ingestible sensor that tracked medication adherence, physical activity, and rest and could connect to an app that also collected self-reported mood data. Moreover, the caregivers were willing to pay US $124.50 (95% CI US $48.18-US $200.81) more per month for an app-connected pill organizer alone than for a nondigital pill organizer [ 16 ]. In contrast, some veterans expressed total disinterest in paid apps, with 1 user citing poor technical literacy (“don’t have the knowledge”) in addition to cost as affecting willingness to pay [ 25 ].

In another novel finding, a speech-language pathologist working with children with ASD preferred a single app including multiple features over separate apps for particular features due to insurance restrictions: “I agree that teaching Apps should be an in-App feature versus their own app because sometimes insurance doesn’t allow us to open the iPads purchased through insurance” [ 20 ]. Although no further detail was provided for this finding, it suggests that there may be restrictions on the use of other apps on devices purchased under insurance, which may have implications for DMHT use in formal care settings due to the lack of financial support.

In a third novel cost-related finding, a small number of participants from a general population of students (3.6%) in one study preferred using an MH app to seeing an MH professional due to cost savings [ 29 ].

A total of 13% (2/16) of studies reported on the subtheme of app stability and technical difficulties, with crashes and poor display quality decreasing DMHT value [ 35 , 37 ]. Participants in the study by Schueller et al [ 35 ] reported that technical difficulties were often an issue for apps developed by medical institutions, which might be effective and safe but less usable than apps from other developers.

Theme 3: Privacy and Security

A total of 13 (50%) out of 26 studies reported findings related to the privacy and security theme, which covered the use and protection of user data by DMHTs. Subthemes were reported relatively equally: data collection and storage (5/13, 38%), personal health information (PHI; 5/13, 38%), privacy policies (4/13, 31%), general privacy (3/13, 23%), and security measures (3/13, 23%).

Quantitative and qualitative findings on general privacy (ie, evidence not categorized under any specific subtheme), the data collection and storage subtheme, and the privacy policies subtheme revealed heterogeneous concerns ( Table 3 ). A total of 74% of a general population of college students reported privacy as a top concern for MH apps, although further details on the specific area of concern were unclear [ 34 ]. In the study by Stiles-Shields et al [ 37 ], participants were highly concerned with data access but less so with general privacy. Echoing the concerns about data collection and storage, 59.1% of veterans with anxiety or MDD in 1 study were concerned about in-app PHI protection [ 32 ]; however, a qualitative study in veterans with posttraumatic stress disorder, alcohol use disorder, or MDD reported that a relatively small number of participants expressed privacy concerns. In the latter study, reasons for the concerns included distrust in Veterans Affairs, belief that digital data are inherently not confidential, and fear of phone hacking [ 25 ]. From an HCP perspective, none of the surveyed behavioral health HCPs agreed with the statement “My patients are concerned about data security,” despite multiple patients within the same study reporting privacy concerns [ 5 ].

Still, privacy policies were important overall, with 70.5% of smartphone MH app users rating having a privacy policy as “very important” or “important” [ 35 ]. Melcher et al [ 34 ] found that although users valued data protection, some reported a lack of awareness about data privacy, and others were concerned about obscure privacy policies and PHI use. As noted in the data collection and storage subtheme, veteran concerns about government use of PHI were heterogeneous [ 25 ].

A novel valuation factor not included in the APA framework related to user concern with PHI privacy and security regarding MH diagnoses and MH app use is a desire to upkeep their personal image or avoid stigma ( Table 3 ) [ 5 , 25 , 29 , 40 ]. For instance, 21.1% of a general college student population preferred MH app use to seeing an MH professional due to anonymity or reduced stigma [ 29 ]. One participant in a study of Veterans Affairs health service users described access to professional care via MH apps as convenient because they could avoid disclosing their use of MH services to explain leaving work early for an appointment [ 25 ].

In line with the overarching concern about PHI privacy and security, users valued app security measures. Schueller et al [ 35 ] reported that 74.2% of users rated data encryption as “important” or “very important.” Users in another study perceived the level of privacy protection as the highest for apps using a combination of a generic app name (ie, not reflecting the indicated MH disorder); easily hidden modules; and secure, user-authenticated web portals for making module changes [ 40 ]. Behavioral health clinic staff echoed the importance of discreet MH app names ( Table 3 ) [ 30 ].

Theme 4: Therapeutic Goal

There were 12 (46%) studies that reported on the factors relating to the integration of DMHTs with users’ therapeutic goals. The clinical actionability and therapeutic alliance subthemes were reported by 83% (10/12) and 58% (7/12) of studies, respectively.

A total of 9 studies reported the value of clinically actionable insights from apps where the users could acquire and practice new skills to make positive changes in their lives ( Table 3 ). For instance, patient and caregiver app users reported interests in “daily tips,” “new ideas,” and “solutions or recommendations” for symptom management [ 26 , 27 , 36 ]. Furthermore, an app that could serve as a resource for multiple management strategies was preferable [ 26 , 28 , 31 ]. Quantitatively, 4% of patients receiving acute treatment in a partial hospitalization program for MH conditions, including mood and psychotic disorders, reported that the primary purpose of their DMHT use was therapy skills practice [ 18 ]. HCPs similarly appreciated that DMHTs could facilitate patients practicing skills outside of formal treatment sessions [ 5 ]. In particular, clinicians from a youth behavioral health clinic noted that DMHTs might be especially beneficial for young users because they could be conveniently and discreetly incorporated into their daily lives [ 30 ].

Users valued easy data sharing with clinicians, particularly for mood- or symptom-tracking features, which could improve communication and the accuracy of symptom reporting during clinical visits [ 5 , 25 - 27 , 34 , 36 ]. For instance, 53% of a general college student population believed that the potential to share information with their clinician was “one of the top benefits” of using DMHTs [ 34 ]. In addition, many HCPs reported active use or interest in the use of DMHTs in clinical practice to facilitate asynchronous communication and increase patient engagement with treatments outside of formal appointments; however, some preferred traditional care strategies for their personalization and flexibility ( Table 3 ) [ 5 , 26 , 30 ].

Theme 5: Clinical Foundation

A total of 8 (31%) studies reported findings related to the clinical foundation of DMHTs, that is, their utility and appropriateness for patients. Clinical validity was the most reported subtheme, with evidence identified from 6 (75%) studies; 2 (25%) studies reported on the user feedback subtheme and 2 (25%) on the impressions of use subtheme, which captured users’ perceptions of app content as accurate and relevant.

Across subthemes, users valued evidence of DMHT benefit or efficacy from various sources. A total of 71.8% of surveyed veterans said that they would use a DMHT if they “saw proof that it worked” for their MH conditions [ 32 ]. Similarly, among the 811 general population participants surveyed, 69.5% ranked direct research evidence as “important” or “very important” for DMHT, and 66.8% ranked indirect research evidence the same [ 35 ]. Qualitative data identified recommendations from HCPs or academic institutions, as well as evidence of DMHT benefit from publications or research studies, as specific sources for clinically valid evidence of benefits ( Table 3 ) [ 27 , 34 , 35 ].

In addition to academic and professional support, the user feedback subtheme captured user interest in whether DMHTs were beneficial for peers or recommended by other trusted individuals. Patients with depression reported that other users’ experiences influenced their app use, with one user wanting to know “...if other people had success using it” [ 27 ]. Quantitatively, user ratings and user reviews were ranked as “important” or “very important” factors in DMHT use by 59.4% and 58.7% of the general population participants, respectively [ 35 ].

Quality Assessment

The risk of bias was overall moderate. Of the 14 studies including quantitative components, only 1 (7%) used relevant validated outcome measurement instruments [ 33 ]; all others used custom questionnaires. Of the 18 studies with qualitative components, 4 (22%) were at risk of selection bias due to participants being exclusively recruited using web-based postings and research registries [ 33 - 35 , 37 ], and only 1 (6%) considered the relationship between researcher and participant when interpreting the results [ 36 ]. Full quality assessments for qualitative and quantitative study components can be found in Tables S11 and S12 in Multimedia Appendix 1 , respectively.

Principal Findings

This SLR aimed to identify and synthesize qualitative and quantitative evidence on how DMHTs are valued by users, payers, and employers in the United States. Evidence from users with or without diagnosed relevant disorders, caregivers, and HCPs was captured across a wide range of demographics. No study reported evaluating an app from a payer or employer perspective. Furthermore, all but one included study focused on mobile apps.

No relevant appraisals of DMHTs were identified from the FDA website searches; however, 8 relevant FDA approval labels or notifications for MH apps or guidance documents for industry and FDA staff were identified. The content of these materials overlapped with some valuation factors identified in this SLR, including evidence of clinical efficacy and safety, app maintenance, and privacy and security.

Engagement style, although not covered by the FDA materials, was the most reported theme by the studies included in this SLR and was found to overlap heavily with other themes. Engagement may be a key consideration for app developers, as app user retention can be low: 1 study showed that >90% of users had abandoned free MH apps within 30 days of installation [ 41 ]. Engagement is also a key clinical concern in terms of DMHT efficacy; one meta-analysis of 25 studies showed that increased use of DMHT modules was significantly associated with positive outcomes regardless of the target MH condition [ 42 ]. The findings of this SLR may therefore be informative to both DMHT designers and HCPs who integrate DMHTs into clinical care by providing insight on DMHT valuation and thus how use and benefit can be improved. For instance, users valued DMHTs that were easy to use and aligned with their needs and priorities, particularly through features that supported their therapeutic goals. In addition, content presented through multiple delivery modes, such as both text and visuals, promoted engagement as well as accessibility.

However, engagement and feature preference varied across populations. For instance, DMHT valuation was affected by technical literacy, which may relate to user demographics; in this SLR, veterans repeatedly emphasized technical literacy as a barrier to DMHT use [ 25 ]. Similarly, offline functionality may be more important for some users. Although 85% of the total United States population owns smartphones, only 59% of Medicare beneficiaries have access to a smartphone with a wireless plan. Moreover, beneficiaries who are older, less educated, disabled, or Black or Hispanic have even lower digital access [ 43 , 44 ]. These findings emphasize the importance of customizability and suggest that app development and selection in the clinical setting should consider the demographics of the target population, particularly in relation to ease of use and offline functionality.

Background and accessibility findings also identified up-front and hidden costs as barriers to DMHT use, with the willingness to pay varying among individuals. This has important implications for app development, considering that many MH apps currently on the market are direct-to-consumer sales and require out-of-pocket payment. App developers often take this approach as it does not require the accumulation of formal evidence of clinical benefit for FDA approval [ 45 ], but it may present a financial barrier to use for consumers.

Privacy and security, reported by 13 (50%) out of 26 studies, was a prevalent theme, with users primarily concerned with data and PHI security within apps. This finding reflects wider research; a 2019 review of 116 depression-related apps retrieved from iTunes and Google Play stores in 2017 found that only 4% of the identified apps had acceptable transparency in privacy and security, with many completely lacking a privacy policy [ 46 ]. Similarly, 39% of MH apps recommended by college counseling centers had no privacy policy, and of those with a policy, 88% collected user data, and 49% shared that data with third parties [ 4 ]. Most evidence identified in this SLR under this theme, as well as findings previously published in the wider literature, focuses on these remote privacy risks. However, local privacy concerns are also important to users. In particular, inconspicuous naming and the ability to hide sensitive modules within MH apps were rated as highly important by both patients and HCPs to maintain user privacy. Users emphasized a desire to avoid the stigma associated with mental illness, which was also reflected by the findings in the engagement style theme: more young adults with psychosis were more interested in in-app messaging with other patients in psychosis recovery (67.1%) than a provider and family member together (47.3%) or their personal support network (59.8%) [ 22 ]. Similarly, youths were interested in apps that could be used discreetly in school or other public settings to avoid potential MH stigma. This is a key, novel finding of this SLR, considering that many app or DMHT components on the market are named after their target disorder.

The use of DMHTs to achieve therapeutic goals was discussed from patient, caregiver, and HCP perspectives, all of which valued DMHTs that had evidence of efficacy, presented clinically actionable information, and facilitated patient-clinician relationships. Of the 5 studies that explored how HCPs value DMHTs in clinical practice, 2 (40%) were restricted to the oncology or ASD settings and were not readily generalizable to wider MH settings [ 20 , 28 ]. In other studies, providers reported interest in using DMHTs to facilitate asynchronous communication with patients and their caregivers, promote patient skill practice, and improve care for children through the use of games and visuals [ 26 , 30 ]. However, while HCPs overall believed that DMHTs improved care, some believed that their clinical training allowed for care personalization beyond what DMHTs could provide. Feature customizability and receipt of input from HCPs and users during app development and testing may be a way to mitigate these concerns, as well as concerns about safety and efficacy, as many available apps do not appropriately address user health concerns [ 47 ].

Findings additionally suggested that training and resources on DMHTs would be beneficial to ensure that HCPs were equipped to integrate DMHTs into their practices [ 5 ]. Collaboration between DMHT specialists and HCPs, along with a shift from randomized controlled trials to effectiveness-implementation hybrid trials, may be a way to streamline the integration of DMHTs into clinical care and provide more training and resources for HCPs [ 30 , 48 ].

This review followed a prespecified protocol and used systematic methods in line with the York Centre for Reviews and Dissemination guidelines [ 49 ] to conduct an exhaustive search of the literature, identifying evidence relevant to the review objectives from multiple databases and supplementary sources. The framework synthesis approach allowed for the inclusion and analysis of both qualitative and quantitative data, providing a detailed picture of not only what DMHT features users value but why they value them, especially in areas where valuation varies across patient demographics. In addition, the APA framework is a robust model created with patient and HCP input that incorporates key valuation themes broadly shared by other frameworks and widely acknowledged in the literature [ 11 - 13 ].

Limitations

Methodological limitations should be considered when interpreting the findings of this SLR. Only publications in English and in United States populations were included. As perceptions of value are influenced by factors including cultures, laws, and health care settings, the findings of this SLR should not be generalized to other countries. For instance, trust in HCPs and rates of longstanding relationships between patients and primary care providers are lower in the United States than in many European nations [ 50 , 51 ], which could impact the type of support users want from DMHTs (ie, engagement style) or interest in DMHT integration with therapeutic goals.

In addition to the prespecified eligibility criteria, deprioritization strategies were implemented due to the large volume of the identified evidence, and this may have resulted in missing relevant articles. In particular, the deprioritization of secondary research and opinion pieces likely led to the exclusion of relevant discussion around payer perspectives and reimbursement, for which no evidence was included in this SLR. Furthermore, although unlikely, there may have been reporting biases in the included studies due to missing results, which this SLR was not able to assess.

This SLR identified no evidence for 3 subthemes included in the APA framework: business model (background and accessibility), which covers DMHT funding sources and potential sources of conflict, medical claims (background and accessibility), which examines whether DMHTs claim to be medical and the trustworthiness of their creators, and data ownership, access, and export (therapeutic goal), which includes sharing data with eHealth records or wellness devices (eg, Apple HealthKit [Apple Inc], Fitbit [Google LLC]). The valuation of these subthemes should be evaluated in future research.

Conclusions

In summary, app usability, cost, accessibility and other technical considerations, and alignment with therapeutic goals were the most reported valuation factors identified by this SLR. Many studies also reported user preference for apps that incorporated privacy and security features that provided protection from stigma. However, individual DMHTs and their features are valued differently across individuals based on demographics and personal preferences. MH apps should be developed and selected with these specific user needs in mind. Feature customizability and input from users and HCPs during development may improve app usability and clinical benefit.

Acknowledgments

The authors thank Max Lee, Costello Medical, US, for medical writing and editorial assistance based on the authors’ input and direction.

Conflicts of Interest

DCM is a consultant for Otsuka Pharmaceutical Development & Commercialization (OPDC) Inc for this project and has received consulting funds from Pear Therapeutics, Sanofi, Avidity, Sarepta, Novartis, and BioMarin. ML, HG, and HCW are employees of OPDC. JC, SB, RSK, and EW are employees of Costello Medical. This research was funded by OPDC.

Electronic database and supplementary search terms, systematic literature review eligibility criteria, publications excluded or deprioritized at full-text review, quality assessments of included studies, and the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram of the identified publications.

PRISMA checklist.

• Content of premarket submissions for device software functions: guidance for industry and Food and Drug Administration staff. U.S. Food and Drug Administration. Jun 14, 2023. URL: https://www.fda.gov/media/153781/download [accessed 2024-07-19]
• Patel NA, Butte AJ. Characteristics and challenges of the clinical pipeline of digital therapeutics. NPJ Digit Med. Dec 11, 2020;3(1):159. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Policy for device software functions and mobile medical applications: guidance for industry and Food and Drug Administration staff. U.S. Food and Drug Administration. URL: https://www.fda.gov/media/80958/download [accessed 2023-01-06]
• Melcher J, Torous J. Smartphone apps for college mental health: a concern for privacy and quality of current offerings. Psychiatr Serv. Nov 01, 2020;71(11):1114-1119. [ CrossRef ] [ Medline ]
• Hoffman L, Benedetto E, Huang H, Grossman E, Kaluma D, Mann Z, et al. Augmenting mental health in primary care: a 1-year study of deploying smartphone apps in a multi-site primary care/behavioral health integration program. Front Psychiatry. 2019;10:94. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Tong A, Flemming K, McInnes E, Oliver S, Craig J. Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Med Res Methodol. Nov 27, 2012;12(1):181. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. Jul 21, 2009;339(jul21 1):b2535. [ FREE Full text ] [ CrossRef ] [ Medline ]
• CASP critical appraisal checklists. Critical Appraisal Skills Programme. URL: https://casp-uk.net/casp-tools-checklists/ [accessed 2022-01-06]
• Carroll C, Booth A, Cooper K. A worked example of "best fit" framework synthesis: a systematic review of views concerning the taking of some potential chemopreventive agents. BMC Med Res Methodol. Mar 16, 2011;11(1):29. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Carroll C, Booth A, Leaviss J, Rick J. "Best fit" framework synthesis: refining the method. BMC Med Res Methodol. Mar 13, 2013;13(1):37. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kolasa K, Kozinski G. How to value digital health interventions? a systematic literature review. Int J Environ Res Public Health. Mar 23, 2020;17(6):2119. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lagan S, Aquino P, Emerson MR, Fortuna K, Walker R, Torous J. Actionable health app evaluation: translating expert frameworks into objective metrics. NPJ Digit Med. 2020;3:100. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lagan S, Emerson MR, King D, Matwin S, Chan SR, Proctor S, et al. Mental health app evaluation: updating the American Psychiatric Association's framework through a stakeholder-engaged workshop. Psychiatr Serv. Sep 01, 2021;72(9):1095-1098. [ CrossRef ] [ Medline ]
• Hong QN, Pluye P, Bujold M, Wassef M. Convergent and sequential synthesis designs: implications for conducting and reporting systematic reviews of qualitative and quantitative evidence. Syst Rev. Mar 23, 2017;6(1):61. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Beel J, Gipp B, Langer S, Genzmehr M. Docear: an academic literature suite for searching, organizing and creating academic literature. In: Proceedings of the 11th annual international ACM/IEEE joint conference on Digital libraries. 2011. Presented at: JCDL '11; June 13-17, 2011:565-566; Ottawa, ON. URL: https://dl.acm.org/doi/10.1145/1998076.1998188 [ CrossRef ]
• Forma F, Chiu K, Shafrin J, Boskovic DH, Veeranki SP. Are caregivers ready for digital? caregiver preferences for health technology tools to monitor medication adherence among patients with serious mental illness. Digit Health. 2022;8:20552076221084472. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Afra P, Bruggers CS, Sweney M, Fagatele L, Alavi F, Greenwald M, et al. Mobile software as a medical device (SaMD) for the treatment of epilepsy: development of digital therapeutics comprising behavioral and music-based interventions for neurological disorders. Front Hum Neurosci. 2018;12:171. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Beard C, Silverman AL, Forgeard M, Wilmer MT, Torous J, Björgvinsson T. Smartphone, social media, and mental health app use in an acute transdiagnostic psychiatric sample. JMIR Mhealth Uhealth. Jun 07, 2019;7(6):e13364. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Borghouts J, Neary M, Palomares K, de Leon C, Schueller SM, Schneider M, et al. Understanding the potential of mental health apps to address mental health needs of the deaf and hard of hearing community: mixed methods study. JMIR Hum Factors. Apr 11, 2022;9(2):e35641. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Boster JB, McCarthy JW. Designing augmentative and alternative communication applications: the results of focus groups with speech-language pathologists and parents of children with autism spectrum disorder. Disabil Rehabil Assist Technol. May 10, 2018;13(4):353-365. [ CrossRef ] [ Medline ]
• Buck B, Chander A, Monroe-DeVita M, Cheng SC, Stiles B, Ben-Zeev D. Mobile health for caregivers of young adults with early psychosis: a survey study examining user preferences. Psychiatr Serv. Aug 01, 2021;72(8):955-959. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Buck B, Chander A, Tauscher J, Nguyen T, Monroe-DeVita M, Ben-Zeev D. mHealth for young adults with early psychosis: user preferences and their relationship to attitudes about treatment-seeking. J Technol Behav Sci. 2021;6(4):667-676. [ CrossRef ] [ Medline ]
• Carpenter-Song E, Noel VA, Acquilano SC, Drake RE. Real-world technology use among people with mental illnesses: qualitative study. JMIR Ment Health. Nov 23, 2018;5(4):e10652. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Casarez RL, Barlow E, Iyengar SM, Soares JC, Meyer TD. Understanding the role of m-health to improve well-being in spouses of patients with bipolar disorder. J Affect Disord. May 01, 2019;250:391-396. [ CrossRef ] [ Medline ]
• Connolly SL, Miller CJ, Koenig CJ, Zamora KA, Wright PB, Stanley RL, et al. Veterans' attitudes toward smartphone app use for mental health care: qualitative study of rurality and age differences. JMIR Mhealth Uhealth. Aug 22, 2018;6(8):e10748. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Cummings JR, Gaydos LM, Mensa-Kwao A, Song M, Blake SC. Perspectives on caregiver-focused mHealth technologies to improve mental health treatment for low-income youth with ADHD. J Technol Behav Sci. Mar 9, 2019;4(1):6-16. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Dinkel D, Harsh Caspari J, Fok L, Notice M, Johnson DJ, Watanabe-Galloway S, et al. A qualitative exploration of the feasibility of incorporating depression apps into integrated primary care clinics. Transl Behav Med. Sep 15, 2021;11(9):1708-1716. [ CrossRef ] [ Medline ]
• Huberty J, Bhuiyan N, Neher T, Joeman L, Mesa R, Larkey L. Leveraging a consumer-based product to develop a cancer-specific mobile meditation app: prototype development study. JMIR Form Res. Jan 14, 2022;6(1):e32458. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kern A, Hong V, Song J, Lipson SK, Eisenberg D. Mental health apps in a college setting: openness, usage, and attitudes. Mhealth. Jun 2018;4:20. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Knapp AA, Cohen K, Nicholas J, Mohr DC, Carlo AD, Skerl JJ, et al. Integration of digital tools into community mental health care settings that serve young people: focus group study. JMIR Ment Health. Aug 19, 2021;8(8):e27379. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kornfield R, Meyerhoff J, Studd H, Bhattacharjee A, Williams JJ, Reddy MC, et al. Meeting users where they are: user-centered design of an automated text messaging tool to support the mental health of young adults. In: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 2022. Presented at: CHI '22; April 29-May 5 2022:1-6; New Orleans, LA. URL: https://dl.acm.org/doi/abs/10.1145/3491102.3502046
• Lipschitz J, Miller CJ, Hogan TP, Burdick KE, Lippin-Foster R, Simon SR, et al. Adoption of mobile apps for depression and anxiety: cross-sectional survey study on patient interest and barriers to engagement. JMIR Ment Health. Jan 25, 2019;6(1):e11334. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mata-Greve F, Johnson M, Pullmann MD, Friedman EC, Griffith Fillipo I, Comtois KA, et al. Mental health and the perceived usability of digital mental health tools among essential workers and people unemployed due to COVID-19: cross-sectional survey study. JMIR Ment Health. Aug 05, 2021;8(8):e28360. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Melcher J, Camacho E, Lagan S, Torous J. College student engagement with mental health apps: analysis of barriers to sustained use. J Am Coll Health. Oct 13, 2022;70(6):1819-1825. [ CrossRef ] [ Medline ]
• Schueller SM, Neary M, O'Loughlin K, Adkins EC. Discovery of and interest in health apps among those with mental health needs: survey and focus group study. J Med Internet Res. Jun 11, 2018;20(6):e10141. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Schueller SM, Neary M, Lai J, Epstein DA. Understanding people's use of and perspectives on mood-tracking apps: interview study. JMIR Ment Health. Aug 11, 2021;8(8):e29368. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Stiles-Shields C, Montague E, Lattie EG, Kwasny MJ, Mohr DC. What might get in the way: barriers to the use of apps for depression. Digit Health. Jun 08, 2017;3:2055207617713827. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Storm M, Venegas M, Gocinski A, Myers A, Brooks J, Fortuna KL. Stakeholders' perspectives on partnering to inform the software development lifecycle of smartphone applications for people with serious mental illness: enhancing the software development lifecycle through stakeholder engagement. In: Proceedings of the 2021 IEEE Global Humanitarian Technology Conference. 2021. Presented at: GHTC '21; October 19-23, 2021:195-199; Seattle, WA. URL: https://ieeexplore.ieee.org/document/9612444 [ CrossRef ]
• Torous J, Wisniewski H, Liu G, Keshavan M. Mental health mobile phone app usage, concerns, and benefits among psychiatric outpatients: comparative survey study. JMIR Ment Health. Nov 16, 2018;5(4):e11715. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zhou L, Parmanto B. User preferences for privacy protection methods in mobile health apps: a mixed-methods study. Int J Telerehabil. Dec 08, 2020;12(2):13-26. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Baumel A, Muench F, Edan S, Kane JM. Objective user engagement with mental health apps: systematic search and panel-based usage analysis. J Med Internet Res. Sep 25, 2019;21(9):e14567. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gan DZ, McGillivray L, Han J, Christensen H, Torok M. Effect of engagement with digital interventions on mental health outcomes: a systematic review and meta-analysis. Front Digit Health. 2021;3:764079. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mobile fact sheet. Pew Research Center. 2021. URL: https://www.pewresearch.org/internet/fact-sheet/mobile/ [accessed 2024-04-29]
• Roberts ET, Mehrotra A. Assessment of disparities in digital access among Medicare beneficiaries and implications for telemedicine. JAMA Intern Med. Oct 01, 2020;180(10):1386-1389. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Powell AC, Torous JB, Firth J, Kaufman KR. Generating value with mental health apps. BJPsych Open. Feb 05, 2020;6(2):e16. [ FREE Full text ] [ CrossRef ] [ Medline ]
• O'Loughlin K, Neary M, Adkins EC, Schueller SM. Reviewing the data security and privacy policies of mobile apps for depression. Internet Interv. Mar 2019;15:110-115. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Akbar S, Coiera E, Magrabi F. Safety concerns with consumer-facing mobile health applications and their consequences: a scoping review. J Am Med Inform Assoc. Feb 01, 2020;27(2):330-340. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. Mar 2012;50(3):217-226. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Systematic reviews: CRD's guidance for undertaking reviews in health care. Centre for Reviews and Dissemination. 2008. URL: https://www.york.ac.uk/media/crd/Systematic_Reviews.pdf [accessed 2024-04-29]
• Gumas ED, Lewis C, Horstman C, Gunja MZ. Finger on the pulse: the state of primary care in the U.S. and nine other countries. The Commonwealth Fund. URL: https:/​/www.​commonwealthfund.org/​publications/​issue-briefs/​2024/​mar/​finger-on-pulse-primary-care-us-nine-countries [accessed 2024-04-29]
• Huang EC, Pu C, Chou YJ, Huang N. Public trust in physicians-health care commodification as a possible deteriorating factor: cross-sectional analysis of 23 countries. Inquiry. Mar 05, 2018;55:46958018759174. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by J Torous; submitted 15.02.24; peer-reviewed by A Mathieu-Fritz, K Stawarz; comments to author 05.05.24; revised version received 20.06.24; accepted 21.06.24; published 30.08.24.

©Julianna Catania, Steph Beaver, Rakshitha S Kamath, Emma Worthington, Minyi Lu, Hema Gandhi, Heidi C Waters, Daniel C Malone. Originally published in JMIR Mental Health (https://mental.jmir.org), 30.08.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on https://mental.jmir.org/, as well as this copyright and license information must be included.

Top of page

Library of Congress Digitization Strategy: 2023-2027

February 13, 2023

Posted by: Trevor Owens

Share this post:

The following post was co-authored with Steve Morris, Chief of Digital Collections Management and Services and Tom Rieger, Manager of Digitization Services. 

The Library of Congress has a new Digitization Strategy for its collections. As we did for the Library’s Digital Collections Strategy , we are excited to share this overview of it with readers of The Signal blog. We get a lot of questions about what we digitize and why, and hopefully this provides a little bit of insight into our institutional plans and priorities.

The Library has expanded the amount and throughput of our digitization efforts dramatically over the past three decades. In 2020 we finished digitizing the last of our presidential papers – all of the personal papers of the presidents from George Washington to Calvin Coolidge are now available to anyone with an internet connection. In 2021, we opened a new Digital Scan Center, which significantly increased digital image production capabilities and postproduction processes. So far, we have digitized more than nine million items in our collections, with particular strengths in newspaper issues, manuscripts, and pictorial materials.

Over the next five years, the Library will expand, optimize, and centralize its collections digitization program to significantly expand access to users across the country to rare, distinctive, and unique collection materials which can be made openly available online and use digitization as a core method for preservation reformatting of rights restricted collection materials. Below are the five guiding strategic objectives for this work.

Strategic Objectives

• Transition from one-off project-based digitization toward the systematic digitization of rare, distinctive, or unique materials in the Library’s collections that can be made openly available online : Over the last two decades the Library has scaled up capacity to support major digitization projects using a targeted project-based approach, with priorities established according to a variety of factors. Evolving user expectations necessitate a more systematic approach that prioritizes material that can be made openly available online and that is not already available online from other institutions.
• Systematically deploy digitization as a core method for preservation and collection management workflows : Over the last five years, the Library has made significant progress in transitioning preservation reformatting from the production of print facsimiles or microfilm to digitization. This shift not only meets current preservation requirements, but it also supports enhanced onsite or offsite access. Going forward, the Library will use digital systems for reformatting, and will integrate digital delivery into collections management processes.
• Enhance the reach of library programs, exhibitions, events, and special initiatives by establishing a rapid-response focused and targeted projects digitization program in support of priority initiatives : A wide range of Library programs, events, and special initiatives draw on and offer opportunities to deepen engagement with the Library’s collections. Enhancing processes and capacity to support a range of rapid-response digitization projects will support these efforts.
• Review and improve the operational effectiveness, organizational structure, governance and prioritization processes for digitization efforts in support of the increasingly central role of digitization for access and preservation: As digitization has become an increasingly important technology for access to and preservation of Library of Congress collections, a wide range of organizational units have developed capacity to support this work on a large scale. It is critical to further mature digitization into core areas of the overall collections program. This includes but is not limited to clarifying roles for prioritizing specific digitization initiatives, and affirming requirements for technical standards.
• Implement end-to-end tracking and reporting on the status of digitization of collection items: The Library has for many years measured digitization efforts in terms of total counts of master files produced. This is a useful metric, but it does not align with other methods for counting Library collection items. Further, tracking total counts of master files produced does not necessarily mean that those images have been fully processed and been successfully made available to Library users. Over the next five years, the Library will implement methods and approaches to fully track digitization of collection materials from the beginning of projects to the point where they have been successfully made available to users.

Comments (6)

Does the Library accept (or has it considered accepting) digitization nominations from researchers or the general public? If so, how would one suggest material to be considered? In any event, thanks to all at the LOC who are making items in the collection accessible to anyone with an internet connection.

It’s exciting to see the expansion of digitization at the Library and to envision the possibility that the entire physical collection will have a digital complement.

To keep up with this wealth of digital assets, the Library needs to invest more time, effort, technology, and creativity to make them more easily and productively discoverable. It’s not enough to match search terms to a list of disconnected results.

The user experience viewing a digitized document can also be improved. Currently the viewing options all give equal weight to each image and assume that the only way to consume information is in page-order sequence. That is not how we read books, periodicals, or newspapers, much less reference works.

Thank you to the Library of Congress for doing such a nice job in bringing us all these services! I have been very impressed by all the audio reading/listening material which has been made available, and look forward to lots more of it.

Dear Library of Congress,

I am a reference archivist at UNESCO (United Nations Educational, Scientific and Cultural Organization). I am part of the research network InterPARES Trust AI, running out of the University of British Columbia. As a partner, UNESCO Archives is leading a study on digitization and AI and developing a model for sustainable digitization projects. We would like to reference the LoC Digitization Strategy 2023-2027. To this end, I would like to ask the following two questions:

1) Is it possible to have a copy of your strategy or to speak with someone about your strategy?

2) We will be disseminating a survey on digitization and AI in early March, and invite you to respond to the survey. If you are willing, could I please have the contact email address of the person within your organization who could respond to the survey? The survey seeks to understand the current state of digitization projects and programs across different types of organizations, whether and how organizations are digitizing for long-term sustainability, and to understand whether organizations are beginning to use AI in digitization.

Thank you in advance. Kind regards, Eng Sengsavang (she/her) Reference Archivist UNESCO Archives

Hi Eng, Thanks for your comment. This sounds like an interesting initiative. I will follow up with you over email about the best points of contact on this.

Is the full text of this strategy published? Where can I find it?

See All Comments

Add a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Towards the Increasing Usage of IPv6 & Its Implication: A Literature Review

New citation alert added.

This alert has been successfully added and will be sent to:

You will be notified whenever a record that you have chosen has been cited.

To manage your alert preferences, click on the button below.

New Citation Alert!

Please log in to your account

Information & Contributors

Bibliometrics & citations, index terms.

Network protocols

Network layer protocols

• Routing protocols

Recommendations

Ipv6 security challenges.

IPv6, the new version of the Internet Protocol, has been developed to provide new services and to support the Internet's growth. An overview of the key security issues outlines the challenges in deploying and transitioning to IPv6.

Green Computing in WAN Through Intensified Teredo IPv6 Tunneling to Route Multifarious Symmetric NAT

IPv4---IPv6 transition rolls out numerous challenges to the world of Internet as the Internet is drifting from IPv4 to IPv6. IETF recommends few transition techniques which includes Dual stack, translation and tunneling. By means of tunneling the IPv6 ...

Bi-directional Mapping System as a New IPv4/IPv6 Translation Mechanism

A new Internet protocol (IPv6) is being deployed with new features and improvements for the devices that are connected to the network. Unfortunately, the current Internet Protocol (IPv4) and the new Internet Protocol (IPv6) are incompatible with each ...

Information

Published in.

Association for Computing Machinery

New York, NY, United States

Publication History

Permissions, check for updates, author tags.

• Internet Protocol
• Internet Security Association and Key Management Protocol (ISAKMP)
• Research-article
• Refereed limited

Contributors

Other metrics, bibliometrics, article metrics.

• 0 Total Citations
• 0 Total Downloads
• Downloads (Last 12 months) 0
• Downloads (Last 6 weeks) 0

View Options

Login options.

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

View options.

View or Download as a PDF file.

View online with eReader .

HTML Format

View this article in HTML Format.

Share this Publication link

Copying failed.

Share on social media

Affiliations, export citations.

• Please download or close your previous search result export first before starting a new bulk export. Preview is not available. By clicking download, a status dialog will open to start the export process. The process may take a few minutes but once it finishes a file will be downloadable from your browser. You may continue to browse the DL while the export process is in progress. Download
• Download citation
• Copy citation

We are preparing your search results for download ...

We will inform you here when the file is ready.

Your file of search results citations is now ready.

Your search export query has expired. Please try again.