research paper on mobile application

Mobile application development process: A practical experience

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

• Open access
• Published: 09 August 2021

The use of mobile applications in higher education classes: a comparative pilot study of the students’ perceptions and real usage

• David Manuel Duarte Oliveira   ORCID: orcid.org/0000-0002-8763-6997 1 ,
• Luís Pedro 1 &
• Carlos Santos 1  

Smart Learning Environments volume  8 , Article number:  14 ( 2021 ) Cite this article

23k Accesses

6 Citations

3 Altmetric

Metrics details

This paper was developed within the scope of a PhD thesis that intends to characterize the use of mobile applications by the students of the University of Aveiro during class time. The main purpose of this paper is to present the results of an initial pilot study that aimed to fine-tune data collection methods in order to gather data that reflected the practices of the use of mobile applications by students in a higher education institution during classes. In this paper we present the context of the pilot, its technological settings, the analysed cases and the discussion and conclusions carried out to gather mobile applications usage data logs from students of an undergraduate degree on Communication Technologies.

Our study gathered data from 77 participants, taking theoretical classes in the Department of Communication and Arts at the University of Aveiro. The research was based on the Grounded Theory method approach aiming to analyse the logs from the access points of the University. With the collected data, a profile of the use of mobile devices during classes was drawn.

The preliminary findings suggest that the use of apps during the theoretical classes of the Department of Communication and Art is quite high and that the most used apps are Social networks like Facebook and Instagram. During this pilot the accesses during theoretical classes corresponded to approximately 11,177 accesses per student. We also concluded that the students agree that accessing applications can distract them during these classes and that they have a misperception about their use of online applications during classes, as the usage time is, in fact, more intensive than what participants reported.

Introduction

The use of mobile devices by higher education students has grown in the last years (GMI, 2019 ). Technological advancements are also pushing society with consequent rapidly changing environments. Higher Education Institutions (HEIs) are not exempted from these technological changes and advancements, and it is compulsory that they follow this technological evolution so that the teaching-learning process is improved and enriched.

HEI’s are trying to integrate digital devices such as mobile phones and tablets, and informal learning situations, assuming that the use of these technologies may result in a different learning approach and increase students’ motivation and proficiency (Aagaard, 2015 ).

In a study by Magda, & Aslanian ( 2018 ), students report that they access course documents and communicate with the faculty via their mobile devices, such as smartphones. Over 40% say they perform searches for reports and access institutions E-Learning systems via mobile devices (Magda, & Aslanian, 2018 ). The EDUCAUSE Horizon Report - 2019 Higher Education Edition (Alexander et al., 2019 ) also mentions M-Learning as the main development in the use of technology in higher education. However, teachers believe students use their gadgets less than they actually do, and mobile devices also challenge teaching practices. Students use devices for off-task (Jesse, 2015 ) or parallel activities and there may be inaccurate references to their actual use of mobile devices.

Mobile device users have very different usage habits of their devices and their applications, and it is important to study and characterize these behaviours in different contexts, as explained below. The reports that usually support these studies are made with questions directed to the users themselves asking them questions about the apps they have on the devices and the reasons for using them. However, Gerpott & Thomas ( 2014 ) argue that other types of studies are needed to properly support this type of research.

Studies are usually conducted in organizations, based on the opinion of the participants, and cannot be replicated and generalized, for example, regarding the use of the internet or mobile applications by the general public, because these devices, unlike desktop devices, can be used anywhere and at any time (Gerpott & Thomas, 2014 ).

Furthermore, in mobile contexts, it becomes difficult for people to remember what they have used, because mobile applications can be used for various tasks, in various contexts, whether professional or personal, and the variety of applications, the use made, the periods of use are usually so wide and differentiated, that it can become difficult for users to refer which services or applications they have used, under which circumstances and how often. (Boase & Ling, 2013 ).

Thus, it is relevant, for several areas and especially for this research area, to have studies that cross-reference reported usage with actual usage. One of the ways to achieve this is with the use of logs of the use of mobile devices and applications, as mentioned by De Reuver & Bouwman ( 2015 ):

Using this approach this pilot study aims to create and validate a methodology:

i) to show the profile of these users,

ii) to reveal the kind of applications they use in the classroom and when they are in the institutions,

iii) and also, to compare the users’ perceptions with the real use of mobile applications.

Knowing the real usage and the usage students mention may provide valuable insights to teachers and HEIs and use this data for decision making about institutional applications to support students and teachers in their teaching and learning activities. Such information can also bring insights on the integration of M-Learning strategies, promoting interaction, communication, access to courses and the completion of assignments using students’ devices.

The central focus of this study is, therefore, to show preliminary results of the use of applications by students in class time during theoretical classes, through logs collected during class time.

The paper is divided into five parts. In the first part, relevant theoretical considerations are addressed, having in mind the current state of the art in terms of the literature and empirical work in this field. The second part outlines the study methodology. In the third part, the technological setting is highlighted. The cases and the results of the data analysis are described in the fourth part. Lastly, the results are interpreted, connected and crossed with the preliminary considerations.

Literature review

The massive use of mobile devices has created new forms of social interaction, significantly reducing the spatial difficulties that could exist, and today people can be reached and connected anytime and anywhere (Monteiro et al., 2017 ). This also applies to the school environment, where students bring small devices (smartphones, tablets and e-book readers) with them, which, thanks to easy access to an Internet connection, keep them permanently connected, even during classes.

In HEIs there is also a growing tendency among members of the academic community to use mobile devices in their daily activities (Oliveira et al., 2017 ), and students expect these devices to be an integral part of their academic tasks, too (Dobbin et al., 2011 ). A great number of users take advantage of mobile devices to search information and, since they do not always have computers available, these devices allow them an easy access to academic and institutional information (Vicente, 2013 ).

One of the challenges educational institutions face today has to do with the ubiquitous character of these devices and with finding ways in which they can be useful for learning, thus approaching a new educational paradigm: Mobile Learning (M-Learning) (Ryu & Parsons, 2008 ).

M-learning allows learning to take place in multiple places, in several ways and when the learner wants to learn. As learning does not necessarily have to occur within school buildings and schedules, M-Learning reduces the limitations of learning confined to the classroom (Sharples, M., Corlett, D. & Westmancott,  2002 ), leading UNESCO to consider that M-Learning, in fact, increases the reach of education and may promote equality in education (UNESCO, 2013 ). The EDUCAUSE Horizon Report - 2019 Higher Education Edition (Alexander et al., 2019 ) also mentions M-Learning as the main development in the use of technology in higher education and, therefore, it becomes increasingly relevant to rethink learning spaces in a more open perspective, both physically and methodologically, namely through mobile learning that places the student at the centre of the learning process.

Quite often studies that intend to determine the use of mobile applications focus on general questions, but the most common ones are related to the frequency and duration of the use of these devices, for example, questions such as “how many SMS or calls are made?” or “how often do you use the device?”

In fact, instruments like questionnaires are widely used in this type of studies. However, since mobile devices are completely integrated in our daily life and we use them quite extensively, it is difficult to retain and define with plausible accuracy the actual use that we make of them.

It is therefore relevant to effectively understand how these students use these devices, more specifically the applications installed on them. To this end, most studies have been based on designs that are focused on the users’ perceptions and based are on these reports.

Thus, it was important to understand if what users report using corresponds to what they actually use, and if this use does not occur for distraction or entertainment, for example.

Considering the above, some studies have focused on the validity of the use of these instruments. One of these first studies, carried out by Parslow et al. ( 2003 ), aimed at determining the number of calls made and received in the days, weeks or months preceding the date of the questionnaire, and their duration. The answers were compared with the logs of the operators and it was concluded that self-report questionnaires do not always represent the actual pattern of use.

Finally, in self-report instruments, which refer to questions of daily activity on mobile devices, this activity may not represent a general pattern of activity, since from individual to individual the patterns of daily use may vary considerably and thus reflect a very irregular use.

In a study by Boase & Ling ( 2013 ), the authors mentioned that about 40% of studies on mobile device use, based on articles published in journals (41 articles between 2003 and 2010), are based on questionnaires.

The questions asked aim to estimate how long or what type of use they have made of their devices on a daily basis, and sometimes aim to know about time periods of several days. In most of these studies, 40% of papers use at least one measure of frequency of use and 27% a measure of duration of use that users make. Another factor that is mentioned is that users do not always report their usage completely accurately. On the other hand, the same study mentions that users may over or under report the use they make for reasons of sociability (Boase & Ling, 2013 ).

Given the moderate correlation between self-report instruments and data from records or logs (Boase & Ling, 2013 ), the author considers that researchers can significantly improve the results if they use, together with other instruments, data from logs to make their studies more accurate and rigorous. Another suggestion would be the use of mobile applications that record these usage behaviours (Raento et al., 2009 ).

Indeed, this kind of instrument is widely used in this type of studies. However, given that mobile devices are fully integrated into our daily lives and we use them quite extensively, it becomes difficult to retain and define with plausible accuracy the use we make of them. In addition to the factors mentioned in the previous paragraph, it is important that these types of studies are validated with other methods, such as the use of logs, as presented in this study. The logs in this study refer to the capture records of the mobile device traffic made by the students.

This article therefore aims to present preliminary results with an approach that uses cross-checking of log data with questionnaire results.

Methodology

This article intends to present and discuss preliminary results of a study that aims to characterize the use of mobile applications at the University of Aveiro through collected logs, crossing its results with questionnaires answered by students during the classes, and also with an observation grid with data from the analysed class and questions to teachers related to what teachers recommend regarding the use of mobile phones during class time.

The research question that motivated this article is: which digital applications/services are most frequently used on mobile devices by the students of the University of Aveiro during their classes?

The study was composed of 40 students, that answered the questionnaires.

The research was based on the Grounded Theory method aiming to analyse the logs from the access points of the University. With the collected data, a usage profile of mobile devices during classes was drawn.

Figure  1 presents a diagrammatic representation of the created methodological process.

General diagram of the study

Therefore 3 instruments were used for the data collection: a questionnaire, an observation grid and logs collected through mobile traffic in the wi-fi network of the university.

The questionnaire allowed for a quantitative assessment of the profile of the participants and collected data on the use that participants claimed to make of their mobile devices. The observation grid served as a guide for the implementation of the study, allowing to record data on the classes where the collections took place and to verify whether certain items were present, such as permission to use mobile devices or planning to use them by teachers. The observation grid would also serve to make the link between use and content in class, but in this pilot, it was not possible to make this link between the class content and the usage of mobile applications, because the author could not observe the applications used by students.

The database containing the usage records enabled the analysis of the logs, resulting in the quantification and verification of the type of activity that each (anonymous) participant made of their device.

The 3 instruments used aimed to i) determine which application(s) students were really using during the classes, through the analysis of the data logs collected from the Wi-Fi network of the University; ii) identify the participants’ representations of their activities by means of several questions regarding mobile usage during class time; iii) observe students’ behaviour and focus via an observation grid that was used by the researcher/observer when he was attending the classes.

The group who participated in this pilot study was selected in accordance with the professors and classes available, so it is considered a convenience sample. The group was constituted by students of undergraduate classes from the Communication and Arts Department of the University of Aveiro.

Table  1 summarizes the schedule of the pilots carried out, the curricular units where they took place, their duration and the instruments used. For ease of management, all the pilots took place in the same department of the University.

The Table  2 summarizes the collected data from questionnaires and logs.

This pilot aimed to build an approach to data analysis, close to the Grounded Theory methodology, in which a provisional theory is built based on the observed and analysed data (Alves et al., 2017 ; Long et al., 1993 ). The data collected in this pilot will serve to define a more complete methodology to be used in a larger study.

This chapter is divided into three parts: context, technological setting and cases analysed. In the context part, the classes which are part of the study will be described, relating the answers from the questionnaires with the teachers’ recommendations about the use of mobile devices. In the technological scenario section, it is intended to describe the technological background underlying the collection process of the logs and in the last part, analysed cases, the objective was to validate if the data to be collected matched the outlined objectives.

In the questionnaire, the questions were divided into two main groups: aspects related to the participant’s profile and aspects directly related to the use of the applications. Aspects related to participants were intended to characterize them. Regarding the use of applications, we aimed to find out the students’ perception of the applications they use in their daily routine, inside and outside of the classroom, and how they do it. Data were collected using a Google Forms form and processed using Microsoft Excel.

In this subchapter, through the data collected from the students’ answers to the questionnaires, and by crossing this information with the data collected from the teachers in the observation grid, we try to describe the context of the pilot.

All of the teachers stated that they allowed their students to use mobile phones during class time, but that they did not plan that use. They also stated that in most part of the classes several students use their mobile phones and apps to search for class related materials. The teachers also showed curiosity about knowing, with more detail, the mobile phone use their students actually have.

In the three classes analysed (Aesthetics, Scriptwriting and Music in History and Culture), when asked about the possibility of using mobile applications as a pedagogical complementary resource 43%, 47% and 55% of students fully agreed that these should be used. In these three classes, 31%, 44%, and 67% of students showed a more moderate opinion: they agreed (but not in such an assertive way) that these should be used.

Another conclusion is that most of the students used a smartphone (88,9%, 75%, 52%) during class time, but many of them also used a computer (66,7%, 100%, 84%). The percentage use of tablets is much lower (11,1%, 0%, 15%).

In the analysed scenario, the majority of the students used the android operating system and 94% also agreed that mobile applications could help to manage the academic tasks, except in the case of the “Aesthetics Curricular Unit”.

When it comes to the time of use, per week, in classes, 53%, 58%, and 22% of the students answered they used these devices between 4 to 5 days a week and 15%, 40% and 70% said they used them between 1 to 3 days a week.

Students were also asked about how frequently they accessed mobile applications during class time and, in all, 77% of the respondents reported accessing apps at least between 1 to 5 times per class. About 20% referred they accessed apps from 6 to 10 times per class.

As for the purposes of accessing apps during classes, most students mentioned categories related i) to support the class / to research (70%, 100%, 77,8%), ii) to access institutional platforms (47.4%, 66.7%, 89, 9%), iii) to access to information (47.4%, 50%, 66.7%) and iv) to work (36.8%, 50%, 44.4%).

Interestingly, the categories communication (52.6%, 41.7%, 22.3%), collaboration (10.5%, 16.7%, 0%), access to institutional services (5.3%, 0% 0%) and “I do not use them” (10.5%, 0%, 0%) presented very low percentages, namely the last one.

When questioned about the use of mobile devices that did not include academic reasons, many students referred to the categories “to be linked/connected” or “to be updated” (42.1%, 66.7%, 33.3%), “to communicate” (57.7% 75.7%, 66.7%), “to share and access content” (31.6%, 58.3%, 33.3%), but few mentioned “for entertainment” (26.3%, 16.7%, 22.2%), “as a habit or routine” (10.5%, 41.7%, 11.1%) and “I do not use them” (10.5%, 0%, 11.1%).

When asked about which mobile applications are most used in an academic context, the most relevant category was “to research / to study” (73.7%, 58.3%, 89.9%), “to check the calendar” (31.6%, 25%, 66.7% %) and “to surf the web” (47.4%, 50%, 55.6%). Again, categories such as “to work” (36.8%, 33.3%, 33.3%), “to take notes” (26,2%, 33.3%, 55.6%) and “to create content” (31.6%, 25%, 11.1%) presented relatively low percentages. It should also be noted that the respondents presented answers that created categories which were not expected such as “to watch films” (10.5%, 8.3%, 0%), “to listen to music” (31.6%, 33.3%, 33.3%), “to take photos” (10.5%, 0%, 0%) and “to play games” (5.3%, 0%, 0%) All the students said that they used applications during classes in at least one of the categories. In fact, in the three courses no one stated “not to use them” (0% in all).

When asked about the teachers’ permission to use the mobile devices in the classroom, most of the students said that teachers allowed free use (52.6%, 100%, 77.8%). Only a few stated that teachers allowed using them specifically when planned (41, 1%, 0%, 22.2%). The respondents of one course stated that teachers did not allow the use of devices (Aesthetics - 5.3%). Finally, when asked about the usefulness of integrating mobile applications in class, there was an overwhelming majority of respondents (100%, 78,9%, 100%) saying they believed that such integration could be enriching and useful.

Below is presented a table describing the most used mobile apps during class activities. It should be noted that only the two answers with the greatest relevance for each category were considered.

Table  3 systematizes what the results have been showing until now: there is an important part of students that use mobile phones during their classes and, even when teachers advise them not to use them, they ignore the recommendations and use them anyway. The main purposes stated were: to be in contact with others through social networking but also to access different kinds of information in browsers. Moreover, the classes where the use of devices is not recommended by the teachers seems to be the one where some applications are most used.

Technological setting

In this section we intend to describe the technological background underlying the process of collecting the logs. The first goal was to register and capture logs from the wi-fi network of the university, which consists of a wireless network that users can access using their universal user credentials.

In order to do that a meeting was scheduled with the university’s technology services, as our main concern was the anonymization of the data collected in order (i) to confer more neutrality to the data treatment, and (ii) to comply with European data protection legislation. Another issue for discussion was the need of powerful machines so that they could process the large amount of data collected.

In this meeting the necessary steps were agreed in order to guarantee the users’ privacy, the authorization of the university’s central services to do the study and the registration method of the logs. The overall procedure demanded several experiences of data collection to fine-tune the final pilot, which works as the basis capture setting for all the main study.

The Wi-Fi traffic capture software (Wireshark) was selected to work both with Android and IOS devices and it was possible to understand the functionalities of the software.

The pilot also helped to understand and solve additional problems that appeared during the previous tests, related to the anonymization of the users’ data. It was necessary to ensure that the users’ personal data were not identifiable, which was a commitment: in fact, only HTTPS Footnote 1 traffic was captured, being all the other information encrypted.

After the first tests, an initial data collection pilot took place in a classroom context. A specific capture system was created to allow the capture of mobile application logs used only by a certain group of students, from a designated Curricular Unit. A specific scenario was set up to ensure that only those students communicating through the IP Footnote 2 defined for the scenario and during that class time were considered and treated under the scope of this study:

If the traffic of the concerned student is communicating through one of the APs (Access Points) covering the room, then the device will be assigned a “Room network” IP;

If the student’s traffic is not communicating through one of the APs covering the room, then the device will be assigned a “Non Room network” IP;

If the student traffic does not belong to the group to be analysed and the device in question is communicating through one of the APs covering the room, then the device will be assigned an IP from a “normal eduroam network”;

In the final steps we resolved the IP’s in Wireshark (software used for the capture) and the unsolved IP’s where filtered in a PHP Footnote 3 script, through the gethostbyaddr method where the unsolved ones are incrementally added.

Finally, using an IP list, we performed a comparison to resolve any unresolved names;

This step allowed to fine tune the process and to make the final test.

Analysed cases

After performing these tests, a scenario for this final pilot was set up to validate if the data to be collected matched the outlined objectives. In this final pilot, logs were collected in a classroom so that the scenario was as close to the desired collection as possible. In this pilot, it was possible to verify that the collected data fulfilled the requirements. At this point, in addition to the HTTPS traffic packets, the packets referring to DNS Footnote 4 traffic were also included. This option made the HTTPS traffic more easily understandable. Furthermore, the researcher could conclude that all authenticated devices belonged to separate accounts.

The results show that the pre-tests/pilots and the final pilot turned out very well and in a very reliable way since they allowed to verify the main problems that could occur and helped to certify that the traffic anonymity condition was respected. In fact, only the HTTPS was considered, and all other communication was encrypted with no risk of corruption of private data. Moreover, this option had an important justification: the fact that HTTPS traffic could be more easily understandable and the fact that it allowed certifying that all the authenticated devices of the wireless network belonged to separate accounts.

To process and create output visualization of the data, the choice was an integrated solution, both for the processing stage and for creating visualisations. Given the variety of tools available, several were tried out and Tableau Software® (Tableau Prep® and Tableau Desktop®) was chosen. Tableau Software is an interactive data processing and visualisation tool that belongs to the Salesforce company and, although it is paid software, it allows for an academic licence that was used in this project.

This solution, besides allowing working with a large amount of data, also allows for a very interactive data treatment and visualisation. This software also allows the importation of data from various sources, which in the case of this study was also an advantage.

This solution allowed us to work with large amounts of data but it also allowed for a very interactive data treatment and visualization. In the case of Tableau Prep, the file with the logs was imported in a CSV format Footnote 5 and treated iteratively in a dynamic way, being refined to the desired data in a second stage. As an example, we can mention the separation of the field “time duration” in hours, minutes and seconds fields; all the IPs were converted to a generic name “student”; all the destinations visited by the students were grouped in main categories, as for instance “Facebook”, as each application had numerous distinct destinations.

About 30 changes in data treatment and in data flow “cleaning” were performed, which were, later, exported to Tableau Desktop. Each file imported to Tableau Prep, in addition to the changes applied to the previous file, was refined with more changes, in an iterative process.

After treating the data on Tableau prep the generated data flow was imported to Tableau Desktop so that dynamic data visualizations were created. At this stage, dimensions, measurements, and filters were created according to the desired data visualization. The software has the big advantage of creating dynamic visualizations of the logs’ data which allows for a different and richer perspective on the data obtained, in order to deepen further studies about the same topic.

Discussion and conclusions

This paper aimed to describe the process of a pilot to carry out a larger study where we wanted to cross-reference actual usage data (logs) of mobile applications in the classroom with data from student questionnaires. In this article we also present the main results of this pilot, both from the point of view of the process of the pilot and from the point of view of the data of use of mobile applications by students in the classroom.

From the preliminary data analysis of this pilot, we can infer that the most used apps are Facebook, Google and Instagram, as we can see in Fig.  2 and Fig.  3 , although some variations between the attendees of the courses were registered when it comes to other apps. For example, in the case of the Design course, there are alternative apps being used such as YouTube or Vimeo.

General use of applications in Scriptwriting class

General use of applications in Aesthetics classe

Another noticeable preliminary result is that students use Facebook more at the beginning of classes and Instagram is used more at the end, as we can see in Fig.  4 and Fig.  5 .

Use of Facebook per hour in Scriptwriting class

Use of Instagram per hour in Scriptwriting class

In addition, the developed model was used in the main study with a bigger convenience sampling approach, which may provide a more accurate representation of the population of mobile-phone-users in the study field.

The visualizations created in a dynamic way during this study showed that the use of logs as a complementary data provider to other instruments, such as questionnaires, can be an added value for this research field.

On the other hand, this pilot contradicts (sometimes slightly, others considerably) the results of the questionnaires answered by the students and whose logs were collected and analysed. Logs show that:

there is a common use of mobile applications during the classes;

the purpose of the access is different: participants report that they use mobile applications mostly for academic reasons, but it can be noted that there is a general use of other mobile applications such as social networks and Youtube;

the usage time is much longer than what participants reported;

the frequency is also different: students stated that they use mobile applications in classes only 1–3 days a week, but we found that, in the analysed classes, there is an almost constant use of them, and finally

students report that they do not use social networks much in class, but the use is, in fact, massive.

The students’ perception of the “use of mobile devices and applications during lessons”, and as already mentioned, during a teaching activity - 70% of the students refer using the applications between 1 to 5 times, 22% between 6 to 10 times and 4% more than 10 times. It should also be noted, as previously mentioned, that only 4% mention not using them. With regards to the use during the week, 56% of the students refer using them between 4 to 5 days per week and 39% between 1 to 3 days per week. There is also a relatively low percentage of students mentioning that they use the devices during class more than ten times (4%).

However, analysis of the logs shows that this use appears to be much more intensive. We performed a calculation based on the average number of accesses, from which we removed 40% of potential automatic accesses and divided by the average number of accesses each application had in the initial test. The results present 6.6 accesses to the device per class/student in the class with the fewest accesses, and for the highest case, 313 accesses to the device per class/student.

This result is reinforced by results from other studies, such as the Mobile Survey Report, which states that students make regular use of laptops and smartphones during lessons (Seilhamer et al., 2018 ).

These conclusions lead us to some very serious insights on this subject. Apparently, even older students have a misperception of their use of online applications during classes. There is a serious discrepancy and incongruency between the behaviours that they claim to adopt and those they actually engage in during the classes. There are authors, who argue for the need for other types of studies that support this type of approach (Gerpott & Thomas, 2014 ), because the perception reported by users may not correspond to the actual use. It means that this gap deserves a deeper reflection. Why does it happen? Are students not motivated in higher education? Is the world offered online more interesting than the one in the physical campus? We will try to answer these questions in the main study.

Availability of data and materials

Some of the visualizations created are publicly available at https://public.tableau.com/profile/davidoliveiraua

HTTPS It is a protocol used for secure communication over a computer network, and is widely used on the Internet

IP is the s a numerical label assigned to each device connected to a computer network that uses the Internet Protocol for communication

PHP is a general-purpose scripting language especially suited to web development

DNS is naming system for computers, services, or other resources connected to the Internet

Unformatted file where values are separated by commas

Abbreviations

Higher Education Institutions

Access Points

Hypertext Transfer Protocol Secure

Internet Protocol

Hypertext Preprocessor

Domain Name System

Comma-separated values

Aagaard, J. (2015). Drawn to distraction: A qualitative study of off-task use of educational technology, Computers & Education, Elsevier

Alexander, B., Ashford-Rowe, K., Barajas-Murphy, N., Dobbin, G., Knott, J., McCormack, M., Pomerantz, J., Seilhamer, R., & Weber, N. (2019). EDUCAUSE Horizon Report: 2019 Higher Education Edition. https://library.educause.edu/-/media/files/library/2019/4/2019horizonreport.pdf?la=en&hash=C8E8D444AF372E705FA1BF9D4FF0DD4CC6F0FDD1

Alves, A. G., Martins, C. A., Pinho, E. S., & Tobias, G. C. (2017). A Teoria Fundamentada em dados como ferramenta de análise em pesquisa qualitativa. 6o Congresso Ibero-Americano Em Investigação Qualitativa - CIAIQ2017, 1, 499–507

Boase, J., & Ling, R. (2013). Measuring Mobile phone use: Self-report versus log data. Journal of Computer-Mediated Communication , 18 (4), 508–519 https://doi.org/10.1111/jcc4.12021 .

Article   Google Scholar  

De Reuver, M., & Bouwman, H. (2015). Dealing with self-report bias in mobile internet acceptance and usage studies. Information Management , 52 (3), 287–294 https://doi.org/10.1016/j.im.2014.12.002 .

Dobbin, G., Dahlstrom, E., Arroway, P., & Sheehan, M. C. (2011). Mobile IT in higher education. Educause , 1–33.

Gerpott, T. J. and Thomas, S. (2014) Empirical research on mobile internet usage: A meta- analysis of the literature, telecommunications policy, 38 (3): 291-310. 2014. [online]. Available: http://www.sciencedirect.com/science/article/pii/S030859611300175Accessed on: Dec., 20, 2016.

GMI. (2019). Mobile Learning Market Size | Global Growth Statistics 2020–2026. https://www.gminsights.com/industry-analysis/mobile-learning-market

Jesse, G. R. (2015). Smartphone and app usage among college students: Using smartphones effectively for social and educational needs. Proceedings of the EDSIG Conference , 2015 , 1–13 http://proc.iscap.info/2015/pdf/3424.pdf

Long, D. R., Strauss, A., & Corbin, J. (1993). Basics of Qualitative Research: Grounded Theory Procedures and Techniques. The Modern Language Journal , 77(2), 235 https://doi.org/10.2307/328955

Magda, A. J., & Aslanian, C. B. (2018). Online college students 2018. Comprehensive Data on Demands and Preferences. https://www.learninghouse.com/knowledge-center/research-reports/ocs2018/

Monteiro, A., Bento, M., Lencastre, J., Pereira, M., Ramos, A., Osório, A. J., & Silva, B. (2017). Challenges of mobile learning – A comparative study on use of mobile devices in six European schools: Italy, Greece, Poland, Portugal, Romania and Turkey. Revista de Estudios e Investigación En Psicología y Educación , 13 , 352. https://doi.org/10.17979/reipe.2017.0.13.3229 –357.

Oliveira, D., Tavares, R., & Laranjeiro, D. (2017). Estudo de avaliação de aplicações móveis de instituições de ensino superior português. https://www.researchgate.net/publication/307981176

Parslow, R. C., Hepworth, S. J., & McKinney, P. A. (2003). Recall of past use of mobile phone handsets. Radiat. Prot. Dosim., 106(3), pp. 233–240. http://search.ebscohost.com/login.aspx?direct=true&db=cmedm&AN=14690324&site=ehost-live , DOI: https://doi.org/10.1093/oxfordjournals.rpd.a006354

Raento, M., Oulasvirta, A., & Eagle, N. (2009). Smartphones: An emerging tool for social scientists. Sociological Methods & Research , 37 (3), 426–454 https://doi.org/10.1177/0049124108330005 .

Ryu, H., & Parsons, D. (2008). Innovative Mobile learning: Techniques and technologies. Information Science Reference - Imprint of: IGI Publishing.

Seilhamer, R., Chen, B., DeNoyelles, A., Raible, J., Bauer, S., & Salter, A. (2018). 2018 Mobile Survey Report. https://digitallearning.ucf.edu/msi/research/mobile/survey2018/

Sharples, M., Corlett, D. & Westmancott, O. (2002). The design and implementation of a mobile learning platform based on android. Proceedings - 2013 International Conference on Information Science and Cloud Computing Companion, ISCC-C 2013, 345–350. https://doi.org/10.1109/ISCC-C.2013.136

UNESCO. (2013). Policy guidelines for mobile learning. In policy guidelines for mobile learning . https://doi.org/ISBN 978-92-3-001143-7. France

Vicente, F. (2013). WelcomeUA: Desenvolvimento de interface e avaliação da usabilidade. 136. https://ria.ua.pt/handle/10773/12403

Download references

Acknowledgements

Not applicable.

The authors declare that they have no funding in this project.

Author information

Authors and affiliations.

Communication and Arts Department, Universidade de Aveiro, Campus Universitário de Santiago, 3810-193, Aveiro, Portugal

David Manuel Duarte Oliveira, Luís Pedro & Carlos Santos

You can also search for this author in PubMed   Google Scholar

Contributions

DO wrote the introduction and discussion, and saw to the article structure, wrote the method section and conducted the data analysis. LP and CS conducted the literature review. All authors contributed to the discussion and conclusion sections, and the overall flow of the article. All authors read and approved the final manuscript.

Corresponding author

Correspondence to David Manuel Duarte Oliveira .

Ethics declarations

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Oliveira, D.M.D., Pedro, L. & Santos, C. The use of mobile applications in higher education classes: a comparative pilot study of the students’ perceptions and real usage. Smart Learn. Environ. 8 , 14 (2021). https://doi.org/10.1186/s40561-021-00159-6

Download citation

Received : 20 January 2021

Accepted : 16 July 2021

Published : 09 August 2021

DOI : https://doi.org/10.1186/s40561-021-00159-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Mobile application
• Mobile usage
• Higher education

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Healthc Inform Res
• v.24(3); 2018 Jul

Effectiveness of Mobile Health Application Use to Improve Health Behavior Changes: A Systematic Review of Randomized Controlled Trials

Myeunghee han.

1 School of Nursing, University of Maryland, Baltimore, MD, USA.

2 College of Nursing, Research Institute of Nursing Science, Kyungpook National University, Daegu, Korea.

The purpose of this study was to examine the effectiveness of mobile health applications in changing health-related behaviors and clinical health outcomes.

A systematic review was conducted in this study. We conducted a comprehensive bibliographic search of articles on health behavior changes related to the use of mobile health applications in peer-reviewed journals published between January 1, 2000 and May 31, 2017. We used databases including CHINAHL, Ovid-Medline, EMBASE, and PubMed. The risk of bias assessment of the retrieved articles was examined using the Scottish Intercollegiate Guidelines Network.

A total of 20 articles met the inclusion criteria. Sixteen among 20 studies reported that applications have a positive impact on the targeted health behaviors or clinical health outcomes. In addition, most of the studies, which examined the satisfaction of participants, showed health app users have a statistically significant higher satisfaction.

Conclusions

Despite the high risk of bias, such as selection, performance, and detection, this systematic review found that the use of mobile health applications has a positive impact on health-related behaviors and clinical health outcomes. Application users were more satisfied with using mobile health applications to manage their health in comparison to users of conventional care.

I. Introduction

The global mobile health (mHealth) application (app) market has been growing at a tremendous rate, and it is expected to continue to flourish [ 1 ]. These mHealth apps provide quick and easy access, transfer, and tracking of health information as well as interactive displays and interventions that can allow users to be highly engaged in promoting health outcomes and changing health-related behaviors [ 2 ]. Thus, health-related apps have a great potential to aid a wide range of target audiences with a variety of health issues [ 3 ].

Despite the evolution and widespread use of these mHealth apps, the factors involved in smartphone and health app use and their effectiveness are not yet fully understood and the field of research related to mHealth apps is still in a nascent stage [ 4 ]. Kitsiou et al. [ 5 ] mentioned that a wide range of mHealth apps have not been strictly evaluated. For this reason, most consumers use mHealth apps without any concrete information about their effectiveness or harm and evidence of the effectiveness of mHealth has been inconclusive and not fully understood [ 4 ].

Thus, research determining the effectiveness of health apps is urgently needed [ 2 ]. This study aimed to demonstrate the effectiveness of mHealth apps in changing health-related behaviors and clinical health outcomes through a systematic review of randomized controlled trials (RCTs).

II. Methods

1. search strategy.

We searched the electronic literature of RCTs published from January 1, 2000 to May 31, 2017, using four databases: the Cumulative Index to Nursing and Allied Health Literature (CINAHL), PubMed, Excerpta Medica dataBASE (EMBASE), and Ovid Medline.

A university librarian was consulted who was a subject expert in the field of teaching and learning of systematic review. Searches used the following medical subject headings terms and keywords in various combination. We derived three broad themes that were then combined using the Boolean operator ‘AND’. The first theme ‘mobile’ was created using the Boolean operator ‘OR’ to combine text words (mobile * , OR smartphone * ). The second theme ‘application’ was created using the Boolean operator ‘OR’ to combine text words (app * , OR application * ). The third theme ‘health behaviors’, was created using the Boolean operator ‘OR’ to combine text words (health behaviors * , OR health behaviors change * , OR behaviors change * ).

In this study, to secure the quality of this systematic review, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines were used [ 6 ]. This is a tool developed to verify the quality of evidence obtained during systematic reviews.

2. Study Selection

Two investigators independently reviewed the titles first and then examined the abstracts. Data extraction was conducted by one reviewer (MH Han) and was rechecked for by another reviewer to confirm the accuracy (EJ Lee). The same investigators read and screened the full texts to make the final decision. The reasons for inclusion and exclusion were recorded.

We included articles with the following characteristics: (1) published in English, (2) published during the period from 2000 to 2017, (3) results related to changes in health behaviors, (4) RCTs designed for app-based interventions to improve any health-related behaviors. The exclusion criteria were being other kinds of study than RCTs; qualitative studies; books; conference proceedings; reviews; dissertations; protocols; or studies examining text messages, Web, emails, Twitter, social network services, or personal digital assistant-based health interventions. We also excluded studies lacking behavior change indicators or outcomes, not using apps as the primary intervention tools; or focusing primarily on app design and development. Conference abstracts, protocol papers, reviews, editorials, and commentary were also excluded.

References that clearly did not meet all criteria were excluded. Full-text articles that appeared to be relevant were retrieved and independently assessed by two reviewers. Disagreements were resolved through a meeting. The initial search revealed 1,247 articles: 66 in CINAHL, 481 in Ovid-Medline, 626 in EMBASE, and 74 in PubMed. Following the PRISMA guidelines ( Figure 1 ), we removed duplicates and screened the titles and abstracts, which narrowed the list down to 57 relevant articles. Two investigators reviewed these 57 articles, 37 of which conducted other interventions with apps, compared two app-functions, examined protocols, or had unclear outcomes. Thus, finally, 20 articles were included for this review.

3. Data Collection and Analysis

From the 20 included articles, the following information was retrieved and analyzed: first author, year of publication, country, study design, themes, participants' character, sample size, mean age, intervention tool, follow-up duration, intervention characteristics, outcome measurements, as well as reported outcomes and significant levels. The search was broad with no limited target health-related behaviors in the search strategy. In this study, apps were considered effective if statistically significant results of health-related behavior changes were reported for them.

4. Risk of Bias Assessment

Risk of bias assessment for included studies was conducted by two authors using a modified version of the Scottish Intercollegiate Guideline Network (SIGN) checklist for RCT [ 7 ]. Specifically, the SIGN checklists were applied to grade the level of evidence of each study. The evaluation items are divided into 7 categories as follows: random sequence generation (selection bias), allocation concealment (selection bias), blinding of participants and personnel (performance bias), blinding of outcome assessment (detection bias), incomplete outcome data (attrition bias), selective reporting (reporting bias), and other bias. Each domain was classified as having a low, high, or unclear risk of bias. This assessment was conducted by two researchers, and disagreements were resolved by discussion.

III. Results

1. characteristics of included studies.

The RCTs included in this review were published between 2014 and 2017. Most (n = 16) were simply RCTs; the rest were an open-labelled RCT, an unmasked RCT, a cluster RCT, and a single-blinded parallel 3-arm pilot cluster RCT. The longest study duration was 8 months [ 8 ].

Five studies had a large number of participants [ 8 , 9 , 10 , 11 , 12 ]. The cluster RCT study had the largest sample size (n = 1,192) [ 11 ]. Two studies had a moderate number of participants [ 13 , 14 ]. The other 13 studies had a small number of participants. All 20 studies had between 80% and 100% retention rates; 18 (90%) studies achieved high (80–100%) retention rates in the intervention group, and only two (10%) studies [ 8 , 14 ] had a moderate retention rate.

The 20 selected studies were analyzed in this systematic review, and the following 16 themes related to health behaviors were created: physical activity (4), alcoholism (1), dietary change (1) adherence of medication or therapy (2), preparation of clinical procedure (1), PTSD management (1), weight loss (2), prenatal education and engagement (1), adherence to follow-up clinic appointments (1), improvement of CPR skill performance (2), suicide prevention (1), prevention of CHD (1), smoking cessation (1), and knowledge improvement of pap testing (1). The comprehensive characteristics of included articles are summarized in Table 1 .

RCT: randomized controlled trial, DSM-IV: Diagnostic and Statistical Manual of Mental Disorders 4th edition, PTST: posttraumatic stress disorder, SBP: systolic blood pressure, VAS: Visual Analogue Scale, NDI: Neck Disability Index, MVFS: maximal voluntary flexion strength, MVES: maximal voluntary extension strength, CPR: cardiopulmonary resuscitation, ART: antiretroviral therapy, HIV: human immunodeficiency virus, CHD: Coronary heart disease, BMI: body mass index.

2. Risk of Bias of Selected Studies

The evaluation of risk of bias for all 20 studies was conducted using the SIGN checklist for RCTs. The results were summarized using the risk of bias table of RevMan 5.3 software. A total of 15 studies properly reported random sequence generation. Only one article did not mention random sequence generation [ 15 ].

For allocation concealment, only 6 studies explicitly mentioned that allocation was concealed [ 8 , 10 , 16 , 17 , 18 ]. However, 8 studies did not discuss allocation concealment adequately. Participants were blinded in 4 studies [ 16 , 17 , 19 , 20 ]. However, due to the traits of mHealth apps, some studies could not be conducted with perfect blinding. For the remaining 16 studies, either they were not blinded or information on blinding was not clearly provided in the reporting.

Among the studies in this review, three reported the blinding of outcome assessment [ 16 , 17 , 19 ]. For reporting bias, 9 studies had a low risk of bias, and 11 were evaluated as unclear on the presence of bias. A summary of Cochrane's risk of bias table is presented in Figure 2 .

3. Content Characteristics of Apps

Some app characteristics of contents were categorized according to the behavior change technique taxonomy by Abraham and Michie [ 21 ] that is, providing information, planning (goal setting), reminding, providing feedback, or monitoring. Furthermore, additional app characteristics, such as entering data, education/training, and communication were derived from this study. Ten apps have multiple functions to manage health-related behaviors. The most common function of mHealth apps is providing the opportunity for education or training.

All participants in the control groups underwent standardized or usual care. Table 2 shows a summary of app characteristics.

IV. Discussion

These days, mHealth apps seem to be ubiquitous, and the body of research indicating their effectiveness has been growing rapidly. However, evidence for the effectiveness of mHealth apps has been uncertain, and much remains unknown in terms of health-related behavior changes and clinical results.

In this study, 20 RCTs were included to evaluate the effectiveness of mHealth apps for health-related behavior change. Seventeen studies among 20 showed a positive contribution to the enhancement of health-related behaviors. This result is similar to other previous evidence reviews [ 3 ]. Therefore, using mHealth apps could be an effective strategy to improve outcomes of users along with the high popularity of smartphone use in the everyday lives of users. However, three studies did not show positive effects of mHealth apps on health-related behavior changes. Laing et al. [ 10 ] found that there was no difference in weight loss between intervention and control groups. In addition, Nord et al. [ 11 ] reported that a DVD-based CPR education group had better performance than an app group. Tighe et al. [ 20 ] also reported that using an app had no effect on decreasing depression symptoms and impulsivity behaviors. However, it is difficult to assess the effectiveness of apps based on the results of a single study, and more studies with controlled research design are needed.

This study differs from other reviews in that we only included RCTs to examine the effectiveness of mHealth apps, because RCTs are considered the ‘gold standard’ in evaluating the effects of intervention and provide a valuable source of evidence in research and treated as a powerful experimental tool to examine the effectiveness of intervention [ 22 ]. Therefore, this systematic review, which only analyzed the results of RCTs, has provided more reliable evidence for the effectiveness of smartphone health apps.

More than half of the reviewed studies had small samples (<60). In addition, 11 reported that the duration of intervention was less than 2 months. According to Man-Son-Hing et al. [ 23 ], trials with larger samples and longer intervention durations or follow-up times are more reliable to appraise the effectiveness of intervention. Based on the results of this review, to demonstrate a certain effect using mHealth apps for health-related behavior changes, more research with long intervention durations and large samples is needed.

According to Zhao et al. [ 3 ], the retention rate is defined as the proportion of participants who remain to complete a study. The Cochrane Handbook for Systematic Reviews of Interventions reported that studies with retention rates over 80% are classified as having low attrition, and studies with retention between 60% and 79% are classified as having moderate attrition [ 24 ]. Eighteen studies achieved high (>80%) retention rates in the intervention group. In this study, over half of the studies had a moderate-high retention rate. It can be assumed that the reasons for the high retention rate were the high feasibility and acceptability of app use in users' everyday lives. Thus, mHealth apps could be effectively adopted for users to improve health-related outcomes by managing and supporting health-related behaviors of users.

In this review, some studies [ 8 , 10 , 13 , 15 , 16 , 17 , 18 , 19 , 25 , 26 ] considered multiple function apps, such as entering data and providing feedback, education, and reminders. In contrast, other studies [ 1 , 9 , 11 , 12 , 14 , 20 , 27 , 28 , 29 , 30 ] considered apps that have only one function; most of these apps had an education function. One study reported that having multiple app functions is much better to manage health status and to improve health-related behaviors [ 4 ]. However, this result might not be concrete because applying apps to the different types of situations, health behaviors, and participants can yield different results and effectiveness. For example, Laing et al. [ 10 ] conducted a study regarding an app with multiple functions for weight loss, and the effect was not significant. However, Zhang et al. [ 29 ] reported a significant result in improvement of coronary heart disease (CHD) knowledge and awareness using a single-function app. This suggests that many unconditional features of apps might not work properly, and a customized app is needed that fits the purpose and intent of the user. In this aspect, mHealth apps could provide individualized information via feedback to users and benefit them.

With respect to the risk of bias of included studies, the categories of selection bias (allocation concealment), performance bias (blinding of participants and researchers), and detection bias (blinding of outcome assessment) indicated high risk of bias. Therefore, to enhance the quality of studies and ensure low risk of bias, researchers should consider rigorous study design and reporting. Based on the results of this review, further studies using meta-analysis are needed to identify the effects of mHealth apps with specific outcomes. In addition, the effectiveness of the apps should be verified with the effects or risks being used with verification.

In this review, there were several limitations. First, we used broad key words, such as ‘health’, ‘behaviors’, ‘smartphone’, and ‘mobile’. For this reason, many articles related to the use of mHealth apps on specific diseases or health conditions—for instance, diabetes, hypertension, or asthma—might not have been included in this study. Second, in this review, only RCTs were included to analyze the effectiveness of mHealth apps. However, some of the RCTs did not fully follow the form of the RCT or applied a modified form of RCT. However, well-structured RCT is needed to verify mHealth app effectiveness. Third, almost all of the studies considered in this review were conducted in developed countries. Hence, it is difficult to generalize our results to developing countries. Fourth, we did not conduct a meta-analysis because of interventions with different kinds of mHealth apps. However, with the results of meta-analysis, the effectiveness of mHealth apps can be verified more clearly.

This systematic review was conducted to examine the effectiveness of mHealth apps to lead to changes in their targeted health-related behavior. This study summarized the characteristics and changes in targeted health outcomes. To our knowledge, there has been no previous systematic review of RCTs for identifying the effectiveness of mHealth apps in improving health-related behaviors. Similar to previous studies, this systematic review also found that the use of mHealth apps has a positive impact on health-related behaviors, such as physical activity, diet change, adherence to medication or therapy, and knowledge enhancement related to clinical procedures. Moreover, most apps seem to promote better clinical health outcomes. Most app users are satisfied with the use of mHealth apps to manage their health in comparison to users of conventional care. Although most studies analyzed indicated statistically significant effects to improve health, more RCTs with larger samples and longer applied interventions are still needed to confirm the effectiveness of mHealth apps. To assess the efficacy of mHealth apps in greater detail, further research is needed.

Conflict of Interest: No potential conflict of interest relevant to this article was reported.

The making of accessible Android applications: an empirical study on the state of the practice

• Open access
• Published: 06 August 2022
• Volume 27 , article number  145 , ( 2022 )

Cite this article

You have full access to this open access article

• Marianna Di Gregorio 1 ,
• Dario Di Nucci 1 ,
• Fabio Palomba   ORCID: orcid.org/0000-0001-9337-5116 1 &
• Giuliana Vitiello 1  

4911 Accesses

3 Citations

58 Altmetric

Explore all metrics

Nowadays, mobile applications represent the principal means to enable human interaction. Being so pervasive, these applications should be made usable for all users: accessibility collects the guidelines that developers should follow to include features allowing users with disabilities (e.g., visual impairments) to better interact with an application. While research in this field is gaining interest, there is still a notable lack of knowledge on how developers practically deal with the problem: (i) whether they are aware and take accessibility guidelines into account when developing apps, (ii) which guidelines are harder for them to implement, and (iii) which tools they use to be supported in this task. To bridge the gap of knowledge on the state of the practice concerning the accessibility of mobile applications, we adopt a mixed-method research approach with a twofold goal. We aim to (i) verify how accessibility guidelines are implemented in mobile applications through a coding strategy and (ii) survey mobile developers on the issues and challenges of dealing with accessibility in practice. The key results of the study show that most accessibility guidelines are ignored when developing mobile apps. This behavior is mainly due to the lack of developers’ awareness of accessibility concerns and the lack of tools to support them during the development.

Similar content being viewed by others

Accessibility Assessment in Mobile Applications for Android

Mobile Usability: Review, Classifications and Future Directions

Mobile User Experience (UX) Design Guidelines Considering the Need for Accessibility

Avoid common mistakes on your manuscript.

1 Introduction

Mobile applications, a.k.a. apps, are nowadays used by billion users for any social and emergency connectivity (Wasserman 2010 ). The trend is tremendously and continuously increasing these days: the rise of social distancing has indeed changed the way people communicate and interact with each other (Martin et al. 2016 ; Statista 2020 ). In such a context, mobile apps represent one of the primary means of allowing human interaction. Therefore, an ever-increasing population of users needs to interact with the functionalities they implement. This aspect does not only represent a challenge for researchers in the field of computer-human interaction (CHI) but also for software maintenance and evolution research, which is called to devise novel instruments to support developers when evolving successful mobile apps that all types of users can use (Martin et al. 2016 ; Yan and Ramachandran 2019 ).

Applications that are not accessible or are only partially accessible are an obstacle for both individuals and businesses (Ballantyne et al. 2018 ; Yan and Ramachandran 2019 ). For a single user, a hard-to-use app will either be a source of stress and frustration or be entirely sidelined in favor of a more accessible alternative (Sevilla et al. 2007 ). For a business, the fewer users can use their mobile application, the lower the translated revenue stream will be (Wentz et al. 2017 ). The pervasiveness of mobile applications has led researchers to reason more and more in terms of accessibility . This trend is giving rise to a research field that aims at developing mobile apps usable by those affected by disabilities (e.g., visual impairments) (Leporini et al. 2012 ; Vitiello et al. 2018 ), which represent over one billion (around 15%) of the world’s population. Ensuring the accessibility of the app functionalities has become more crucial than ever (Yan and Ramachandran 2019 ) when people affected by disabilities are more dependent on their mobile devices.

The two main operating systems for tablets and smartphones, i.e., iOS and Android , are equipped with pre-installed accessible functions, including screen reading functionalities as in the case of TalkBack for Android . The unique needs of individuals with disabilities and their right to participate in the digital age cannot be ignored by developers. However, differently from iOS, accessibility work in Android apps is very limited (Martin et al. 2016 ; Yan and Ramachandran 2019 ) and, as such, it is unclear to what extent developers implement universal design principles or use accessibility features in their mobile applications.

So far, most of the research on accessibility has focused on the web and mainly provided guidelines and instruments that developers can employ to implement accessible websites (e.g., Flatla 2011 ; Friedman and Bryen 2007 ; Sevilla et al. 2007 ). On the contrary, the accessibility of mobile applications has not been examined so thoroughly (Yan and Ramachandran 2019 ) and, as a matter of fact, it still represents an open research challenge to face.

In the recent past, empirical investigations have been conducted to study how developers discuss the matter on StackOverflow (Vendome et al. 2019 ) and how existing accessibility features support users with disabilities (Kocieliński and Brzostek-Pawłowska 2013 ; Walker et al. 2017 ). Nevertheless, there is still a notable lack of knowledge on the way developers approach the problem of accessibility and whether they implement the available guidelines to develop accessible applications. An improved understanding of these aspects is crucial to guide future software maintenance and evolution research efforts toward the definition of design, evolutionary, and testing techniques that can better support practitioners while developing mobile accessible applications.

In our previous registered report (Di Gregorio et al. 2020 ), Footnote 1 we designed an exploratory empirical investigation into the making of mobile apps from the perspective of accessibility to bridge the current gap of knowledge concerning the relation between mobile apps and accessibility. We focused on Android for a two-fold reason. On the one hand, it has been the subject of previous accessibility studies. On the other hand, although a vast number of apps is developed worldwide on this platform, still little is known on how to best engineer the problem in Android devices—as opposite to iOS and Apple , which provide an integrated set of devices and features to handle accessibility (Darvishy 2014 ). More particularly, we discussed our plan toward this goal by defining two research questions to understand (i) whether and to what extent the available accessibility guidelines are implemented in Android applications and (ii) the developer’s opinions about the matter. We sought to elicit the state of the practice and the key issues and challenges faced by developers when dealing with accessibility.

In this paper, we follow up on the registered report and present the results of our study. The study has adopted a two-step methodology. We first conducted manual coding activities to quantify how existing accessibility guidelines are implemented in the context of 50 top-rated Android applications. Then, we conducted a survey study with 70 mobile developers and ten semi-structured interviews to gather insights into the issues and challenges of developing accessible apps and understand the extent to which developers are implementing accessibility support in Android apps.

The key results yielded by our study are that only a subset of the available guidelines is typically implemented in Android apps, and these mainly relate to aspects like color contrast and interactive content. While surveying developers, we could recognize a general lack of awareness of accessibility concerns; furthermore, developers indicated the lack of (semi-)automated support to control accessibility while developing mobile applications.

The findings of the paper allow us to provide the research community with a set of open issues and challenges that represent the next research avenues that should be addressed to provide developers with usable accessibility tools.

To sum up, our study provides the following contributions:

An empirical study reporting the accessibility guidelines that are and are not implemented in Android applications, which can be used by researchers and tool vendors as a basis for either prioritizing accessibility concerns within techniques/tools or conducting further analyses into the specific reasons why certain guidelines are more/less applied in practice;

Insights into the developer’s perception that researchers can use to understand the underlying motivations leading practitioners not to apply accessibility guidelines as well as by tool vendors to tune current tools based on the opinions that developers have of specific accessibility concerns;

A list of current issues and challenges that developers face when dealing with accessibility in practice that can be useful for researchers to motivate and conduct additional studies into the matter;

A publicly available replication package (Di Gregorio et al. 2021 ) containing the data to address the goals of our study—data are anonymized whenever needed. The package includes a novel dataset reporting the accessibility guidelines implemented in a set of 50 Android apps— researchers can use it as ground truth to evaluate novel accessibility tools.

Structure of the Paper

Section  2 discusses the background on accessibility guidelines and overviews the related literature. In Section  3 we describe the research questions and methodology employed to address our study, while Section  4 reports the achieved results. In Section  5 , we summarize the main findings, discuss the limitations of the study, and outline the key implications that our work has for the research community. Section  6 overviews and discusses how we mitigated possible threats to validity. Finally, Section  7 concludes the paper and presents our future research agenda on the matter.

2 Background and Related Work

In this section, we define accessibility, provide an overview of the currently available guidelines to create accessible apps, and discuss the related literature, comparing it with the methodology employed in our study.

2.1 Accessibility: Definition and Guidelines Overview

According to Iwarsson and Ståhl ( 2003 ), accessibility is defined as “the quality of being easily reached, entered or used by people with disabilities” . Mobile accessibility refers to making websites and apps more accessible to people with disabilities when using smartphones and other mobile devices (W3C 2020 ).

While the definition explicitly targets people suffering from disabilities, it is worth mentioning that accessibility is a desirable property for other groups of people as well, since these could also benefit from the availability of accessibility functionalities (Lawrence and Giles 2000 ). As such, we could generalize the concept of accessibility and state that it is the practice of making websites and apps usable by as many people as possible (Lawrence and Giles 2000 ). Indeed, there are two key elements driving accessibility: (1) The attention to the problems of accessing websites and apps for people with disabilities; and (2) The attention to guaranteeing universality of access, that is, not to exclude anyone: not only people with disabilities in the strict sense but, for example, also those suffering from temporary disabilities, those with obsolete equipment or slow connections.

Various mobile accessibility standards have been proposed, including those defined by W3C (W3C 2020 ) and by the UK BBC Standards and Guidelines for Mobile Accessibility (BBC 2021 ). Within these standards, several recommendations have been formulated to provide better support for people with different types of disabilities, including motor, hearing, and vision problems. Several companies have also created their list of developer guidelines based on standards such as the Android Accessibility Developer Guidelines (Android 2021 ), Apple’s Accessibility Developer Guidelines (Apple 2021 ) and the IBM accessibility checklist (IBM 2020 ). In our work, we focused on accessibility issues in Android apps and considered the recommendations provided by Android, W3C, and BCC.

Standard controls, objects, and elements should be used to ensure a higher level of accessibility as custom controls tend not to implement accessibility as fully as standard platform controls. When standards and guidelines are implemented using non-standard techniques, there is a risk that users who depend on platform-specific accessibility features such as accessibility settings and speech output are excluded from accessing the content.

Progressive enhancement is recommended to ensure that users with accessibility settings or assistive technology enabled using older phones and platforms can access the content. This mechanism ensures that the content and features are accessible even if the experience is slightly altered. All content must be accessible and navigable using the platform navigation paradigm for assistive technology. For example, the directional controller must be supported on Android to allow TalkBack screen reader users to review and navigate the page content. Android requires that all elements be accessible from the keyboard to be accessed with a D-pad or trackball. In this respect, Android 4.0 has reduced this requirement somewhat by including an “Explore by touch” method.

When applications or sites block, disable, or interfere with platform-specific accessibility features or technology, users with disabilities may not be able to use the site or the app. Potential problems include zoom suppression, garbled screen content, or the inability to run assistive technology. This behavior could occur when the technology directly controls audio, video, or CPU resources preventing assistive technology from accessing these resources promptly.

Some users with disabilities may require more accessibility features because they may have more disabilities. For example, a user may be deaf and blind or may have poor eyesight and may be unable to use a pointing device or touchscreen. More modes of operation should be supported that allow users to access content according to their preferences. On Android, for example, built-in keyboard support should not prevent other standard touch events.

2.2 Accessibility: State of the Art

The topic of accessibility is rapidly gaining interest in software engineering and closely related research communities, like computer-human interaction and computer-supported cooperative work. These multi-disciplinary research opportunities allow for treating the problem from different perspectives. However, although accessibility has long been investigated in the context of web applications (Harper and Chen 2012 ; Sierkowski 2002 ; Sloan et al. 2006 ), and many books are now available to drive developers toward the construction of accessible websites (Harper and Yesilada 2008 ; Paciello 2000 ; Rutter et al. 2007 ), the definition of accessibility principles and guidelines for mobile applications can be still considered a work in progress.

From an empirical standpoint, Kocieliński and Brzostek-Pawłowska ( 2013 ) investigated the currently available accessibility features with virtual QWERTY keyboards on mobile devices, comparing them against the use of an integrated Braille notetaker. The controlled study involved visually impaired users who were called to complete tasks using the two treatments. The main findings reported were that the existing mobile support is not sufficient to assist visually impaired users. Indeed, the integration of a Braille notetaker led to better results in terms of task completion time. Zhong et al. ( 2015 ) augmented the touch feature of Android devices in order to facilitate less precision to a targeted location and assist with gestures that might not be suitable for individuals with tremors, e.g., double-tapping the same location. Mehta et al. ( 2016 ) conducted an empirical study—involving 12 blind users—to study the accessibility of date-pickers and proposed to augment the current capabilities of mobile devices with additional features able to support blind users. On a similar note, Xie et al. ( 2015 ) focused on the understanding and improvement of the support provided for GUIs responsiveness when connecting smartphones to external displays, while Milne et al. ( 2014 ) studied the accessibility of mobile health sensors for blind users: interestingly, they found that most of the accessibility problems identified could be solved with a small amount of effort. Conversely, another interesting and little investigated problem faced by Ichioka et al. ( 2020 ) is the percentage of malware in apps that use accessibility services that is constantly increasing. Therefore, in the future it is necessary to investigate the identification of specific countermeasures for malware using accessibility services.

Case studies have also been performed to study specific types of disabilities. Serra et al. ( 2015 ) assessed four Brazilian government applications against the W3C guidelines, discovering that most of them were not applied. In this scenario, Quispe et al. ( 2020 ) investigated the prioritization of mobile accessibility guidelines extracted from e-MAG (the Brazilian Government Accessibility Model) to help dealing with limited resources while also addressing accessibility. Walker et al. ( 2017 ) evaluated several weather apps and their usability/accessibility for blind and sighted users: as a result, they discovered that most of the considered apps were not designed to be universally accessible. Al-Subaihin et al. ( 2013 ) reported that, if appropriately used, structural HTML elements can make the functionality of TalkBack and VoiceOver similar in mobile web apps and native applications. Also, Krainz et al. ( 2018 ) proposed a change in the mobile app development to support accessibility, concluding that a model-driven approach with automated code generation might potentially avoid many of the accessibility problems experienced by visually impaired users. Eler et al. ( 2019 ) analyzed comments made at the Google Play Store and FDroid, aiming to identify whether users comment about accessibility problems. When they evaluated the ratings of the apps, they noticed that users generally do not mention accessibility issues in their reviews.

Some more recent studies focused on defining specific instruments and methods to support users with special needs. For instance, Araújo et al. ( 2017 ) defined a manual test that can assess whether mobile audio games meet the need of visually impaired users. Similarly, Díaz-Bossini et al. ( 2014 ) and Díaz-Bossini and Moreno ( 2014 ) proposed guidelines to make mobile applications closer to the needs and requirements of older users. Park et al. ( 2014 ) and Ross et al. ( 2018 ) analyzed the image-based button labeling problem by focusing on missing and alternative text labels, respectively. Subsequently, again Ross et al. ( 2020 ) performed a large-scale analysis of free Android apps, exploring the frequency of accessibility barriers and the factors that may have contributed to barrier prevalence. They tested a population of 9,999 apps but limited themselves to just testing seven accessibility barriers. Our study does not analyze a large-scale app population but we tested all accessibility guidelines identified.

When turning the attention to the software engineering research community, Armaly and McMillan ( 2016 ), Armaly et al. ( 2017 , 2018 ) conducted several studies aiming at comparing program comprehension tactics applied by blind and sighted programmers. Their key findings reported that, despite having different reading processes, both tend to prioritize the understanding of method signatures; furthermore, audio highlight facilities might provide additional support to blind programmers when skimming code on the web. McMillan and Rodda-Tyler ( 2016 ) also reported on a didactic software engineering course setting that allows blind and sighted programmers to collaborate more effectively and improve their capabilities to share programming expertise and knowledge.

The above-mentioned papers are different from the study proposed herein. Most of them focused on accessibility from the perspective of specific users and aim at characterizing the limitations of currently available guidelines compared to the needs and requirements of such users. On the contrary, our focus is on developers and how they act when it turns to keep accessibility into account, highlighting the challenges they face when implementing accessibility guidelines and the additional instruments that they would need to build more accessible mobile applications. The preliminary results of de Almeida and Gama ( 2021 ) show, in fact, that developers have a worse perception than interface designers on this topic.

Vendome et al. ( 2019 ) first performed a mining study showing the limited usage of accessibility APIs in a set of 13,817 apps. Then, they focused on the developer’s perspective by mining StackOverflow posts related to accessibility. From this analysis, they identified the aspects that developers mainly implement in their apps and those requiring more effort. Compared to that study, ours can be seen as complementary, mainly because we adopted a mixed-research method that allowed us to gain more precise information on the extent to which accessibility features are applied during development.

The closest work is that of Alshayban et al. ( 2020 ). The authors have conducted an empirical study aimed at understanding the accessibility of the Android apps. They reported a large-scale analysis, however, analysing only 11 accessibility guidelines. The authors also presented the results of a survey to detect current practices and challenges in Android apps with regard to accessibility. Compared to this study, on the one hand, we conducted manual analyses to test a large number of accessibility guidelines and verify which of them are implemented in Android applications. In this way, we can be more precise in indicating the specific guidelines that developers tend to care about more when including accessibility concepts. Furthermore, such an analysis leads to additional insights into whether and how developers implement critical guidelines that are relevant for users (e.g., the MUST ones). On the other hand, we directly inquired developers having two key advantages. First, we could ask more complete and specific questions on their perceptions and opinions of accessibility in practice, rather than letting them emerge from the analysis of posts. Second, we could involve a broader population of developers rather than focusing on those subscribed to StackOverflow —which might provide a limited view on the matter.

2.3 Accessibility: State of Practice

When considering the tools to assess accessibility of Android apps in practice, there exist three officially supported instruments, namely Accessibility Scanner (AS), Footnote 2 Lint , Footnote 3 and Node Tree Debugging (NTD). Footnote 4

The former takes a snapshot of an application as input. It scans each GUI component to identify accessibility issues related to content labels, touch target size, clickable items, and text/image contrast. The tool is based on dynamic analysis; therefore, it requires the app under investigation to be installed on a device. Lint is instead based on static analysis and runs as part of the Android SDK, even though it is also integrated within the Android Studio IDE. It has a broader scope than AS , as it reports micro-optimization opportunities to security, performance, and other non-functional aspects of source code. Lint also operates in terms of accessibility/usability and can detect problems related to missing content descriptions and accessibility labels. Finally, NTD is a testing tool for Android apps that can be employed to test for accessibility concerns. In particular, the tool describes how an AccessibilityService in the app interprets the GUI components and provides information as well as improvement recommendation related to focusable elements and their assistive descriptions.

It is also worth mentioning the existence of unofficial tools that are not integrated within the Android SDK but can provide developers with additional insights into mobile app accessibility. One of the most popular tools in this category is Enhanced UI Automator Viewer (Patil et al. 2016 ): it extends the standard UI Automator in order to verify unlabeled UI elements and color contrast.

Our work has an empirical connotation and, therefore, does not aim to improve the capabilities of the above-mentioned tools directly. Nevertheless, we provide pieces of information concerning accessibility guidelines and developers’ takes that are actionable for both tool vendors and researchers. The former can exploit them to tune the available tools, while the latter can devise novel approaches that better assist practitioners. We elaborate on these points when distilling the concrete implications of our work in Section  5 .

3 Research Methodology

The goal of our empirical study is to understand the state of the practice of accessibility in mobile applications, with the purpose of providing an overview of how mobile developers currently deal with this problem as well as the issues and challenges they face when implementing accessibility guidelines. The perspective is of both researchers and tool vendors. The former are interested in gathering insights into the current state of the practice on accessibility to devise novel possible instruments to support mobile developers when dealing with accessibility in practice. The latter are interested in tuning and providing new features that might further assist developers in assessing and improving accessibility aspects in mobile apps.

3.1 Research Questions

We structure our investigation around two main research questions ( RQ s). In the first place, we seek to understand how the existing accessibility guidelines are implemented in mobile applications, namely the extent to which developers adopt these guidelines when developing their apps. This leads to our first RQ :

Once established how the accessibility guidelines are implemented, we proceed with a finer-grained understanding of developers’ perspective regarding the problem, particularly collecting their opinions on (i) the issues and challenges of implementing accessible applications and (ii) the tools currently supporting them. An improved understanding of those aspects would allow the research community to understand the developer’s needs to support further. Hence, we pose our second RQ :

To address our RQ s, we conducted mixed-method research (Creswel 2009 ), combining manual coding analyses with surveys and semi-structured interviews with developers (Rossi et al. 2013 ). It is important to note that the empirical study has an exploratory connotation and, as such, it must be seen as an investigation whose outcome produces a number of implications that further research can exploit to generate research hypotheses.

3.2 Material and Objects

The objects of the empirical study are represented by (i) mobile applications and (ii) accessibility guidelines.

As for the former, we focus on the 50 top-rated Android apps coming from the AndroidTimeMachine dataset (Geiger et al. 2018 ), which collects a reliable set of real open-source Android apps. We focus on these apps for two main reasons. On the one hand, we seek to analyze popular apps used by thousands, if not millions, users worldwide: this allows us to verify the behavior of developers who should be more sensitive to accessibility issues given the number of users they can potentially attract. On the other hand, we have to limit the number of applications to consider because of the time- and effort-intensive manual activities that we need to perform to address our research questions (further details in this respect are discussed later in Section  3.4 ).

As for the latter, Table  1 reports the entire set of accessibility guideline categories currently available for the design of Android applications. Each category groups a set of guidelines to account for when considering a specific aspect of the mobile application (e.g., ‘Audio and Video’ ).

3.3 Subjects

The subjects of the study are developers of Android applications. We have involved both original and external developers of the applications that are the objects of the study. While the former can provide us with feedback on implementing the accessibility guidelines in their applications and their view of the problem, surveying a larger population of developers may provide additional insights into the issues and challenges of dealing with accessibility in practice. We collected participants’ background and demographic information to understand the representativeness of our results. We followed the sampling strategies defined in literature (Topp et al. 2004 ) to define a sample that meets our goals. More details on the recruitment strategies applied in our empirical study are reported in the next section.

3.4 Execution of the Empirical Study

In this section, we report the methodological details that describe the execution of the empirical study—we discuss the two RQ s independently.

RQ 1 . Accessibility Guidelines in Practice

To address RQ 1 , we manually tested the considered applications to verify the implementation of accessibility guidelines—this strategy allows us to interact with an app and its accessibility services directly, much like a user would normally do. Overall, the guidelines to be verified were 54, divided into the 11 categories presented in Table  2 . To perform such a manual test, we adopted a closed-coding strategy (Vaughn and Turner 2016 ): this is a systematic methodology that, in our case, involves the analysis of all the graphical user interfaces of an application and the subsequent labeling of the guidelines implemented as functionalities of the app, starting from a pre-established coding scheme, which is our case is represented by the set of guidelines available for Android applications.

More specifically, we have created a data extraction form, implemented using an Excel sheet, to facilitate the verification of the guidelines. For each of them, the form contains four pieces of information: (i) the name of the guideline to verify, (ii) the procedure to follow to discover whether the guideline is implemented, e.g., activate the notifications to verify that they are both visible and audible, (iii) the excepted visual/audio effect to observe in case the guideline is implemented, and (iv) the outcome to add once evaluating the guideline. The extraction process of an app was conducted by the first author of this paper and consisted of the following steps:

The author downloaded the app from the Google Play Store on a Huawei Y5 smartphone.

The author selected the next guideline to test and the corresponding instructions from the data extraction form.

Depending on the selected guideline, she has activated the accessibility function required to verify it if needed. Otherwise, she has gone straight to the next validation step.

The author has been exercising the app to identify the feature connected to the accessibility guideline, if available. For instance, this concerns the identification of the app’s media in case the guideline refers to ‘Audio and Video’ accessibility aspects. If identified, the author proceeded with the next step; otherwise, she went back to Step 2 and continued with another guideline.

Once the element is identified, the author determined if the guideline is implemented in the app. If so, she annotated the data extraction form by putting, in the row corresponding to the considered guideline, a ‘true’ in the fourth column. Otherwise, she annotated the column with ‘false’ .

Using the above-described methodology, we have collected 50 Excel sheets, one for each application considered. These were later analyzed to address the first research question.

RQ 2 —Surveying Mobile Developers

To address RQ 2 , we conduct a survey study aiming at gathering insights regarding accessibility concerns from a broad audience of Android developers. The survey is composed of three main sections—we report the full list of questions in Table  3 . The first one presents a total of nine questions about accessibility and how developers consider it in practice. We ask questions on the relevance of the problem, i.e., how important is accessibility for the participants, what reasons would make them willing to implement accessibility features in their applications, and whether they are aware of the existence of guidelines to make an app accessible. Afterward, we continue with questions more related to the implementation of accessibility guidelines. In particular, how often developers implement them in their applications, how difficult they are to apply, and why. Finally, we ask participants to report up to five challenges they usually face when dealing with accessibility concerns and report whether and which tools they currently use when performing the task.

In the second part of the survey, we allow participants to provide us with additional insights and feedback. They can leave their e-mail address if they are interested in receiving a summary of our findings and can express their consent to a follow-up interview to discuss the problem of accessibility in practice further. sFinally, the third section of the survey concerns background information that we collect to understand better our sample of developers and possibly analyze the generalizability of our results.

The survey is designed to last 15/20 min and is created using a Google survey module. Before releasing the survey on a large scale, we ran a pilot with two developers of our contact network to evaluate if the survey is short and understandable enough to reduce the risk of having a low response rate and be appropriately filled out. Based on the pilot results, we have changed the text of some questions, add/remove some of them, or change the response type to make the questionnaire easier to understand or quicker to be compiled.

To gather insights from the original developers, we extract the e-mail addresses from the Github repositories of the considered applications. Then, we invite developers to fill the survey out, first asking whether they would like to participate. In other words, we recruit only volunteers to avoid privacy issues or other developer concerns. In addition, to gather insights from external mobile developers, we advertise the online survey using the personal social network accounts of the authors (i.e., Facebook , Twitter , and LinkedIn ). It is worth remarking that we were aware that the reliance on social media might negatively impact selecting a valid sample. Therefore, we integrated social media with other sources to ensure the quality/completeness of the information gathered when addressing RQ 2 , still relying on a large sample of developers for our study. On the one hand, we involved additional developers from our private contacts (e.g., former University students or other practitioners who are currently mobile developers). On the other hand, we advertised the survey on a specialized practitioners’ blog such as Reddit lto acquire information from developers who have a solid knowledge of programming (Vassallo et al. 2020 ). In particular, Reddit contains more than 100 different subreddits dedicated to Android development that we exploit to potentially reach thousands of Android developers. We track the source used by participants to access the survey to better comment on the validity of the sample. To further stimulate the participation, we allow participants to indicate a non-profit organization of their choice to which we would donate 2 USD for the research against COVID19.

The answers are anonymized to preserve the privacy of participants. As a result of this study, we have a clearer view of the relevance of accessibility in practice and the major challenges developers face when dealing with the problem. Based on the answers received to question #13, we also planned follow-up semi-structured interviews with Android developers. Their main goal is to clarify ambiguous or contrasting answers received during the survey and to have a better picture of the current practices, issues, and challenges experienced by developers when dealing with accessibility in Android environments. From a practical perspective, we summarize the survey results to the interviewees and ask them to comment on the answers from which we could not derive a definitive outcome. The semi-structured interviews are conducted through Skype, last 30/40 min, and are transcribed for further analysis.

3.5 Data Analysis

Once we gathered data from the closed-coding exercise and the survey study, we proceeded with their analysis.

As for RQ 1 , we first provided descriptive statistics on the extent to which accessibility guidelines are implemented in the sample of Android applications. We computed minimum, mean, median, standard deviation, and maximum number of accessibility guidelines implemented in the considered apps. Secondly, we provided a finer-grained overview of each specific category of guidelines. We discussed (i) to what extent each of them is present in the sample by reporting descriptive statistics, i.e., minimum, mean, median, standard deviation, and maximum number accessibility guidelines for each category, and (ii) the relative and absolute frequency of implementation of the guidelines included in each category. Then, we focused on the guideline requirements, i.e., ‘MUST’ , ‘MUST NOT’ , ‘SHOULD’ , and ‘SHOULD NOT’ : in this case, we aimed at understanding whether developers take them into account, e.g., if the ‘MUST’ guidelines are implemented in the considered apps. Finally, we verified the relation between the guidelines and the type of application considered. We grouped the apps by category, as provided by the Google Play Store , and we computed descriptive statistics to grasp if some categories are more prone to accessibility concerns.

As for RQ 2 , we first described the background of survey participants by discussing the answers they provide in Section III of the survey. This detail allowed us to understand the sample of developers and reason about the generalizability of our findings. In the second place, we distinguished the analysis procedures to use when considering closed and open questions. The former was analyzed employing statistics: we plotted and discussed the distribution of answers provided by participants through the Likert scale evaluations. The latter was subject of an iterative content analysis : in particular, we conducted the following methodological steps:

The first author of the paper went through the content of the participant’s answers and the possible semi-structured interviews. She split sentences using standard text separators (e.g., commas) and assigned initial labels to each sentence: these labels represent the main concepts discussed by participants. Then, the three authors not involved so far validated the initial labels assigned and provided feedback on how to improve them, for instance, by proposing to aggregate two semantically similar labels. When this step was accomplished, we computed a measure of agreement between the labels assigned by the first author and those recommended by the other three.

The first author used the suggestions and feedback received in the first step to conduct a second iteration over the labels assigned. This step resulted in a set of themes deemed important by participants when addressing each survey question.

All the authors were involved in reaching a final agreement concerning the names and meanings of each label. This step led to a theoretical saturation , i.e., the point in which no further labels are required because the existing ones already correctly represent the concepts expressed by the study participants.

The themes coming from this data analysis procedure concern each specific open question posed in the survey. We discussed each theme and provided qualitative insights by presenting the most significant answers for a specific theme. In addition, when analyzed the answers to questions #8, #9, and #10 of the survey, we also provided statistical data reflecting the number of times a specific issue/challenge/tool named by the participants, hence providing a kind of prioritization of the concerns and tools that developers have concerning the problem of accessibility.

3.6 Verifiability and Replicability

The data generated from our study are made persistently publicly available through Figshare (Di Gregorio et al. 2021 ). In particular, we release raw data about the accessibility guidelines implemented in our dataset, the survey structure, the anonymized responses, and all scripts used for data analysis.

4 Analysis of the Results

This section presents the results of the empirical study, which we discuss by addressing the two research questions independently.

4.1 RQ 1 . Accessibility Guidelines in Practice

In the context of RQ 1 , an iterative manual verification was performed to evaluate which accessibility guidelines were implemented within the mobile applications that are the subject of the study. As explained in Section  3.4 , these were evaluated for their general applicability verifying whether each guideline was implemented or not in the application.

According to the results obtained, we observe that no application implemented all the guidelines. This result was somehow expected, other than reasonable, since the accessibility guidelines do not represent fixed rules. Their applications must therefore be considered based on the specific application domain and context. Nonetheless, we noticed that most of the guidelines were applied at least once in our dataset: as such, we can report that the mobile developers of the considered apps sometimes take care of them.

Looking deeper into the considered apps, we observed that 94% of the guidelines (51/54) could be assessed, i.e., the apps contained features that might have enhanced through the implementation of accessibility mechanisms. Other guidelines were instead not applicable. For instance, this is the case of the ‘Metadata’ guideline, which cannot be currently applicable in Android . Indeed, it does not support a mechanism for navigating between containers within native applications. A user can only navigate through a single item at a time. As a consequence, the ‘Containers and landmarks’ guideline is also not applicable. Finally, Android does not provide tooltips or additional hint text other than aria:contentDescription . Therefore, the ‘Tooltips and supplementary information’ guideline is not applicable—however, users can still obtain tooltips by long-pressing on icons in the Action Bar.

Based on the considerations above, we considered the number of guidelines that could be assessed while measuring the total amount of guidelines implemented within the considered applications. For instance, let consider the Budget application. In this case, 30 guidelines were assessable and, among these, ten were violated (i.e., 1/3 of them).

Figure  1 shows the percentage of guidelines implemented by the developers of the 50 apps considered. In particular, the x-axis represents apps (i.e., each bar is an app) while the y-axis reports their accessibility coverage level (i.e., the percentage of guidelines implemented). From the figure, we could immediately understand that the number of accessibility guidelines implemented in the considered apps was typically low, with a minimum of 24% and an average of 41%. In the best case, 63% of the guidelines were implemented. Consequently, we could first conclude that, overall, mobile developers tend not to implement accessibility guidelines while developing and maintaining their apps, even though they might have the chance to do that.

Guidelines coverage of the 50 apps. A coverage (y-axis) equal to 1 means that all guidelines were implemented in the analyzed apps

Figure  2 provides a more detailed overview of which are the individual accessibility guideline categories implemented in our dataset.

Percentage of guidelines assessable against percentage of guidelines actually implemented in the considered mobile applications, grouped by category

From the figure, we could observe that some guideline categories seem to be more considered by developers. For instance, the ‘Images’ category showed the highest ratio between the number of accessibility guidelines implemented and those actually assessable (37/41). This category refers to evocative visual content that allows the user to interpret the meaning of the features implemented in the applications. The highest implementation ratio is somehow reasonable and expected since the use of images to reflect the content of a piece of text is something that human beings typically do to convey meaningful messages (Turkle 2011 ). As such, independently from the knowledge that developers might have of the specific guidelines ruling the usage of images, we could have expected to observe the category to be highly implemented.

The category having the highest ratio of guidelines violated was ‘Audio and Video’ (only 44% of the guidelines were implemented over the total assessable). This indicates that mobile developers do not often take care of the characteristics that the interactive content should have in terms of font size, style/position of controls, and so on. In this case, it is likely that developers are not keen nor aware of the need to put themselves in the others shoes and offer functionalities that facilitate users to interact with the app.

As for the other guideline categories, we could delineate a general trend from the analysis of Fig.  2 . A notable percentage of guidelines were violated: while we could not speculate on the reasons making them less implemented at this stage, we sought to understand this aspect further in the context of our second research question.

When lowering the granularity of our investigation to the individual guidelines within each category, we could first observe that the highest amount of guidelines implemented pertained to ‘Design’ , with an average of 28.8 out of a maximum of 50, i.e., around 29 applications contained implementation of accessibility guidelines related to the design of the application. More specifically, the ‘Style and readability’ design guideline, with a value of 47, appeared to be the most implemented, followed by the ‘Spacing’ guideline with a value of 45. On the contrary, the least evaluable category was ‘Audio and Video’ , which was also the least implemented with an average of 3.6. In this case, the ‘Volume control’ guideline was implemented only two times out of 50 applications.

When it turns to the guidelines requirements, i.e., ‘MUST’ , ‘MUST NOT’ , ‘SHOULD’ and ‘SHOULD NOT’ , Fig.  3 shows the distribution of guidelines applied grouped by their associated requirement—we visualize the distribution in ascending order based on the number of guidelines abide by. Such an assessment was intended to understand whether developers consider the guideline requirement while deciding which accessibility guidelines to apply. However, as depicted in the figure, we could not find any relation between those requirements and the application of the guidelines, meaning that developers do not likely consider whether a certain guideline must/should or not be applied. This aspect is further considered in our RQ 2 , where we inquired mobile developers on their expertise on accessibility concerns.

Distribution of accessibility guidelines implemented, grouped by requirements

To conclude the discussion, from the first research question we could observe that developers of the top mobile applications considered often tend not to implement accessibility guidelines. This result is not connected to whether the guideline must/should (not) be applied. In our discussion, we also identified possible reasons behind the way developers operate in terms of accessibility. The next research question aims to elicit directly from the developer’s experience the main problems and challenges they face when dealing with the problem of accessibility of mobile applications., with the ultimate goal of deriving concrete actionable items and take-away messages that researchers and practitioners might consider to further investigate and address the problem of accessibility in mobile applications.

4.2 RQ 2 . The Developer’s Take on Accessibility

While the previous research question allowed us to understand, from a quantitative perspective, how accessibility guidelines are implemented in Android applications, we could only delineate some conjectures on the reasons why developers decide to apply or not specific guidelines. The survey study conducted in RQ 2 aimed at shedding light on the developer’s perspective of the accessibility problem. In particular, our survey was answered by a total of 75 developers, of which 63 male, four female, one transgender person, and seven who preferred not to declare it. Given the nature of the dissemination mechanisms, we cannot estimate the response rate—we are not aware of how many potential developers were reached over the various social networks and blogs considered. Nonetheless, we can report that 65% of the participants had access to the survey via personal contact, 11% via Telegram , 9% via Reddit , 8% via Facebook 5% via Twitter and 1% via Tandem .

4.2.1 Developers’ Background

Figure  4 shows the background of our participants. Among the 75 respondents, 60% (45 participants) declared to have a high level of experience in programming, and 43% (32 participants) had high experience in Android programming. About 42% of the participants (mainly) work as developers and 24% (18) work in medium-sized companies with more than 100 employees. From these descriptive statistics, we can say that our sample consisted of various developers with sufficient programming experience and whose opinions may provide us with valid and reliable insights on how they deal with the accessibility problem. Furthermore, 15% of participants work in a large team of 10 to 200 people, 24% within a team of 5–10 people, while the majority (43%) in a small team (i.e., 2–5 people).

Background of our participants

4.2.2 Relevance of the Problem

In the first part of the questionnaire, we aimed at understanding how developers consider accessibility in practice when developing mobile apps.

Figure  5 shows how many participants responded with values between the minimum and maximum to the first question of the survey. As shown, most of the participants (60%) considered the problem of accessibility as very important for mobile app development. At the same time, only three developers (4% of our sample) claimed that this is negligible. Hence, as expected, we can confirm that accessibility is a significant concern for most of the developers involved in the survey. As an example, one of the participants commented:

#26 - All users need to have the same possibilities. Fig. 5 Relevance of the accessibility problem from the developer’s perspective Full size image

The high perceived relevance of the problem allows us to claim that accessibility is definitively something that researchers should further explore with the aim of providing automated support or even empirical studies that may increase the developer’s awareness about the problem.

4.2.3 Reasons for Implementing (or Not) Accessibility Guidelines

Personal ethics (39%) and the widening of the potential user base (37%) were mentioned as the main reasons making developers willing (or unwilling) to implement the accessibility guidelines, as depicted in Fig.  6 . A smaller percentage of participants (17%) declared that their applications are dedicated to people with disabilities and, therefore, they have to follow accessibility guidelines. Only 7% of the developers reported that their companies implement policies aimed at ensuring mobile accessibility. Looking at these results, we can observe that the main driver for the implementation of accessibility guidelines is the personal willingness of developers to provide additional functionalities that would enable the usage of the app to a wider variety of users. By matching these observations with the poor implementation of accessibility guidelines discovered in RQ 1 , our findings seem to suggest that more work should be done on motivating developers and stimulating their willingness to apply accessibility guidelines while developing their apps. This result is confirmed by the analysis of questions #4 and #5 of our survey (Fig.  7 ): although most of the participants have a medium-high knowledge of accessibility guidelines, a large majority of participants apply them only in a few cases.

Results for Question n.3—what makes you willing (or unwilling) to implement the accessibility guidelines

Results for Questions n.4 and n.5 —awareness and implementation of accessibility guidelines

While the results of our analysis mainly report on the need for making developers aware of the relevance of accessibility, they might be also read under an orthogonal point of view. According to the opinions collected, the developers who typically apply accessibility guidelines do that because of personal motivations, which are by nature connected to their degree of sensibility. Hence, besides raising awareness of the problem, developers might benefit from additional instruments such as, for instance, improved advertisement strategies on the relevance of accessibility in practice and how a lack of it might impact the life of people with disabilities. In this respect, we could envision a multidisciplinary effort conducted by multiple stakeholders.

4.2.4 The Opinion on the Individual Categories

Once investigated the general behavior of developers, our survey aimed at seeking their opinions and experience with the implementation of the individual categories of guidelines. Figure  8 reports the results obtained in this respect. It is worth remarking that our survey allowed participants to express additional opinions on the factors making hard the implementation of specific guidelines. While the comments left were helpful in most cases, others were not clear enough to elicit those factors. In these cases, we took advantage of the follow-up semi-structured interviews conducted with the participants to discuss them further. In particular, eight developers left their email addresses in response to question #13 of our survey and were later interviewed. For the sake of readability and conciseness, we discuss the results by guideline, reporting data from the survey and accompanying the discussion with the insights coming from the semi-structured interviews whenever needed.

In the first place, our analysis revealed that most developers encounter little or any difficulty in implementing the guidelines. As a matter of fact, for the ‘Audio and Video’ , ‘Forms’ , and ‘Text equivalent’ guidelines, no developer found it very difficult to implement the category of guidelines. This result was quite surprising and, at the same time, interesting: while the involved developers consider the vast majority of accessibility guidelines as easy (or fairly easy) to implement, they are still reluctant to implement them—hence, confirming that the problem is strictly connected to the willingness or, perhaps, a limited understanding of how significant might be implementing those guidelines for users with disabilities. However, there are some exceptions.

Diving deeper into the individual guideline categories identified by at least one developer as hard to implement, three participants declared the ‘Design’ category to be a difficult one. From the survey analysis, we found that developers mostly focus (or need to focus) on the aesthetics of the application rather than on its accessibility. As such, they would need more precise guidelines for implementing the design principles that address accessibility concerns. This consideration is also common to other developers, who commented, for instance, saying that:

#23 - Standards are not defined precisely.

The semi-structured interviews confirmed that the guideline definitions are sometimes vague and not easily interpretable, potentially complicating their implementation. For this reason, Interviewee #3 claimed that an improved accessibility guideline should provide informal definitions and concrete examples on how to integrate them within various types of applications. This tooling would help developers to learn by examples, simplifying and speeding up the implementation process.

82% of the participants considered the implementation of the guideline to be not very difficult or not difficult at all. The remaining 14 developers, instead, rated this guideline as complex or very complex to apply. By looking at the open comments left by those participants, we could understand that the guideline is not complex. However, the time required for implementing it is too high and/or there is a lack of resources available. Two developers commented, indeed, that:

#57 - A lot of businesses just don’t have enough resources to comply with all such consistency across all clients. #63 - It is very tedious and takes a significant amount of time to label interactive elements and images.

The vast majority of the participants did not consider this guideline hard to implement. Only one participant justified the complexity of the implementation by saying that:

#17 - I saw many examples of buggy focusability in android development and sometimes providing for example good keyboard navigation on the screen is really really hard due to these bugs.

In other words, the developer suggested that the Android APIs to use for implementing this guideline may sometimes be defective, increasing the time and effort required to ensure the focusability of the app. Once again, this seems not to be strictly connected to the guideline itself but to the surrounding environment required to implement accessibility guidelines.

13% of the participants reported that ensuring the accessibility of images can be hard or very hard. By looking at the open answers provided, we could understand that this is mainly due to the role played by images in the graphical user interface of mobile applications. One of the developers commented as follows:

#56 - Difficult as the images play an important graphic role.

Unfortunately, the comment could not provide us with a clear understanding of the key issues connected to the implementation of the guideline. For this reason, we have further elaborated the question in our follow-up interviews with the developers. From the discussion, it turned out that this is due to the lack of proper usability skills, which might lead to complex implementations of this guideline. Indeed, optimizing the use of images while keeping accessibility under control is not easy, as implementing the guidelines risk affecting the overall aesthetics and look-and-feel of the app, potentially creating more issues than benefits. This result suggests that the definition of recommendation approaches that may suggest how to best implement the GUI of mobile apps by balancing usability and accessibility might be a nice addition for mobile developers.

This category presents a very similar situation as for the previous guideline, with a lower percentage (40%) of users who did not rate the implementation difficult. Nonetheless, we were surprised to see some open comments like the one shown below:

#72 - Didn’t even know.

By discussing this further in the semi-structured interviews, we understood that some developers were not even aware of the existence of this guideline. Interviewee #6 explained that most of the developers with whom s/he worked were not only unaware of accessibility guidelines but also unable to find helpful information on the web. As a result, the implementation difficulty is sometimes due to the retrieval of appropriate information on usability and accessibility, making it hard for developers to correct the problem in their apps.

Notifications

As shown in Fig.  8 , 5% of the respondents to our survey (4) declared that implementing notification-related accessibility guidelines is hard in practice. In this case, the comments left in the open answers were already clear enough to elicit the main reasons behind this result. One of the participants stated:

#70 - Notifications in Android are often very finicky and device-dependent so we can’t expect them to conform reliably to specific behaviors.

As reported, the mechanisms enabling notifications in Android are not always easy to use. This challenge may be related to the different notification types that can be visualized differently, and developers should apply the guidelines to the specific implementations. Among all the guidelines discussed so far, this seems to be the most source code-related one: in this sense, the definition of smart mechanisms may potentially address the problems raised by the participants, e.g., dynamic wizards helping developers to select the most appropriate set of notifications along with the accessibility rules to implement.

Results Questions n.7 and n.7.1—difficulty in implementing the specific guidelines

Scripts and Dynamic Content

As for this category, 14% of developers rated its implementation as fairly or very difficult. As stated by one of the developers, the implementation of this guideline:

#64 - Would harm the simple user interface, adding effort and making the development of simple apps unfeasible.

As reported, some developers would not find enough benefits from the implementation of this guideline, as it may have possible negative effects on the aesthetics and usability of the app. While discussing this issue further in the semi-structured interviews, Interviewee #4 commented on the statement above by discussing the exemplary case of Progressive Functionality . This fine-grained guideline is related to creating graphical user interfaces that allow users to do actions in a stepwise, progressive manner. S/he reported that:

Interviewee #4— The fundamental problem with progressive content is that it takes what was previously one requirement (do x when user enters the screen) and turns it into three or four requirements (do x or do y or do z based on condition A B or C). These types of multi-requirements are actually quite difficult to communicate about (conversations will be full of confusion and miscommunication). Naturally they increase the workload but if the requirements are clear it is actually not all that much work. The problem is getting the requirements clear in the first place. Additionally, it can be very difficult for the quality team to actually exercise all of these different pathways, so it increases the work there too. Most developers are going to advise against these type of progressive interfaces and instead promote that you create one interface that works for all situations, perhaps with the ability to have some content hidden by default.

In other words, the implementation of scripts and dynamic content enforces developers to increase the number of app requirements, requiring them to create more test cases or develop more code review activities. In addition, the definition of the requirements might be a source of miscommunication that possibly leads to the introduction of undesired defects. Based on our results and discussion, implementing this guideline seems to be related to multiple aspects and levels of expertise covering the entire software life cycle. As such, developers might be more reluctant to consider its actual application.

The last guideline we discuss is concerned with the structured content. More particularly, one participant reported that:

#74 - Sometimes it is difficult to maintain the original structure and refactoring is required.

This answer was later discussed in the semi-structured interviews. The main point here is concerned with the moment in which accessibility is considered. If an application is not designed to be accessible in the first place, refactoring for accessibility can be effort-prone and costly since it may imply the re-design of entire screens of the app. In addition, Interviewee #7 pointed out the lack of automated mechanisms and integrated tools that can provide accessibility feedback directly within the IDE. In her opinion, the availability of these tools might help to address the problem of accessibility from the start, hence avoiding costly refactoring that is rarely implemented.

4.2.5 The Challenges of Accessibility

The last part of our survey was reserved for the issues and challenges of accessibility in practice. Table  4 reports the top-5 list of challenges identified when analyzing the participants’ answers.

As shown in the table, two of these challenges are related to accessibility awareness. In the first place, developers expressed their inability to understand the exact needs of users with disabilities: this represented the main, most popular challenge mentioned in our survey. Participants also reported that one of the challenges concerns the involvement of those users during the development: this is made complicated by identifying the right target audience and the mechanisms to use for involving them. For example, developers mentioned the complexity of requirement elicitation, which naturally leads to ineffective solutions. Our participants (and our interviewees) suggested using a user-centered methodology to develop mobile apps, where real users are surveyed and involved throughout the application development process.

At the same time, our participants mentioned the awareness of companies and customers. When discussing this further, the developers told us that only a small percentage of users need accessibility in mobile applications and, therefore, companies tend to underestimate the problem. In addition, accessibility is often considered non-portable and essential only for large apps. Perhaps more importantly, all interviewees raised another social issue connected to accessibility: they argued that their customers sometimes dictate not to follow accessibility guidelines to get the product up and running in the least amount of time. Consequently, they found it difficult to convince the customers of the additional time required to implement a universally usable product.

Additional challenges are more on the technical side. On the one hand, standardizing accessibility guidelines is related to defining techniques that help developers implement the guidelines while the app is still under development: accessibility should be considered a first-class citizen. On the other hand, developers need mechanisms that allow for a trade-off between the aesthetics of the graphical user interface and its accessibility.

As further elaborated in Section  5 , the challenges identified impact the mobile application development from requirement elicitation to low-level design and implementation, other than letting emerge important socio-technical implications of accessibility. In the first place, there exist communication barriers that prevent developers to engage with disabled users. The redesign of current requirement elicitation strategies seems therefore to be the next reasonable step to pursue. The definition of new communication channels that might allow users with disabilities to advertise their needs, the creation of accessibility interest groups, or even the definition of regulations and policies that rule the certification of mobile apps with respect to accessibility requirements would be the next big challenges for practitioners, researchers, and decision makers. At the same time, our results seem to suggest the need for novel continuous validation and verification mechanisms that would reduce the development effort when dealing with accessibility. In this sense, the definition of user-centered methodologies that may put users with disabilities in the loop would provide additional opportunities for developers to get in contact with minorities and account for their opinions and constant support when evolving mobile apps.

4.2.6 The Current Accessibility Support

As a final step of the survey, we asked participants if they use tools to verify the implementation of the accessibility guidelines. 77% of the respondents do not use tools. The remaining 13% reported the usage of the Accessibility Scanner app, the Google Play pre-launch report, and the definition of beta tests with users. Our results clearly show that it is not very common for developers to rely on tools to verify the accessibility of the apps being developed. Three of the respondents reported that the missing usage of tools is because they provide minimal information while lacking a more detailed and careful analysis of both the different categories of disabilities that should be considered and the specific guidelines that should be implemented. In addition, the interviewees highlighted that different types of devices must be taken into consideration; often, different brands/models of devices behave differently. Therefore, the implementation of some accessibility UIs requires complex logic to include all target devices. Last but not least, some brands’ battery-saving policies may affect or even suspend accessibility services.

The results coming from this analysis point out the need for further automated support from the software engineering community. We believe that our findings might be especially interesting for the software testing perspective: The developer’s answers indeed revolve around the verification of how accessibility guidelines are implemented, other than the compatibility concerns that arise when the accessibility guidelines are implemented on multiple devices.

5 Discussion and Implications

The results achieved when addressing our research questions provided several insights that need to be further discussed and implications for both researchers and tool vendors, which we elaborate on in the following.

Conclusion 1—Accessibility Problems are Widespread in Apps

One of the main results coming from our analyses refers to the poor adoption of accessibility guidelines in practice. We discovered that, in each app, most of the guidelines were not considered by developers in the implementation phase or incorrectly applied. Furthermore, our results show that ‘Design’ , ‘Script and Dynamic Content’ , and ‘Text Equivalents’ are the most problematic categories of guidelines to implement. The feedback received by developers through surveys and follow-up semi-structured interviews allowed us to elicit the main reasons behind such difficulties. We conclude that several technical aspects connected to the implementation of accessibility guidelines should deserve further attention in the future. On the one hand, more research is needed around this subject. We hope that the reported results might serve as a basis for stimulating software engineering researchers to proactively consider novel mechanisms to support mobile developers. A critical challenge here concerns the definition of (semi-)automated techniques that might support developers while evaluating the level of accessibility of their applications and developing accessible mobile apps. On the other hand, the lack of standardized methods to apply accessibility guidelines represents a challenge for the designers of these guidelines. While there exist catalogs that suggest the general universal design principles to apply, the results of our study pointed out the lack of practical recommendations and patterns to follow during the development. This latter challenge is, in particular, one of the main points of our future research agenda: the definition of accessibility design patterns will be the next challenge to face.

Conclusion 2—Lack of Developer’S Awareness of Different Types of Disabilities

Not only the problem of accessibility is widespread in practice, but also developers generally lack knowledge of the different disabilities of users and how they should be considered from a software engineering perspective. This clear issue represents a call to action for researchers working in multiple fields, from medical branches to software engineering and human-computer interaction. Indeed, the results of our study revealed the need for multidisciplinary research that can formulate novel instruments and methods to increase the sensibility of developers around the matter. This finding also highlighted the lack of software engineering education and training on accessibility and the need to review the existing guidelines of software engineering college curricula to focus more broadly on accessibility as a quality attribute and be considered throughout the software development lifecycle. Accessibility is typically mainly taught in Human-Computer Interaction courses within computer science education. Our findings may instead stimulate discussions on how to improve study plans in order to incorporate accessibility differently, for instance by adding new courses or remodularizing existing ones. Along this line, Waller et al. ( 2009 ) have recently proposed an educational approach where accessibility is not treated as a separate topic but rather as an integral part of software design and development. We can envision even further adjustments like the definition of accessibility requirements in greenfield software engineering projects developed by students at both Bachelor and Master levels, so that students can engage with accessibility within the more complex design of software projects and possibly take decisions driven by the accessibility requirements fixed. In a similar way, we can envision the integration of accessibility modules within courses of Mobile and Web Development or Basic and Advanced Programming. As pointed out by Jia et al. ( 2021 ), these requirements could instill in students greater awareness of accessibility in programming topics without affecting the learning objectives of basic computer science. Last but not least, educators should consider the various forms of disability when training the next generation of mobile developers, trying to convey and teach the principles of Universal Design. For example, students could analyze and discuss case studies to learn methods to design, implement, and test accessible systems to assemble fully inclusive systems, even when the target audience is not restricted to disabled users.

Conclusion 3—Software Engineering Meets Human-Computer Interaction

As pointed out by several developers involved in RQ 2 , the accessibility perspective of the mobile engineering process is all but defined. Developers argued the difficulties they experience with accessibility requirement elicitation and management, other than the challenges concerning design, refactoring, and testing for accessibility. Besides the actions that developers might take individually, e.g., the user-centered usability testing processes mentioned above, our findings are more general and recall the need of software engineering methods for accessibility . This aspect is one of the main implications of our work. We argue the definition of symbiotic methods that would allow human-computer interaction and pure software engineering to more closely collaborate to define unified processes that may enable improved engineering of mobile applications that take accessibility and usability into account. The rise of a tighter collaboration between the two research communities would enable the definition of combined methods that optimize the quality of mobile apps simultaneously with usability, possibly leading to the production of better software.

More specifically, we believe that our findings have implications for both software engineering methodology and practice. In the former case, we argue the definition of a brand new branch of software engineering specifically focused on human-computer interaction methods for accessibility (and usability in general). This encompasses the entire software engineering life-cycle. We first envision novel methods to manage the management and development complexity of accessibility guidelines. On the one hand, this recalls the need for understanding the orthogonal expertise that mobile app developers require to properly deal with the matter or even how developers can interact with designers or UX experts. Researchers working at the intersection between project management and social debt might be interested in assessing how such mixture of expertise can be managed or lead to sub-optimal communication/collaboration practices, other than exploring the many ways these practices may impact the development and commercial success of mobile apps. The results reported in our paper are also of the interest of researchers and agencies working on the definition of standards: the guidelines available are indeed not yet transferred into suitable standards nor manuals that can be practically used by developers. In this respect, our findings open the way to other developer-centered investigations into how accessibility standards can be devised and integrated within the developer’s workflow.

From a technical side, software maintenance, evolution, and testing researchers are critically affected by our findings. Among the various points raised by developers in the context of RQ 2 , we first identified traces of a new type of technical debt (Kruchten et al. 2012 ) connected to the management of accessibility concerns. In particular, one approach to mitigate accessibility issues is to plan for accessibility early in the design phase rather than managing it as an afterthought at the end of the development phase. In other words, our results allow us to define an accessibility technical debt as the accumulated long-term cost caused by choosing an early, sub-optimal user interface or design solution. So far, this unknown debt has been neglected by the research community. We argue that additional analyses and researches would be needed and desirable to devise new accessibility technical debt detectors and refactoring recommenders. These tools could work at different granularities (e.g., in the IDE rather than during code review) and stages (e.g., designing early prototypes, such as UI sketches, refactoring existing functionalities). In a similar fashion, we could envision novel catalogs of accessibility design patterns , that may support the practical implementation of novel standards. Last but not least, the lack of tooling reported by the involved developers multiple times in our analysis clearly open the door for verification and validation researchers interested in defining instruments to check and verify the source code for the presence of accessibility concerns.

Conclusion 4—On the Generalizability of our Findings

The generalizability of the findings is a crucial aspect for any empirical investigation. In the context of our investigation, there are some observations to make in this respect. In the first place, when considering the accessibility guidelines implemented in practice, we could focus only the limited set composed of 50 top-rated mobile applications belonging to the AndroidTimeMachine dataset (Geiger et al. 2018 ) because of the intensive manual work required to address RQ 1 . We found that, on average, only 41% of the guidelines are actually applied in those apps. While the reader must deem these results valid within the specific context analyzed in our work and look for larger-scale replications of the study, the rationale behind the poor application of accessibility guidelines let us believe that the main findings of the analysis might be observed in other apps as well. The lack of awareness and tooling to deal with accessibility, along with the other reasons identified, may indeed limit the overall applicability of the guidelines, independently from the size or popularity of the apps. In this sense, we expect to discover similar findings when considering a larger sample of apps. Additionally, it is also worth remarking that our study targeted top-rated apps, which are supposed to take care of accessibility concerns with the aim of enlarging their user base. It is likely to believe that lower-rated apps follow a similar behavior to gather more and more users. In terms of generalizability of the findings reported in RQ 2 , the motivations provided by the sample of 75 developers look reasonable enough to believe they can be considered valid by other developers as well. At the same time, inquiring a different sample of developers might have let to the analysis of additional perspectives, thus potentially leading to the identification of more challenges/barriers to accessibility, other than less obvious/popular motivations not to implement guidelines. In this respect, the reader might only consider the set of observations and conclusions provided as partial. Replications of the survey study would indeed provide additional insights into the problem and, subsequently, the conclusions drawn on the status of accessibility.

6 Threats to Validity

Several threats might have affected the validity of our results and the conclusions drawn. This section discusses how we mitigated them.

Threats to Construct Validity

Possible issues in this category refer to the methods used to set up the empirical study. The first point of discussion concerns the dataset of mobile apps exploited in the study. We focused on 50 top-rated Android apps coming from the AndroidTimeMachine dataset (Geiger et al. 2018 ): the decision was made to collect and analyze data of real open-source Android apps that are used by thousands of users worldwide.

Another possible concern is connected to how we elicit the set of issues and challenges developers face when dealing with accessibility. We opted for a survey-based investigation through which participants could share their past/current experiences with the matter. Of course, those participants performed the task in a remote setting. While we could not completely avoid the lack of conscientious responses, the follow-up data analysis allowed us to verify the meaningfulness of the answers. It could have possibly detected data to be removed for the sake of reliability. In addition, we performed semi-structured interviews to complement the survey study and discuss questions for which we obtained contrasting outcomes.

Threats to Conclusion Validity

In the context of RQ 1 , we conducted an iterative manual analysis to verify the presence of accessibility guidelines. Similarly, in RQ 2 , we conducted an additional coding procedure to analyze the developer’s open answers. In both processes, the first author of the paper was the main responsible. Nevertheless, to double-check her actions and mitigate possible misinterpretation, the other authors have constantly been involved and took action whenever needed. This continuous collaboration and the level of agreement reached make us confident of the results reported in the study. Although we cannot claim the statistical relevance of our sample across the entire Android ecosystem, the analysis depicts the status of accessibility in the best-rated open-source Android apps. We are, however, aware of the need of further replications or even complementary studies that might corroborate the conclusions drawn by employing our methodology.

Threats to External Validity

Threats in this category refer to the generalizability of the conclusions drawn from our study. As already discussed in Section  5 , our findings should mainly be deemed valid with respect to the sample of apps and developers considered. More specifically, we targeted 50 top-rated open-source mobile applications and involved 75 survey respondents. As for the former, the considered apps belonged to different domains and had various characteristics that enabled us to investigate accessibility under different perspectives. As for the latter, the participants had previous experience and expertise with Android development, hence being able to provide us with meaningful insights into the problem of accessibility. While some of the results identified in the context of our analysis look reasonable enough to be potentially considered applicable to other apps and developers, we cannot claim the generalizability of our results to mobile applications having a different connotation, e.g., closed-source or industrially developed apps. As such, further larger-scale replications of our study in different contexts would be desirable, other than already part of our future research agenda.

7 Conclusion

The growing popularity of mobile devices, coupled with constant technological improvements in the field, has led to an increasing number of mobile applications. In this context, usability aspects play a pivotal role both when considering the design and implementation phases. Although usability is already recognized as a crucial aspect of mobile development, only a few studies analyzed the accessibility of mobile applications. In this research, we aimed at advancing the state of the art by analyzing (1) the extent to which a set of known accessibility guidelines are applied in practice and (2) the developer’s take on the accessibility problem.

We conducted a quantitative investigation of 50 Android applications finding that most of the guidelines available are not implemented within applications. Afterward, we interviewed 75 developers, conducting eight semi-structured interviews, showing that accessibility is perceived necessary, but several socio-technical barriers often prevent developers from applying the accessibility guidelines. The overall output of our research identified several challenges that must not only be considered by the software engineering research community but also by experts of other disciplines like human-computer interaction, medicine, and others.

The identified challenges represent the main input for our future research on the subject. We aim to further explore accessibility on a more extensive set of systems, possibly considering how the same application on different operating systems can generate a different level of accessibility. Perhaps more importantly, we will seek to elicit a set of accessibility design patterns that would enable developers to more practically deal with the accessibility guidelines and define novel automated instruments to facilitate the adoption of accessibility guidelines. Finally, we plan to enlarge the scope of our analyses to understand the app customer’s perspective, namely how the accessibility of mobile applications should be improved from the point of view of the users with disabilities.

The registered report was accepted at the 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME) and uploaded to the Open Science Framework (OSF); the report is available at https://osf.io/3yghp/ .

https://support.google.com/accessibility/android/answer/6376570

https://developer.android.com/studio/write/lint

https://developer.android.com/guide/topics/ui/accessibility

https://developer.android.com/guide/topics/ui/accessibility/index.html

https://www.w3.org/TR/mobile-accessibility-mapping/

https://www.bbc.co.uk/guidelines/futuremedia/accessibility/mobile

Al-Subaihin A A, Al-Khalifa A S, Al-Khalifa H S (2013) Accessibility of mobile web apps by screen readers of touch-based mobile phones. In: International conference on mobile web and information systems. Springer, pp 35–43

Alshayban A, Ahmed I, Malek S (2020) Accessibility issues in android apps: state of affairs, sentiments, and ways forward. In: 2020 IEEE/ACM 42nd international conference on software engineering (ICSE). IEEE, pp 1323–1334

Android (2021) Accessibility Developer Guidelines. https://developer.android.com/guide/topics/ui/accessibility

Apple (2021) Accessibility Developer Guideline. https://developer.apple.com/accessibility/

Araújo M C, Façanha A R, Darin T G, Sánchez J, Andrade R M, Viana W (2017) Mobile audio games accessibility evaluation for users who are blind. In: International conference on universal access in human-computer interaction. Springer, pp 242–259

Armaly A, McMillan C (2016) An empirical study of blindness and program comprehension. In: Proceedings of the 38th international conference on software engineering companion, pp 683–685

Armaly A, Rodeghero P, McMillan C (2017) A comparison of program comprehension strategies by blind and sighted programmers. IEEE Trans Softw Eng 44(8):712–724

Article   Google Scholar  

Armaly A, Rodeghero P, McMillan C (2018) Audiohighlight: code skimming for blind programmers. In: 2018 IEEE international conference on software maintenance and evolution (ICSME). IEEE, pp 206–216

Ballantyne M, Jha A, Jacobsen A, Hawker J S, El-Glaly Y N (2018) Study of accessibility guidelines of mobile applications. In: Proceedings of the 17th international conference on mobile and ubiquitous multimedia, pp 305–315

BBC (2021) Mobile Accessibility Guidelines. https://www.bbc.co.uk/accessibility/forproducts/guides/mobile

Creswel J W (2009) Research design: qualitative, quantitative, and mixed methods approaches University of Nebraska–Lincoln

Darvishy A (2014) Accessibility of mobile platforms. In: International conference of design, user experience, and usability. Springer, pp 133–140

de Almeida V L, Gama K (2021) Mobile accessibility guidelines adoption under the perspective of developers and designers. In: 2021 IEEE/ACM 13th international workshop on cooperative and human aspects of software engineering (CHASE). IEEE, pp 127–128

Di Gregorio M, Di Nucci D, Palomba F, Vitiello G (2020) The making of accessible android applications: an empirical study on the state of the practice. In: 2020 IEEE international conference on software maintenance and evolution (ICSME), pp 857–861

Di Gregorio M, Di Nucci D, Palomba F, Vitiello G (2021) The making of accessible android applications: an empirical study on the state of the practice—replication package https://tinyurl.com/accessible-mobile-apps

Díaz-Bossini J-M, Moreno L (2014) Accessibility to mobile interfaces for older people. Procedia Comput Sci 27:57–66

Díaz-Bossini J-M, Moreno L, Martínez P (2014) Towards mobile accessibility for older people: a user centered evaluation. In: International conference on universal access in human-computer interaction. Springer, pp 58–68

Eler M M, Orlandin L, Oliveira A D A (2019) Do android app users care about accessibility? An analysis of user reviews on the google play store. In: Proceedings of the 18th Brazilian symposium on human factors in computing systems, pp 1–11

Flatla D R (2011) Accessibility for individuals with color vision deficiency. In: Proceedings of the 24th annual ACM symposium adjunct on User interface software and technology, pp 31–34

Friedman M G, Bryen D N (2007) Web accessibility design recommendations for people with cognitive disabilities. Technol Disabil 19(4):205–212

Geiger F -X, Malavolta I, Pascarella L, Palomba F, Di Nucci D, Bacchelli A (2018) A graph-based dataset of commit history of real-world android apps. In: Proceedings of the 15th international conference on mining software repositories, pp 30–33

Harper S, Chen A Q (2012) Web accessibility guidelines. World Wide Web 15(1):61–88

Harper S, Yesilada Y (2008) Web accessibility: a foundation for research. Springer

IBM (2020) IBM Accessibility Requirements. https://www.ibm.com/able/guidelines/ci162/accessibility_checklist.html

Ichioka S, Pouget E, Mimura T, Nakajima J, Yamauchi T (2020) Accessibility service utilization rates in android applications shared on twitter. In: International conference on information security applications. Springer, pp 101–111

Iwarsson S, Ståhl A (2003) Accessibility, usability and universal design–positioning and definition of concepts describing person-environment relationships. Disabil Rehabil 25(2):57–66. The information about affiliations in this record was updated in December 2015. The record was previously connected to the following departments: Division of Occupational Therapy (Closed 2012) (013025000), Transport and Roads (011034010)

Google Scholar  

Jia L, Elglaly Y N, Baker C M, Shinohara K (2021) Infusing accessibility into programming courses. In: Extended abstracts of the 2021 CHI conference on human factors in computing systems, pp 1–6

Kocieliński D, Brzostek-Pawłowska J (2013) Improving the accessibility of touchscreen-based mobile devices: integrating android-based devices and braille notetakers. In: 2013 Federated conference on computer science and information systems. IEEE, pp 655–658

Krainz E, Miesenberger K, Feiner J (2018) Can we improve app accessibility with advanced development methods?. In: International conference on computers helping people with special needs. Springer, pp 64–70

Kruchten P, Nord R L, Ozkaya I (2012) Technical debt: from metaphor to theory and practice. IEEE Softw 29(6):18–21

Lawrence S, Giles C L (2000) Accessibility of information on the web. Intelligence 11(1):32–39

Leporini B, Buzzi M C, Buzzi M (2012) Interacting with mobile devices via voiceover: usability and accessibility issues. In: Proceedings of the 24th Australian computer-human interaction conference, pp 339–348

Martin W, Sarro F, Jia Y, Zhang Y, Harman M (2016) A survey of app store analysis for software engineering. IEEE Trans Softw Eng 43 (9):817–847

McMillan C, Rodda-Tyler A (2016) Collaborative software engineering education between college seniors and blind high school students. In: 2016 IEEE/ACM 38th international conference on software engineering companion (ICSE-c). IEEE, pp 360–363

Mehta Y, Joshi A, Joshi M, Jadhav C (2016) Accessibility of date picker for touchscreens. In: Proceedings of the 8th Indian conference on human computer interaction, pp 64–69

Milne L R, Bennett C L, Ladner R E (2014) The accessibility of mobile health sensors for blind users. In: International technology and persons with disabilities conference scientific/research proceedings (CSUN 2014), pp 166–175

Paciello M (2000) Web accessibility for people with disabilities. CRC Press

Park K, Goh T, So H -J (2014) Toward accessible mobile application design: developing mobile application accessibility guidelines for people with visual impairment. In: Proceedings of HCI Korea, HCIK ’15. Seoul, Hanbit Media, Inc, pp 31–38

Patil N, Bhole D, Shete P (2016) Enhanced ui automator viewer with improved android accessibility evaluation features. In: 2016 International conference on automatic control and dynamic optimization techniques (ICACDOT). IEEE, pp 977–983

Quispe F E, Scatalon L P, Eler M M (2020) Prioritization of mobile accessibility guidelines for visual impaired users. In: ICEIS (2), pp 563–570

Ross A S, Zhang X, Fogarty J, Wobbrock JO (2018) Examining image-based button labeling for accessibility in android apps through large-scale analysis. Association for Computing Machinery, New York, pp 119–130

Ross A S, Zhang X, Fogarty J, Wobbrock J O (2020) An epidemiology-inspired large-scale analysis of android app accessibility. ACM Trans Access Comput(TACCESS) 13(1):1–36

Rossi P H, Wright J D, Anderson A B (2013) Handbook of survey research. Academic Press

Rutter R, Lauke P H, Waddell C, Thatcher J, Henry S L, Lawson B, Kirkpatrick A, Heilmann C, Burks M R, Regan B et al (2007) Web accessibility: Web standards and regulatory compliance. Apress

Serra L C, Carvalho L P, Ferreira L P, Vaz J B S, Freire A P (2015) Accessibility evaluation of e-government mobile applications in Brazil. Procedia Comput Sci 67:348–357

Sevilla J, Herrera G, Martínez B, Alcantud F (2007) Web accessibility for individuals with cognitive deficits: a comparative study between an existing commercial web and its cognitively accessible equivalent. ACM Trans Comput-Hum Interact (TOCHI) 14(3):12–es

Sierkowski B (2002) Achieving web accessibility. In: Proceedings of the 30th annual ACM SIGUCCS conference on user services, pp 288–291

Sloan D, Heath A, Hamilton F, Kelly B, Petrie H, Phipps L (2006) Contextual web accessibility-maximizing the benefit of accessibility guidelines. In: Proceedings of the 2006 international cross-disciplinary workshop on Web accessibility (W4A): building the mobile web: rediscovering accessibility? pp 121–131

Statista (2020) Number of smartphone users worldwide https://www.statista.com/statistics/330695/number-of-smartphone-users-worldwide/

Topp L, Barker B, Degenhardt L (2004) The external validity of results derived from ecstasy users recruited using purposive sampling strategies. Drug Alcohol Depend 73(1):33–40

Turkle S (2011) Evocative objects: things we think with. MIT Press

Vassallo C, Panichella S, Palomba F, Proksch S, Gall H C, Zaidman A (2020) How developers engage with static analysis tools in different contexts. Empir Softw Eng 25(2):1419–1457

Vaughn P, Turner C (2016) Decoding via coding: analyzing qualitative text data through thematic coding and survey methodologies. J Libr Adm 56 (1):41–51

Vendome C, Solano D, Liñán S, Linares-Vásquez M (2019) Can everyone use my app? An empirical study on accessibility in android apps. In: 2019 IEEE International conference on software maintenance and evolution (ICSME). IEEE, pp 41–52

Vitiello G, Sebillo M, Fornaro L, Di Gregorio M, Cirillo S, De Rosa M, Fuccella V, Costagliola G (2018) Do you like my outfit? Cromnia, a mobile assistant for blind users. In: Proceedings of the 4th EAI international conference on smart objects and technologies for social good, Goodtechs ’18, New York, pp 249–254

W3C (2020) Mobile Accessibility at W3C https://www.w3.org/WAI/standards-guidelines/

Walker B N, Tomlinson B J, Schuett J H (2017) Universal design of mobile apps: making weather information accessible. In: International conference on universal access in human-computer interaction. Springer, pp 113–122

Waller A, Hanson V L, Sloan D (2009) Including accessibility within and beyond undergraduate computing courses. Association for Computing Machinery, New York, pp 155–162

Wasserman A I (2010) Software engineering issues for mobile application development. In: Proceedings of the FSE/SDP workshop on future of software engineering research, pp 397–400

Wentz B, Tressler K, et al. (2017) Exploring the accessibility of banking and finance systems for blind users. First Monday

Xie Z, Li N, Luo L (2015) A study and implementation of vga multi-resolution on android platform. In: 2015 International conference on computer and computational sciences (ICCCS). IEEE, pp 110–115

Yan S, Ramachandran P (2019) The current status of accessibility in mobile apps. ACM Trans Access Comput (TACCESS) 12(1):1–31

Zhong Y, Weber A, Burkhardt C, Weaver P, Bigham J P (2015) Enhancing android accessibility for users with hand tremor by reducing fine pointing and steady tapping. In: Proceedings of the 12th Web for all conference, pp 1–10

Download references

Acknowledgments

The authors would like to thank the associated handling editor and the anonymous reviewers for their insightful suggestions and feedback, which were instrumental to improve the quality of our manuscript. Fabio acknowledges the support of the Swiss National Science Foundation through the SNF Project No. PZ00P2_186090.

Open access funding provided by Università degli Studi di Salerno within the CRUI-CARE Agreement.

Author information

Authors and affiliations.

University of Salerno, Fisciano, Italy

Marianna Di Gregorio, Dario Di Nucci, Fabio Palomba & Giuliana Vitiello

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Marianna Di Gregorio .

Ethics declarations

Conflict of interest.

The authors declared that they have no conflict of interest.

Additional information

Communicated by: Neil Ernst

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article belongs to the Topical Collection: Registered Reports

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Di Gregorio, M., Di Nucci, D., Palomba, F. et al. The making of accessible Android applications: an empirical study on the state of the practice. Empir Software Eng 27 , 145 (2022). https://doi.org/10.1007/s10664-022-10182-x

Download citation

Accepted : 20 May 2022

Published : 06 August 2022

DOI : https://doi.org/10.1007/s10664-022-10182-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Mobile accessibility
• Mobile app evolution
• Universal design

Advertisement

• Find a journal
• Publish with us
• Track your research

This paper is in the following e-collection/theme issue:

Published on 23.5.2024 in Vol 26 (2024)

Mobile Health Apps, Family Caregivers, and Care Planning: Scoping Review

Authors of this article:

• Marjorie M Kelley 1 , MS, RN, PhD   ; 
• Tia Powell 2 , MD   ; 
• Djibril Camara 3 , MPH, MD   ; 
• Neha Shah 4 , MSPH   ; 
• Jenna M Norton 4 , MPH, PhD   ; 
• Chelsea Deitelzweig 5 , BA   ; 
• Nivedha Vaidy 4   ; 
• Chun-Ju Hsiao 6 , PhD   ; 
• Jing Wang 7 , MPH, RN, PhD   ; 
• Arlene S Bierman 5 , MS, MD  

1 The Ohio State University College of Nursing, Columbus, OH, United States

2 Montefiore Einstein Center for Bioethics, Albert Einstein College of Medicine, Bronx, NY, United States

3 Credence Management Solution, USAID Global Health Technical Professionals, Washington, DC, United States

4 National Institute of Diabetes and Digestive and Kidney Diseases, Bethesda, MD, United States

5 Agency for Health Care Research and Quality, Rockville, MD, United States

6 Center for Evidence and Practice Improvement, Agency for Health Care Research and Quality, Rockville, MD, United States

7 Florida State University College of Nursing, Tallahassee, FL, United States

Corresponding Author:

Arlene S Bierman, MS, MD

Agency for Health Care Research and Quality

5600 Fishers Lane

Rockville, MD, 20857

United States

Phone: 1 (301) 427 1104

Email: [email protected]

Background: People living with multiple chronic conditions (MCCs) face substantial challenges in planning and coordinating increasingly complex care. Family caregivers provide important assistance for people with MCCs but lack sufficient support. Caregiver apps have the potential to help by enhancing care coordination and planning among the health care team, including patients, caregivers, and clinicians.

Objective: We aim to conduct a scoping review to assess the evidence on the development and use of caregiver apps that support care planning and coordination, as well as to identify key factors (ie, needs, barriers, and facilitators) related to their use and desired caregiver app functionalities.

Methods: Papers intersecting 2 major domains, mobile health (mHealth) apps and caregivers, that were in English and published from 2015 to 2021 were included in the initial search from 6 databases and gray literature and ancestry searches. As per JBI (Joanna Briggs Institute) Scoping Review guidelines and PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews), 2 authors independently screened full texts with disagreements resolved by a third author. Working in pairs, the authors extracted data using a pilot-tested JBI extraction table and compared results for consensus.

Results: We identified 34 papers representing 25 individual studies, including 18 (53%) pilot and feasibility studies, 13 (38%) qualitative studies, and 2 experimental or quasi-experimental studies. None of the identified studies assessed an intervention of a caregiver app for care planning and coordination for people with MCCs. We identified important caregiver needs in terms of information, support, and care coordination related to both caregiving and self-care. We compiled desired functionalities and features enabling apps to meet the care planning and care coordination needs of caregivers, in particular, the integration of caregiver roles into the electronic health record.

Conclusions: Caregiver needs identified through this study can inform developers and researchers in the design and implementation of mHealth apps that integrate with the electronic health record to link caregivers, patients, and clinicians to support coordinated care for people with MCCs. In addition, this study highlights the need for more rigorous research on the use of mHealth apps to support caregivers in care planning and coordination.

Introduction

In 2020, between 17.7 and 40 million Americans were family caregivers of adults aged 65 years or older [ 1 ], defined as unpaid relatives, partners, or friends who assist persons in daily activities due to disease, disability, or other conditions. The need for family caregivers is projected to increase by 2030 with the older adult population and complexity of care increasing [ 2 ]. Many care recipients have multiple chronic conditions (MCCs) defined as the presence of 2 or more chronic physical or mental health conditions [ 3 ]. Over a quarter of the US adult population (27.2%) struggles with MCCs, with the highest prevalence (76.9%) among adults with both Medicare and Medicaid [ 3 ]. People living with MCCs are high users of care, including outpatient, emergency, inpatient, postacute, home, and long-term care, as well as prescription drugs [ 4 ]. People with MCCs account for 64% of all clinician visits, 70% of all in-patient stays, 83% of all prescriptions, 71% of all health care spending, and 93% of Medicare spending [ 5 ].

Complex care routines are common among patients with MCCs and often difficult for people living with MCCs and their caregivers to maintain, leading to avoidable adverse events, poor health outcomes, increased health spending, duplication of services, and polypharmacy [ 6 ]. The many challenges associated with care complexity and care planning add to the physical, psychological, and financial burdens associated with caregiving [ 7 ]. In fact, 14.5% of American caregivers have reported that they experienced mental health decline for at least half the days in a month [ 2 ].

Poor caregiver health and unmet needs have been widely documented and include mental and physical health concerns [ 8 ], unmet need for information on medication and care management to support the care recipient [ 7 ], limited access to supportive services [ 7 ], issues with communication across the care continuum [ 9 ], and burdens associated with work, social isolation [ 7 ], and finances [ 10 ]. Importantly, assistance with care coordination and planning has been consistently noted as an unmet need for caregivers [ 11 ].

Care Planning and Care Coordination

Developing care plans and organizing care involves the marshaling of personnel and other resources needed to carry out essential patient care activities and requires the exchange of information among participants responsible for different aspects of care [ 12 ]. Care planning is a collaborative process focused on discussing patient and clinical goals of care, conducting shared decision-making to identify strategies for clinical and self-management to achieve these goals based on evidence and patient preference, clarifying roles for different members of the care team, and empowering patients and caregivers [ 13 ]. These processes link health professionals, caregivers, and patients in the tasks of designing and implementing care.

Developing a comprehensive care plan both requires and supports care coordination by aggregating and streamlining data on health and social concerns, goals, care management strategies, and health status. Effective care coordination entails the organization of patient care activities to facilitate the appropriate and timely delivery of health care services by multiple clinicians in multiple care settings [ 12 ]. Care coordination involves the patient, clinicians, health care teams including nurses, pharmacists, physical therapists, and social workers, and caregivers. Such care coordination has been shown to benefit multiple domains, including decreased symptoms and mortality, and increased quality of life [ 14 ].

Digital Solutions

Digital solutions offer an opportunity to alleviate some of the care planning and coordination burdens currently shouldered by caregivers and patients. Digital health solutions encompass a variety of information or communication technologies applied to health needs. Digital health is mobile health (mHealth) when implemented on mobile devices. Digital health apps—or programs designed to accomplish specific tasks—fall into the category of mHealth when they are designed to operate on a mobile device.

mHealth apps have the inherent capability of increasing the reach of interventions, and transcending geography and time. They are also often more broadly accessible in the United States, as the uptake of mobile devices is greater than desktop computers [ 11 ]. Furthermore, they can be explicitly tailored to individual needs. Recent advances in technology and software now allow apps to be linked to other digital devices and the electronic health record (EHR).

Several systematic reviews outlined challenges associated with existing apps for caregivers, especially insufficient scientific evidence to support the efficacy of these apps [ 15 - 20 ]. However, no review has focused either on care planning and coordination apps overall or on caregivers of people with MCCs. Moreover, no review focused on the importance of care planning and coordination between the caregiver, care recipient, and professional health care providers. We conducted a scoping review to examine the evidence on the development and use of caregiver apps designed to support care planning and coordination, identify key factors related to their use (ie, needs, barriers, and facilitators), and characterize desired functionality. This review was undertaken to inform the development of a comprehensive, interoperable electronic care plan with clinician-, patient-, and caregiver-facing components to enhance care planning and coordination, address fragmentation of health care, and enhance the collection and sharing of critical patient-centered data across community, clinical, and research settings for people living with MCCs. The Agency for Healthcare Research and Quality (AHRQ) and the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK), with support from the Assistant Secretary for Planning and Evaluation’s Patient-Centered Outcomes Research Trust Fund, are working in partnership to develop an interoperable e-care plan.

We conducted a scoping literature review using JBI (Joanna Briggs Institute) Scoping Review guidelines [ 21 ] and the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) [ 22 ] to guide our methods and reporting. Papers published in English between January 2011 and June 2021 were included. Hence, our initial search activity specific to care planning and coordination revealed a dearth of papers, we broadened our search to include papers intersecting 2 major domains: mHealth apps and caregivers ( Figure 1 ). We hoped to capture available information relevant to care planning and coordination from the perspective of the caregiver. We included mobile health apps like native apps (ie, residing on smartphones) as well as web-based apps designed for smartphone formats. We included all diseases and conditions and care settings (eg, ambulatory, hospital, home, hospice, and long-term care). Study types included pilot and feasibility and experimental and quasi-experimental study designs. Source documents included academic peer-reviewed journals, dissertations and theses, government policy documents, and white papers published by caregiver advocacy organizations (eg, AARP [American Association of Retired Persons] and National Alliance for Caregiving). Studies including paid caregivers or caregivers of patients aged younger than 18 years were excluded. Interventions delivered via social media, phone calls (including interactive voice response), video, telehealth, or text messaging alone were excluded. We also excluded interventions delivered in low- and middle-income countries given significant differences in information technology infrastructure and patterns of use [ 23 ]. As such, comparisons would be difficult. Research interventions involving assistive technologies (ie, motion sensors), non–health related, and health literacy alone were excluded. Source documents such as opinion or editorial papers, conference posters or abstracts, study protocols, blogs, and websites were excluded. Key search terms ( Textbox 1 ) alone or in combination, were used to create our search protocols in 6 databases: PubMed, Cochrane, CINAHL, SCOPUS, Web of Science, and Embase. We conducted ancestry searches of caregiver app reviews and caregiver literature reviews and searched several domain-specific journal databases including the Journal of the American Medical Informatics Association , Journal of Medical Internet Research , International Journal of Medical Informatics , Journal of the American Medical Association , and New England Journal of Medicine .

• Caregiver; caretaker; care provider; carer; care

mHealth app

• mHealth; “mobile health” app; applications; “digital application”; eHealth; and smartphone
• Medical Subject Headings terms: telemedicine [encompasses mHealth]; mobile applications

We exported search results into EndNote, a reference management software platform to eliminate duplications, then uploaded them into Covidence, a web-based systematic review platform, to streamline evidence synthesis and author collaboration. Covidence allowed the research team to work collaboratively when screening papers at the title, abstract, and full-text level. In total, 2 authors independently screened titles and abstracts for eligibility with full-text screening conducted in the same manner. Screening disagreements were resolved through discussion or review by a third author. In keeping with scoping review methodological practices, critical appraisal, and risk of bias were not assessed.

Working in pairs, authors independently extracted data after adapting the JBI data extraction template and a previously used and pilot-tested data extraction table [ 24 ]. Then, each author compared results with the other for consensus about the extracted element. Data extraction elements included first author, publication date, health care domain of the care recipient, country, title, participant demographics, study purpose, study design, intervention description, app name and hyperlink if available, primary app users (ie, patient, caregiver, health care provider, and other), key or primary findings, app features and functionality—including desired functionality, how app supported care coordination, and how app supported caregivers ( Multimedia Appendix 1 [ 25 - 45 ]). For qualitative studies, we extracted data elements associated with caregiver needs and desires. We used conventional content analysis methods, previously described by Hsieh and Shannon [ 46 ], to code and group categories as the phenomena of interest was new with little of the theoretical or literature available to guide the analysis. In keeping with conventional content analysis methods [ 47 ], we relied on inductive category development as categories and subcategories emerged from the literature, followed by deductive category and subcategory assignment.

Of the 3019 nonduplicative records screened, 34 papers [ 25 - 45 , 48 - 60 ] representing 25 individual studies were included in this scoping literature review ( Figure 1 ; Multimedia Appendix 1 ). Publication dates ranged from 2015 to 2021, with 29 (76%) papers published between January 2018 and August 2021. In total, 18 (53%) papers were feasibility, usability, or pilot studies [ 25 - 27 , 29 - 37 , 39 - 43 , 45 ] with qualitative or needs assessment papers representing 38% (n=13) [ 48 - 60 ]. Only 3 papers [ 28 , 38 , 44 ] reported using quantitative research methods to assess intervention efficacy ( Textbox 2 ). Research was predominantly conducted in the United States (22 of 34). Further, 5 papers were from Australia, 3 from Spain, and one each from Canada, the United Kingdom, South Korea, and Turkey. In total, 14 papers focused on cancer caregiving, 7 on dementia caregiving, 6 on general caregiving, 2 each for stem cell transplant and mental health, and one each on heart failure, liver, mental health, and hospice. See Textbox 2 for details of the health care domain and paper type.

Cancer (n=14)

• Experimental and quasi-experimental (n=1)
• Pilot, feasibility, or usability (n=9)
• Qualitative (n=4)

Dementia (n=7)

• Pilot, feasibility, or usability (n=2)

General caregiving (n=6)

• Pilot, feasibility, or usability (n=4)
• Qualitative (n=1)

Mental health (n=2)

• Qualitative (n=2)

Stem cell transplant (n=2)

• Pilot, feasibility, or usability (n=1)

Heart failure (n=1)

Liver (n=1)

Hospice pain management (n=1)

Experimental and Quasi-Experimental Studies

Of the 3 quasi-experimental or experimental studies [ 28 , 38 , 44 ], Park and colleagues [ 38 ] developed an app for caregivers focused on knowledge of dementia, communication, and coping. Ferré-Grau and colleagues [ 28 ] conducted a randomized controlled trial of an app intervention designed to promote caregiver mental health. Finally, research conducted by Uysal et al [ 44 ], used an app for caregivers of patients with cancer focused on caregiver self-care and education. Overall, these studies, like many mHealth interventions for caregivers, addressed important caregiver needs including quality of life. However, none of these apps linked to information in the EHRs or leveraged data standards to support interoperability of data across the care team, nor did the apps provide enhanced communication among caregivers and the health care team. None of the studies investigated or measured care planning or coordination.

Pilot and Feasibility Studies

In total, 18 pilot and feasibility papers [ 25 - 27 , 29 - 37 , 39 - 43 , 45 ], representing 12 studies, were included in this review. The majority (n=14, 78%) of these studies used small convenience samples. Furthermore, 11 of the papers focused exclusively on caregiver mental health or included a component of caregiver mental health in the interventions [ 25 - 27 , 32 - 34 , 36 , 37 , 39 , 40 , 43 ]. In total, 5 reported on apps that included disease education or caregiving education [ 25 , 26 , 31 , 37 , 41 ]. Further, 3 focused on caregiver communications with family and friends [ 41 , 42 , 45 ] but did not assess care coordination or communication with health care providers. One included education on the skills necessary to communicate with health care professionals but did not assess care planning, coordination, or communication as an outcome as it was a feasibility study [ 45 ].

Most of the pilot and feasibility studies focused on the important goal of supporting caregivers’ wellness but did not address care planning or coordination. For example, in one study—with results described in 3 papers [ 27 , 39 , 40 ]—the researchers conducted a 12-week feasibility study using a psycho-educational intervention delivered via video sessions with a goal of caregiver stress reduction. In another study [ 25 , 37 ], investigators used a mindfulness app and assessed cultural sensitivity and barriers to use as feasibility criteria. Kubo and colleagues [ 32 - 34 ] evaluated a commercially available mindfulness app to assess the feasibility of use to improve caregivers’ mental health. Similarly, Sikder and colleagues [ 43 ] pilot-tested an app focused on improving depression symptoms among caregivers.

In total, 7 papers included caregivers only as participants [ 27 , 31 , 36 , 39 , 40 , 42 , 43 ], while 9 papers included caregivers and care recipients as participants [ 25 , 29 , 30 , 32 - 35 , 37 , 41 ]. Only 2 feasibility studies, one conducted by Brown et al [ 26 ] and the other conducted by Wittenberg and colleagues [ 45 ], also included health professionals as participants. Brown and colleagues [ 26 ] examined the feasibility of an app for dementia caregivers, and included caregivers, homecare case managers, and primary health care providers as participants. The platform, CareHeros, was designed with the goal of bidirectional sharing of care recipients’ information between caregivers and health care professionals. The platform did not communicate with EHRs, and bidirectional communication was only reported between case managers and primary care providers, exclusive of caregivers and care recipients. There was limited uptake of the app, with participants logging into CareHeros an average of only 2.18 times over the 11-week period of this study. Wittenberg and colleagues [ 45 ] demonstrated the feasibility of an mHealth app to support caregiver communication skills related to caregiving. The overall objectives of the app development included: (1) to improve caregiver communication skills related to caregiving, (2) to facilitate information sharing among family members, (3) to provide self-care resources for caregivers, and (4) to increase caregiver knowledge. The app was not designed to connect to the EHR, nor was it designed to increase or support communication between caregivers and health care professionals. Caregivers and health care professionals participated in the design and the development of the app as well as usability and acceptability testing. Both groups found the app to be usable and acceptable for helping caregivers with educational needs and communication skills related to caregiving.

While none of the 18 pilot and feasibility studies directly evaluated care planning or coordination as an aim or outcome, 2 [ 30 , 35 ] investigated apps that could assist in care delivery—with caregivers assessing care recipients’ pain [ 35 ] and caregivers assessing care recipients’ hepatic encephalopathy [ 30 ]. Ganapathy and colleagues [ 30 ] used the PatientBuddy app, which sent alerts with critical values regarding hepatic encephalopathy to dyads of patients and caregivers as well as clinicians to support care management, obtaining a positive impact reducing 30-day readmissions in a small cohort. Mayahara et al [ 35 ] conducted a pilot study using e-Pain Reporter, which assisted caregivers in assessing and managing the pain of family members in home hospice. The e-Pain Reporter was designed to provide information on patient pain and pain management to nurses in real time. However, this pilot study did not assess the communication aspect of the app.

In summary, among these pilot and feasibility studies, heterogeneity in study design, interventions, and outcomes preclude meta-analysis, generalization, and direct comparisons. Additionally, most failed to provide support for care planning or coordination and none linked with the EHR or leveraged interoperable data standards. As with most pilot and feasibility studies, these results were preliminary, not powered to identify statistically significant differences in outcomes, and were specific to the app under investigation. Still, a small number of promising studies [ 26 , 30 , 35 , 45 ] attempted to enhance communication or information sharing, a component of care planning and coordination.

Qualitative Studies

In total, 13 (38%) papers [ 48 - 60 ] included in this review were qualitative studies assessing caregiver needs associated with mHealth apps. These caregiver needs were synthesized into 3 broad categories: (1) needs associated with providing care, (2) needs associated with self-care, and (3) desired app features and functionality. In terms of providing care (category 1), caregivers needed information, support, and help with care coordination. For self-care (category 2), caregivers reported a need for information and support. A detailed list of desired mHealth app features and functionality (category 3) is provided in Textbox 3 .

Needs associated with providing care

• Adjusting to a new role
• Information on disease or condition of care recipient
• Information on disease or condition common comorbidities
• Symptom, behavior, or safety
• When to seek help
• Changing nature of caregiving
• Financial and legal services (financial assistance, job help, and health care payment)
• On-demand education and training
• Community support links (transportation or community reintegration)
• Content tailored to care recipients’ needs
• Simple—easy to understand
• Up-to-date scientific evidence and mechanism for updating the information
• Multimodal delivery of information: video, audio, text, or animations
• Always accessible
• Support for care recipients’ physical and emotional needs
• Support with rehabilitation and activities of daily living (oral, bathing, dressing, grooming, toileting, feeding and nutrition, transferring, and ambulation)
• Decision-making support
• Medication management
• Tracking and monitoring of care recipient—mental, physical, emotional, and social (including symptoms, vital signs)
• Content tailored to care recipient’s needs
• Family or personal relationships (asking for help, safety, and communication)
• Integrated app with health care system—care coordination
• Ability to complete questionnaires at home, unrushed
• Finding care equipment
• List of important contacts and contact information for quick reference
• Information and connection to support services (specialty care, first responders, advocacy organizations, and respite services)
• Relationships with health care providers (personal contact)
• Feedback from health care providers—instant
• Automated data entry and reminders or prompts
• “One-stop-shopping”—all information in 1 place

Needs associated with self-care

• Information to help improve caregivers’ health (stress management, peer support, and support groups)
• Activities, programs, and therapy to improve mental, physical, and social support of caregivers
• Content tailored to caregivers’ needs
• Family or personal relationship help (safety or asking others for help or support)
• Preventing social isolation
• Tracking and monitoring of caregiver—mental, physical, emotional, or social (including symptoms, mental health, vital signs)
• Content tailored to caregiver needs
• Social media—“people like me” with expert moderator
• Peer mentor, support, or coaching

Desired mHealth app features and functionality

• Easy to use
• Easy to learn
• Integrated with phone contacts and other apps (exercise and weight management)
• Ability to report care recipient status or symptoms to health care providers and get a response, feedback, or follow-up quickly
• Task reminders (appointments, medication management, etc)
• Integrate with other platforms or devices (electronic health records, smart watches, or pharmacy)
• Share information with family members
• Integrate music or other entertainment
• Track patient symptoms or issues over time
• Track caregiver issues over time
• Customizable
• App from a trusted source and evidence-based content
• Data secure
• Integrated across health care systems
• Not too much information—just in time with the right information
• Font or screen size readable—Americans with Disabilities Act Standards for Accessible Design compliant
• Sustainable
• Help for digital naïve
• Does not reduce time with physician
• Clear perceived benefit
• Ability to personalize features and functions
• Automated data entry

Principal Findings

This scoping review synthesized the evidence on the development and use of caregiver apps designed to enable or support caregiver participation in care planning or care coordination. We identified key factors (ie, needs, barriers, and facilitators) related to care planning and coordination. We described important functionalities and features enabling caregiver apps to meet care planning and coordination needs and facilitate caregiving activities. This comprehensive summary of caregiver needs related to health apps and care coordination may be useful to developers and researchers as it relates to caregivers of those living with MCCs. A better understanding of usability and overall needs will enhance ongoing research efforts to improve e-care planning and care coordination among these populations.

Of the 34 papers, representing 25 individual studies included in this review, only 3 were experimental or quasi-experimental intervention studies [ 28 , 38 , 44 ]. None of the studies included in this review focused on care planning, care coordination, or care recipients with MCCs. This paucity of research precluded generalizations about caregivers’ apps, much less in care planning and coordination. Although most of the studies included in this review addressed important caregiver factors including caregiver education, coping, and self-care, these standalone interventions lacked components to reduce caregiver burdens associated with planning and coordinating complex care. An app designed to specifically improve care planning and coordination, thus reducing this burden, is needed—particularly for the increasing number of care recipients with MCCs.

Most studies within this review were qualitative studies or pilot and feasibility studies. Yet, a few of these studies [ 26 , 30 , 35 , 45 ] identified elements important for care planning and coordination in mHealth apps. By definition, these studies are preliminary in nature thus precluding generalizations; they do not represent proven efficacy or settled science. However, they provide a foundation for future exploration of the role of mHealth interventions in promoting care planning and coordination.

Comparison to Prior Work

Our findings parallel and extend the results identified in a recent review focused on native apps for informal caregiving [ 61 ]. Native apps are apps residing on smartphones as opposed to web-based apps. The principal findings specific to native apps [ 61 ] align with our more comprehensive review (including both native and web-based apps) in that the nascent technology has not matured enough to make meaningful recommendations beyond that of caregiver needs and wants. More rigorous research is needed, specifically among caregivers of patients with MCCs.

In terms of caregiver needs associated with care planning and coordination, caregivers and care recipients in included studies identified several important areas of needs and wants including apps that delivered “one-stop-shopping” or all the information in 1 place. These needs and wants were similar to those identified by Margarido and colleagues [ 61 ] in their 2022 scoping review. The results from both indicated caregivers wanted apps that integrated with the health care system (including the EHR) and could allow them to complete questionnaires at home in an unrushed fashion. They wanted apps that could help them find care equipment and information about support services and support contacts. Relationships with health care providers and feedback from the providers were of key importance, as were timely reminders and prompts (eg, upcoming appointments and medication changes).

Future Directions

More research is needed as this scoping review did not identify any of the following: an app designed to provide access and enhance communication among caregivers, patients, and health care workers, with access for all 3 groups to the EHR; use of data standards in apps to promote interoperability of data across the care team, including caregivers and care recipients; a focus on care planning and coordination; a free and publicly available digital platform; or demonstration of successful usability, efficacy, and sustainability.

The potential exists for emerging mHealth apps to contribute to care coordination by linking caregivers, patients, and clinicians to information and resources that improve the ability of the entire care team to actively engage. Ongoing research focused on developing and evaluating [ 62 - 65 ] interventions to support caregiver engagement in health care through direct EHR access and other digital means could provide important insights. Today, mHealth app-facilitated care planning and coordination remains a possibility, not a reality. This scoping review provides further evidence that existing caregiver-facing mHealth apps are not sufficiently supported by research, with many studies focused on well-educated, tech-savvy female caregivers [ 30 , 37 , 50 ]. There is a need for app development to meet caregiver needs in diverse populations. Most such apps address the burdens of caregiving through interventions aimed at education, self-care, and stress reduction. Though these are helpful, they do not address the fundamental challenges related to care planning and coordination.

Current government federal policies encourage care planning and coordination. There is a federal mandate through the Office of the National Coordinator for Health Information Technology for third-party mHealth apps to integrate with the EHR. These technologies need to be implemented into current health care workflows, but data blocking and the inability to write back to the EHR present challenges. Current workforce shortages, especially for nurses, are well documented and may increase the difficulty of introducing new technologies and tasks, requiring both additional training and time from an already overburdened workforce. On the other hand, a well-designed app that facilitates information sharing, care planning, and communication could potentially reduce the burden.

Strengths and Limitations

We acknowledge several limitations of this study. First, papers included in the scoping review do not include work published after June 2021. It is possible our search terms failed to identify relevant papers in this rapidly developing field. Second, the review included only studies published in English. Though digital health literature is predominantly published in English, there is the possibility of missing important work in other languages. In keeping with guidelines for scoping reviews, we neither assessed the risk of bias nor methodologies in the included studies. Finally, the heterogeneity of included research precluded a meta-analysis of findings across all studies.

This scoping review synthesizes the current evidence on developing mHealth apps to support caregivers in care planning and coordination, providing insights to inform future mHealth app development to engage caregivers as members of the health care team, share critical information across the entire health care team, reduce the burdens caregivers experience in trying to coordinate care, as well as identifying the functionality caregivers desired. Few experimental studies involving apps with needed functionality were identified in the scoping review, even though use of digital technology for caregiver support is a growing interest. We found no studies focused on care planning or coordination, and a very small number of pilot and other preliminary studies addressing specific aspects of care coordination, such as communication. Given the limited number of studies and the preliminary nature of many, there is insufficient evidence on mHealth apps to support caregivers in care planning and coordination. However, the need and potential for further work to achieve these aims is substantial.

Conclusions

In sum, research and evidence on the effective use of mHealth apps to support caregivers involved in care planning and coordination for people living with MCCs is limited. Apps to support caregivers have yet to be integrated into the EHRs. Multidirectional communication between caregivers, care recipients, and health care providers through the EHR holds great promise for relieving the burden on clinicians, patients, and their caregivers alike. The development and implementation of an mHealth app linking the 3 key stakeholder groups to work together to [ 65 ] enhance care planning and coordination, remains an unmet need. Prior work on the functionality desired by caregivers can inform this work.

Acknowledgments

No generative artificial intelligence was used. The findings and conclusions in this document are those of the authors, who are responsible for its content, and do not necessarily represent the views of the Agency for Healthcare Research and Quality (AHRQ), the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK), or the National Institutes of Health. No statement in this report should be construed as an official position of the AHRQ, NIDDK, National Institutes of Health, or the US Department of Health and Human Services. This project was conducted with support from the Assistant Secretary for Planning and Evaluation's Patient Centered Outcomes Research Trust Fund.

Conflicts of Interest

None declared.

Quantitative and pilot, feasibility, or acceptability study information.

PRISMA-ScR checklist.

• Committee on Family Caregiving for Older Adults, Board on Health Care Services, Health and Medicine Division, National Academies of Sciences, Engineering, and Medicine. In: Schulz R, Eden J, editors. Families Caring for an Aging America. Washington, DC. The National Academies Press; 2016.
• Caregiving for family and friends—a public health issue. Center for Disease Control and Prevention. 2018. URL: https://www.cdc.gov/aging/caregiving/pdf/caregiver-brief-508.pdf [accessed 2024-03-14]
• Boersma P, Black LI, Ward BW. Prevalence of multiple chronic conditions among US adults, 2018. Prev Chronic Dis. 2020;17:E106. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Buttorff C, Ruder T, Bauman M. Multiple Chronic Conditions in the United States. Santa Monica, CA. RAND Corporation; 2017.
• Gerteis J, Izrael D, Deitz D, LeRoy L, Ricciardi R, Miller T, et al. Multiple chronic conditions chartbook. In: Agency for Healthcare Research and Quality. Capitol Hill, NC. HHS; 2010.
• Xu X, Mishra GD, Jones M. Evidence on multimorbidity from definition to intervention: an overview of systematic reviews. Ageing Res Rev. 2017;37:53-68. [ CrossRef ] [ Medline ]
• Price ML, Surr CA, Gough B, Ashley L. Experiences and support needs of informal caregivers of people with multimorbidity: a scoping literature review. Psychol Health. 2020;35(1):36-69. [ CrossRef ] [ Medline ]
• Queluz FNFR, Kervin E, Wozney L, Fancey P, McGrath PJ, Keefe J. Understanding the needs of caregivers of persons with dementia: a scoping review. Int Psychogeriatr. 2020;32(1):35-52. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Li J, Luo X, Cao Q, Lin Y, Xu Y, Li Q. Communication needs of cancer patients and/or caregivers: a critical literature review. J Oncol. 2020;2020:7432849. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Aung SHH, White K, Bloomfield J. The experiences and the needs of caregivers of patients with head and neck cancer: an integrative review. Cancer Nurs. 2021;44(6):E361-E373. [ CrossRef ] [ Medline ]
• McGilton KS, Vellani S, Yeung L, Chishtie J, Commisso E, Ploeg J, et al. Identifying and understanding the health and social care needs of older adults with multiple chronic conditions and their caregivers: a scoping review. BMC Geriatr. 2018;18(1):231. [ FREE Full text ] [ CrossRef ] [ Medline ]
• McDonald KM, Sundaram V, Bravata DM, Lewis R, Lin N, Kraft SA, et al. Closing the Quality Gap: A Critical Analysis of Quality Improvement Strategies (Vol. 7: Care Coordination). Rockville, MD. Agency for Healthcare Research and Quality; 2007.
• Baker A, Cronin K, Conway PH, DeSalvo KB, Rajkumar R, Press MJ. Making the comprehensive shared care plan a reality. NEJM catalyst. 2016. URL: https://catalyst.nejm.org/doi/full/10.1056/CAT.16.0838 [accessed 2024-02-23]
• Kastner M, Cardoso R, Lai Y, Treister V, Hamid JS, Hayden L, et al. Effectiveness of interventions for managing multiple high-burden chronic diseases in older adults: a systematic review and meta-analysis. CMAJ. 2018;190(34):E1004-E1012. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chelberg GR, Neuhaus M, Mothershaw A, Mahoney R, Caffery LJ. Mobile apps for dementia awareness, support, and prevention—review and evaluation. Disabil Rehabil. 2022;44(17):4909-4920. [ CrossRef ] [ Medline ]
• Désormeaux-Moreau M, Michel CM, Vallières M, Racine M, Poulin-Paquet M, Lacasse D, et al. Mobile apps to support family caregivers of people with Alzheimer disease and related dementias in managing disruptive behaviors: qualitative study with users embedded in a scoping review. JMIR Aging. 2021;4(2):e21808. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lorca-Cabrera J, Martí-Arques R, Albacar-Riobóo N, Raigal-Aran L, Roldan-Merino J, Ferré-Grau C. Mobile applications for caregivers of individuals with chronic conditions and/or diseases: quantitative content analysis. Int J Med Inform. 2021;145:104310. [ CrossRef ] [ Medline ]
• Petrovic M, Gaggioli A. Digital mental health tools for caregivers of older adults-a scoping review. Front Public Health. 2020;8:128. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sala-González M, Pérez-Jover V, Guilabert M, Mira JJ. Mobile apps for helping informal caregivers: a systematic review. Int J Environ Res Public Health. 2021;18(4):1702. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Vandersman P, Perimal-Lewis L, Tieman J. Is there an app for that? A scoping review of apps for care workers in the aged care sector. In: Telehealth Innovations in Remote Healthcare Services Delivery. Amsterdam, Netherlands. IOS Press eBooks; 2021:124-134.
• Peters MDJ, Marnie C, Tricco AC, Pollock D, Munn Z, Alexander L, et al. Updated methodological guidance for the conduct of scoping reviews. JBI Evid Synth. 2020;18(10):2119-2126. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for Scoping Reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467-473. [ FREE Full text ] [ CrossRef ] [ Medline ]
• McCool J, Dobson R, Whittaker R, Paton C. Mobile health (mHealth) in low- and middle-income countries. Annu Rev Public Health. 2022;43:525-539. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kelley MM, Kue J, Brophy L, Peabody AL, Foraker RE, Yen PY, et al. Mobile health applications, cancer survivors, and lifestyle modification: an integrative review. Comput Inform Nurs. 2021;39(11):755-763. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Beer JM, Smith KN, Kennedy T, Mois G, Acena D, Gallerani DG, et al. A focus group evaluation of breathe easier: a mindfulness-based mHealth app for survivors of lung cancer and their family members. Am J Health Promot. 2020;34(7):770-778. [ CrossRef ] [ Medline ]
• Brown EL, Ruggiano N, Page TF, Roberts L, Hristidis V, Whiteman KL, et al. CareHeroes Web and Android™ apps for dementia caregivers: a feasibility study. Res Gerontol Nurs. 2016;9(4):193-203. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Carr AL, Jones J, Gilbertson SM, Laudenslager ML, Kutner JS, Kilbourn K, et al. Impact of a Mobilized Stress Management Program (Pep-Pal) for caregivers of oncology patients: mixed-methods study. JMIR Cancer. 2019;5(1):e11406. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ferré-Grau C, Raigal-Aran L, Lorca-Cabrera J, Lluch-Canut T, Ferré-Bergadà M, Lleixá-Fortuño M, et al. A mobile app-based intervention program for nonprofessional caregivers to promote positive mental health: randomized controlled trial. JMIR mHealth uHealth. 2021;9(1):e21708. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Frisbee KL. Variations in the use of mHealth tools: the VA mobile health study. JMIR mHealth uHealth. 2016;4(3):e89. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ganapathy D, Acharya C, Lachar J, Patidar K, Sterling RK, White MB, et al. The patient buddy app can potentially prevent hepatic encephalopathy-related readmissions. Liver Int. 2017;37(12):1843-1851. [ CrossRef ] [ Medline ]
• Heynsbergh N, Heckel L, Botti M, Livingston PM. A smartphone app to support carers of people living with cancer: a feasibility and usability study. JMIR Cancer. 2019;5(1):e11779. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kubo A, Altschuler A, Kurtovich E, Hendlish S, Laurent CA, Kolevska T, et al. A pilot mobile-based mindfulness intervention for cancer patients and their informal caregivers. Mindfulness (N Y). 2018;9(6):1885-1894. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kubo A, Kurtovich E, McGinnis M, Aghaee S, Altschuler A, Quesenberry C, et al. A randomized controlled trial of mHealth mindfulness intervention for cancer patients and informal cancer caregivers: a feasibility study within an integrated health care delivery system. Integr Cancer Ther. 2019;18:1534735419850634. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kubo A, Kurtovich E, McGinnis M, Aghaee S, Altschuler A, Quesenberry C, et al. Pilot pragmatic randomized trial of mHealth mindfulness-based intervention for advanced cancer patients and their informal caregivers. Psycho-Oncology. 2024;33(2):e5557. [ CrossRef ] [ Medline ]
• Mayahara M, Wilbur J, Fogg L, Breitenstein SM, Miller AM. Feasibility of e-Pain reporter: a digital pain management tool for informal caregivers in home hospice. J Hosp Palliat Nurs. 2019;21(3):193-199. [ CrossRef ] [ Medline ]
• Otero P, Hita I, Torres, Vázquez FL. Brief psychological intervention through mobile app and conference calls for the prevention of depression in non-professional caregivers: a pilot study. Int J Environ Res Public Health. 2020;17(12):4578. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Owens OL, Smith KN, Beer JM, Gallerani DG, McDonnell KK. A qualitative cultural sensitivity assessment of the breathe easier mobile application for lung cancer survivors and their families. Oncol Nurs Forum. 2020;47(3):331-341. [ CrossRef ] [ Medline ]
• Park E, Park H, Kim EK. The effect of a Comprehensive Mobile Application Program (CMAP) for family caregivers of home-dwelling patients with dementia: a preliminary research. Jpn J Nurs Sci. 2020;17(4):e12343. [ CrossRef ] [ Medline ]
• Pensak NA, Carr AL, Jones J, Mikulich-Gilbertson SK, Kutner JS, Kilbourn K, et al. A pilot study of mobilized intervention to help caregivers of oncology patients manage distress. Psycho-Oncology. 2021;30(4):520-528. [ CrossRef ] [ Medline ]
• Pensak NA, Joshi T, Simoneau T, Kilbourn K, Carr A, Kutner J, et al. Development of a web-based intervention for addressing distress in caregivers of patients receiving stem cell transplants: formative evaluation with qualitative interviews and focus groups. JMIR Res Protoc. 2017;6(6):e120. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Quinn CC, Staub S, Barr E, Gruber-Baldini A. Mobile support for older adults and their caregivers: dyad usability study. JMIR Aging. 2019;2(1):e12276. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Quintana Y, Fahy D, Crotty B, Gorenberg M, Jain R, Kaldany E, et al. A usability evaluation of the InfoSAGE app for family-based medication management. Stud Health Technol Inform. 2019;257:352-357. ITCH. [ Medline ]
• Sikder AT, Yang FC, Schafer R, Dowling GA, Traeger L, Jain FA. Mentalizing imagery therapy mobile app to enhance the mood of family dementia caregivers: feasibility and limited efficacy testing. JMIR Aging. 2019;2(1):e12850. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Uysal N, Bağçivan G, Özkaraman A, Eşer AK, Çetin F, Çalışkan BB, et al. Empowering caregivers in the radiotherapy process: the results of a randomized controlled trial. Support Care Cancer. 2021;29(5):2395-2404. [ CrossRef ] [ Medline ]
• Wittenberg E, Xu J, Goldsmith J, Mendoza Y. Caregiver communication about cancer: development of a mHealth resource to support family caregiver communication burden. Psycho-Oncology. 2019;28(2):365-371. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277-1288. [ CrossRef ] [ Medline ]
• Mayring P. Qualitative content analysis: theoretical background and procedures. In: Bikner-Ahsbahs A, Knipping C, Presmeg N, editors. Approaches to Qualitative Research in Mathematics Education. Advances in Mathematics Education. Dordrecht, Germany. Springer; 2015.
• Bamgboje-Ayodele A, Levesque JV, Gerges M, Girgis A. What male caregivers of women with breast cancer want in an online intervention: a qualitative study. J Patient Exp. 2020;7(6):1771-1777. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Boele FW, van Uden-Kraan CF, Hilverda K, Weimer J, Donovan HS, Drappatz J, et al. Neuro-oncology family caregivers' view on keeping track of care issues using eHealth systems: it's a question of time. J Neurooncol. 2017;134(1):157-167. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chaar D, Shin JY, Mazzoli A, Vue R, Kedroske J, Chappell G, et al. A mobile health app (Roadmap 2.0) for patients undergoing hematopoietic stem cell transplant: qualitative study on family caregivers' perspectives and design considerations. JMIR mHealth uHealth. 2019;7(10):e15775. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Frisbee KL. The impact of mobile health (mHealth) technology on family caregiver's burden levels and an assessment of variations in mhealth tool use. The George Washington University. 2015. URL: https://scholarspace.library.gwu.edu/concern/gw_etds/1544bp27m?locale=de [accessed 2024-03-14]
• Heynsbergh N, Botti M, Heckel L, Livingston PM. Caring for the person with cancer: information and support needs and the role of technology. Psychooncology. 2018;27(6):1650-1655. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Heynsbergh N, Heckel L, Botti M, Livingston PM, Seung CO. Development of a smartphone app for informal carers of people with cancer: processes and learnings. JMIR Form Res. 2019;3(2):e10990. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Huerta-Ramos E, Escobar-Villegas MS, Rubinstein K, Unoka ZS, Grasa E, Hospedales M, et al. Measuring users' receptivity toward an integral intervention model based on mHealth solutions for patients with treatment-resistant schizophrenia (m-RESIST): a qualitative study. JMIR mHealth uHealth. 2016;4(3):e112. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kales HC, Gitlin LN, Stanislawski B, Marx K, Turnwald M, Watkins DC, et al. WeCareAdvisor™: the development of a caregiver-focused, web-based program to assess and manage behavioral and psychological symptoms of dementia. Alzheimer Dis Assoc Disord. 2017;31(3):263-270. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Owen JE, Jaworski BK, Kuhn E, Hoffman JE, Schievelbein L, Chang A, et al. Development of a mobile app for family members of veterans with PTSD: identifying needs and modifiable factors associated with burden, depression, and anxiety. J Fam Stud. 2017;26(2):286-307. [ CrossRef ]
• Rathnayake S, Moyle W, Jones C, Calleja P. Co-design of an mHealth application for family caregivers of people with dementia to address functional disability care needs. Inform Health Soc Care. 2021;46(1):1-17. [ CrossRef ] [ Medline ]
• Ruggiano N, Brown EL, Shaw S, Geldmacher D, Clarke P, Hristidis V, et al. The potential of information technology to navigate caregiving systems: perspectives from dementia caregivers. J Gerontol Soc Work. 2019;62(4):432-450. [ CrossRef ] [ Medline ]
• Shreve J, Baier RR, Epstein-Lubow G, Gardner RL. Dementia caregivers' technology preferences: design insights from qualitative interviews. Gerontechnology. 2016;14(2):89-96. [ FREE Full text ] [ CrossRef ]
• Wali S, Keshavjee K, Nguyen L, Mbuagbaw L, Demers C. Using an electronic app to promote home-based self-care in older patients with heart failure: qualitative study on patient and informal caregiver challenges. JMIR Cardio. 2020;4(1):e15885. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Margarido MG, Shah A, Seto E. Smartphone applications for informal caregivers of chronically ill patients: a scoping review. NPJ Digit Med. 2022;5(1):33. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Piette JD, Striplin D, Marinec N, Chen J, Trivedi RB, Aron DC, et al. A mobile health intervention supporting heart failure patients and their informal caregivers: a randomized comparative effectiveness trial. J Med Internet Res. 2015;17(6):e142. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Aikens JE, Trivedi R, Heapy A, Pfeiffer PN, Piette JD. Potential impact of incorporating a patient-selected support person into mHealth for depression. J Gen Intern Med. 2015;30(6):797-803. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wolff JL, Darer JD, Berger A, Clarke D, Green JA, Stametz RA, et al. Inviting patients and care partners to read doctors' notes: OpenNotes and shared access to electronic medical records. J Am Med Inform Assoc. 2017;24(e1):e166-e172. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Piette JD, Marinec N, Janda K, Morgan E, Schantz K, Yujra ACA, et al. Structured caregiver feedback enhances engagement and impact of mobile health support: a randomized trial in a lower-middle-income country. Telemed J E Health. 2016;22(4):261-268. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by T Leung; submitted 30.01.23; peer-reviewed by J Wolff, Y Chu; comments to author 27.05.23; revised version received 28.09.23; accepted 01.03.24; published 23.05.24.

©Marjorie M Kelley, Tia Powell, Djibril Camara, Neha Shah, Jenna M Norton, Chelsea Deitelzweig, Nivedha Vaidy, Chun-Ju Hsiao, Jing Wang, Arlene S Bierman. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 23.05.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.