literature review requirements elicitation

User Stories in Requirements Elicitation: A Systematic Literature Review

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

• Open access
• Published: 02 February 2024

Recommendation systems-based software requirements elicitation process—a systematic literature review

• Faiz Akram   ORCID: orcid.org/0000-0003-0794-2175 1 ,
• Tanvir Ahmad 1 &
• Mohd. Sadiq   ORCID: orcid.org/0000-0002-6491-5221 2  

Journal of Engineering and Applied Science volume  71 , Article number:  29 ( 2024 ) Cite this article

780 Accesses

Metrics details

Requirements elicitation is one of the fundamental sub-processes of requirements engineering which is used to find the needs of stakeholders. There are several activities in this sub-process, i.e., identification of stakeholders and their requirements, software requirements prioritization, and analysis. Recommendation systems have been intertwined with the requirements elicitation process to predict the stakeholders’ requirements based on their preferences for functional and non-functional requirements. A number of systematic literature reviews (SLRs) have been carried out in the area of requirements elicitation. These SLRs do not support the applications of the recommendation systems during the requirements elicitation process. To deal with this issue, we present an SLR on recommendation systems-based software requirements elicitation processes, from 2009 to 2022, undertaking four research questions: (a) What are the different activities of the software requirements elicitation methods? (b) What are the applications of recommendation systems in the identification of the software requirements? (c) How the recommendation systems can facilitate the identification of stakeholders in the requirements elicitation process? (d) What are the ways to automate the selection of requirements elicitation techniques? The aim of this study is to identify the research gaps in the area of recommendation systems-based requirements elicitation processes and suggest future research directions.

Introduction

Requirements elicitation is the first sub-process of requirements engineering (RE) which is employed to understand and identify the needs of stakeholders. Several methods and techniques have been developed to elicit the needs of stakeholders like contextual methods, cognitive methods, traditional methods, and goal-oriented methods [ 1 ]. Various techniques have been applied to strengthen the requirements elicitation process like fuzzy logic, rough-set theory, and recommendation systems. Among these techniques, recommendation systems are widely used in the literature to deal with a large set of requirements. The aim of a recommender system is to identify the items of interest according to the needs of stakeholders based on their past information or preference relations by applying different kinds of filtering methods, i.e., content-based filtering, collaborative-based filtering, and hybrid-based filtering [ 2 ]. In the literature of RE [ 2 ], the following problems have been addressed by recommendation systems, i.e., (a) identification of stakeholders of a project, (b) elicitation of customer requirements or features of a system, and (c) requirements selection and prioritization. Different datasets have been used in the literature in which recommendation system is used for eliciting the needs of customers, i.e., Replacement Access, Library, and ID Card (RALIC) [ 3 ] and Institute Examination System (IES) [ 4 ]. For example, Lim and Finkelstein [ 3 ] proposed a hybrid approach for eliciting a large set of requirements using social networks and collaborative filtering techniques. The applicability of this approach was discussed by considering the dataset of RALIC. This dataset contains 76 stakeholders and 104 requirements. Hassan et al. [ 4 ] focused on analyzing the stakeholders using confidence value where linguistic variables were used during the recommendation of the stakeholders. The authors have used the “canonical representation of multiplication associated with the \({{\text{L}}}^{-1}{{\text{R}}}^{-1}\) inverse arithmetic principle with graded mean integration representation” for selecting stakeholders based on the recommendations by other stakeholders for developing an IES. The RALIC dataset was also adopted by Shambour et al. [ 5 ] for dealing with one of the issues of requirements elicitation, i.e., information overload. In their work, the authors have developed a hybrid user-item-based method for eliciting the requirements using collaborative filtering.

There are several activities in the requirements elicitation process like stakeholder identification and their requirements, prioritizing the software requirements (SRs), etc., so that the product can be developed in accordance with the needs of the stakeholders. Improper elicitation of stakeholders’ requirements, lack of clarity of goals, and wrong cost estimation are the prominent reasons for a failed or challenged software system. According to the CHAOS report published in 2020, 19% of the projects failed while the outcomes of half of the projects were challenged and 31% of the projects were successful. One of the reasons for the failures of software was inefficient SRs’ identification and their prioritization [ 6 ]. Few studies have used recommendation systems to automate the different activities of requirements elicitation process. There are different types of recommendation systems that are primarily classified on the basis of their recommendation pattern, e.g., collaborative filtering recommendation systems, content-based filtering recommendation systems, and hybrid recommendation systems [ 7 ]. These systems are defined as follows:

Collaborative filtering recommendation system: It provides recommendations on the basis of the prior preferences of similar stakeholders. For example, consider two similar stakeholders \({S}_{1}\) and \({S}_{2}\) , both selecting requirements \({R}_{1}\) and \({R}_{2}\) for a software product. If stakeholder \({S}_{1}\) selects another requirement \({R}_{3}\) for the software product, then \({R}_{3}\) will be recommended to \({S}_{2}\) as well; see Fig.  1 . Collaborative filtering is the prominent technique that is employed in recommender systems. Such recommender systems are dependent on two important computations, i.e., computing similarity and rating prediction. There are different techniques that are used in literature for computing similarity, e.g., Pearson’s correlation coefficient, cosine similarity, adjusted cosine similarity, and Euclidean distance [ 7 ]. Equation  1 describes the formula to compute the similarity between the two stakeholders \({S}_{1}\) and \({S}_{2}\) using Pearson correlation , i.e. ( \(stakeSim\left({S}_{1},{S}_{2}\right)\) ).

Illustration of collaborative filtering recommendation system

Here ,  \({CR}_{{S}_{1},{S}_{2}}\) = Set of co-rated requirements between two stakeholders \({S}_{1}\) and \({S}_{2}\)

\({r}_{{S}_{1}R}\) : Rating given by stakeholder \({S}_{1}\) to requirement R

\({r}_{{S}_{2}R}\) : Rating given by stakeholder \({S}_{2}\) to requirement R

\({\overline{r} }_{{S}_{1}}\) : Average rating of stakeholder \({S}_{1}\)

\({\overline{r} }_{{S}_{2}}\) : Average rating of stakeholder \({S}_{2}\)

Values generated by Eq.  1 lie between “1” and “−1”. The perfect agreement between the stakeholders \({S}_{1}\) and \({S}_{2}\) is represented by value “1”; on the other hand, “−1” represents the value of perfect disagreement between them. This process is repeated to compute the pair-wise similarities between different stakeholders. Further, a neighborhood is generated for each stakeholder by choosing “ N ” for most similar stakeholders. The interest level of stakeholders, e.g., \({S}_{1}\) , for requirement “ R ” which \({S}_{1}\) has not yet rated is predicted by the following equation:

Here \({S}_{2}\in nbr({S}_{1})\) : \({S}_{2}\) is a neighbor of \({S}_{1}\)

Content-based filtering recommendation system: It generates recommendations on the basis of the past preferences of a stakeholder [ 7 ]. For example, if a stakeholder \({S}_{1}\) selects a requirement \({R}_{1}\) for a software product, then requirements similar to \({R}_{1}\)  (e.g., \({R}_{2}\) ) will be recommended to \({S}_{1}\) , as illustrated in Fig.  2 . Content-based filtering in requirements elicitation requires representation of requirements, stakeholder profiling, and calculating similarity measure for generating recommendations. Each requirement \({R}_{i}\) is mathematically represented as a feature vector \({V}_{i}\) . Stakeholder profile \({P}_{S}\) is then created after a combination of the feature vectors of the stakeholders’ requirements in which weights are assigned based on stakeholders’ preferences; see Eq. ( 3 ).

where \({w}_{S,i}\) is the weight assigned to the requirements based on stakeholders’ preferences.

Illustration of content-based filtering recommendation system

Similarity measure is computed between stakeholder profile \({P}_{S}\) and each requirement’s feature vector \({V}_{i}\) . Equation ( 4 ) represents the formula to calculate cosine similarity between \({P}_{S}\) and \({V}_{i}\) .

where \({||P}_{S}||\, {\text{and}}\, {||V}_{i}||\) are the Euclidean norms of vectors \({P}_{S} \,{\text{and}} \,{V}_{i}\) respectively.

Based on the similarity values, the requirements are ranked signifying the corresponding relevance during requirements elicitation.

Hybrid recommendation system: It generates recommendations by amalgamating the features of both collaborative and content-based recommendation as per the requirements of the product [ 7 ].

Various SLRs have been conducted to investigate the strength and weaknesses of the existing requirements elicitation techniques. For example, Pacheco and Garcia [ 8 ] conducted an SLR on the stakeholders’ identification methods applied in requirements elicitation. Consequently, it was found that the methods based on stakeholders’ identification have been categorized into the following: (a) methods describing stakeholders, (b) methods based on the interaction between the stakeholders, (c) methods assessing the stakeholders. The assessment of the stakeholders depends on the priority interest and the skills of the stakeholders in the project development. One way of getting the expected quality of software product is to understand the environmental domain where the software project is expected to be developed. An efficient and effective stakeholders’ selection results in an improved coverage of requirements [ 8 ]. Hujainah et al. [ 9 ] performed an SLR on the significance of stakeholders and techniques in requirements prioritization process. The authors have categorized the requirements prioritization techniques based on the manual, semi-automated, and fully automated types. Aldave et al. [ 10 ] investigated the requirements elicitation techniques within agile software development. To date, the most recent SLR is on the data-driven requirements elicitation performed by Lim et al. [ 11 ] emphasizing automated requirements elicitation for information systems. A list of selected SLRs in the area of SRs elicitation is summarized in Table  1 . To our knowledge, no SLR has been performed with an emphasis on the applications of recommender systems in requirements elicitation. Thus, the objective of this study is to perform an SLR in the area of recommendation systems-based requirements elicitation so that key stakeholders and their requirements can be identified and recommended during the requirements elicitation process.

The subsequent sections are organized as follows: The “ Research methodology ” section discusses the research methodology for conducting the SLR. An insight into the threats to validity is described in the “ Threats to validity ” section. The results and discussion based on the research questions (RQs) are summarized in the “ Results and discussion ” section. A comparative study between the SLR on the recommendation systems-based requirements elicitation process and other selected SLRs related to requirements elicitation processes is performed in the “ Comparative study ” section. And in the end, the conclusion, challenges, and future work are summarized in the “ Conclusions, challenges and future work ” section.

Research methodology

The guidelines proposed by Kitchenham and Charters [ 14 ] are applied to identify the research gaps in the area of recommendation systems-based SRs elicitation process. In accordance with [ 14 ], the following steps are taken to perform the SLR: (a) research questions, (b) search strategy, (c) study selection, and (d) data synthesis. A detailed description about these steps is given below:

Research questions

The objective of recommendation systems-based methods for the elicitation of SRs is to capture the needs and preferences of the stakeholders so that different types of SRs can be identified in global software development according to the needs of the stakeholders. These stakeholders are placed at different locations in a country or abroad. To achieve this objective, the following RQs are formulated:

RQ-1 : What are the different activities of the SRs elicitation methods?

RQ-2 : What are the applications of recommendation systems in the identification of the SRs?

RQ-3 : How the recommendation systems can facilitate the identification of stakeholders in requirements elicitation process?

RQ-4 : What are the ways to automate the selection of requirements elicitation techniques?

Search strategy

The following keywords have been derived from the above RQs:

“software requirements elicitation,” “recommendation systems,” “stakeholder,” “automation,” “elicitation technique.”

A search string was constructed based on the above keywords. The synonyms of these keywords were also made to complete the string so that relevant studies based on the RQs can be identified. The terms that appeared in the keyword were expanded using Word Net Version 3.0 [ 15 ] and Oxford Dictionary ( https://www.oxfordlearnersdictionaries.com/ ) of English synonyms. Finally, the following search string was created to identify the primary studies from the electronic databases:

Search string: ((Software requirements elicitation OR Software requirements engineering OR Requirements elicitation OR Requirements engineering) AND (Recommendation system OR Recommender system OR Automation OR Limitations OR Weaknesses OR Strength OR Advantages OR Disadvantages) AND (Review OR Systematic review OR Literature review OR Systematic literature review OR Survey OR Journey OR Literature mapping OR Systematic literature mapping OR State-of-the-art)).

We used the following five electronic databases to search the relevant primary studies based on the search string: IEEE Xplore , ACM digital library , Springer , ScienceDirect , and Google Scholar .

Study selection

The search strategy for the study selection process is exhibited in Fig.  3 . We have selected the primary studies published from 2009 to 2022; see Fig.  4 . Initially, 170 primary studies were selected from the five electronic databases. After being subjected to scrutiny on the basis of title, 60 of the studies were unlisted from the SLR as they were unnecessary and irrelevant. As a result, 110 primary studies were selected based on the title. These studies were further analyzed on the basis of the abstract and conclusion and 60 studies were shortlisted.

Search strategy for the study selection process

Year-wise distribution of primary studies

The selected studies were finally assessed on the basis of the following quality assessment (QA) criteria:

QA-1: “Do the selected studies serve the purpose of answering the RQs?”

QA-2: “Is the aim of the research clearly conveyed?”

QA-3: “Is there any case study to support the research?”

QA-4: “Does the research add any value to industry or academia?”

A grade point of 0.5 is assigned for a QA question when a study partially answers the question. When the study satisfactorily answers the QA criteria, it is assigned a grade point of 1.0. For every selected study, the sum of the grade points corresponding to the QA criteria is evaluated. If the sum is greater than or equal to 2.0, the study is selected as a primary study for the SLR. Finally, 50 studies were identified as primary studies and selected for the SLR. A list of the selected 50 primary studies is summarized in Appendix 1 and the quality assessment scores of these studies are presented in Appendix 2 . The primary studies include research papers from the following journals and conferences of international repute, e.g., “IEEE Transactions on Software Engineering,” “Requirements Engineering Journal,” “Conference proceedings published by LNCS,” “International Conference on Software Engineering,” and “International Conference on Requirements Engineering.” The formulated RQs have been answered by considering the selected 50 primary studies.

Data synthesis

The data related to the 50 primary studies are synthesized for answering the formulated RQs. The following ways are adopted to synthesize the data: The answer to the RQ-1 is depicted by a bar graph to illustrate the various activities carried out during SRs elicitation methods; see Fig.  5 . The answer to the RQ-2 is tabulated in Table  2 , which summarizes the different activities of SRs elicitation where recommendation system facilitates the identification of SRs. The result for RQ-3 is represented in a tabular form (see Table  3 ), which depicts various techniques employed by recommendation systems to automate the identification of stakeholders in requirements elicitation process along with the level of automation and the scale of the software project under consideration. The observations to the RQ-4 are documented in the form of text.

Different activities in requirements elicitation methods

Threats to validity

The aim of this section is to discuss the potential issues that can affect the conclusion of an SLR. There are four major threats to validity in any SLR, “conclusion validity,” “internal validity,” “construct validity,” and “external validity” [ 16 ]. These threats are an essential part of any SLR. For example, Sadiq et al. [ 16 ] discussed these threats for selecting the SRs with incomplete linguistic preference relations. A brief discussion about different types of validation is given below:

One of the key threats to the conclusion validity is the biasness in the selection of appropriate primary studies and the synthesis of data. To weaken this threat, a systematic study selection approach was designed for the inclusion and exclusion of the primary studies based on the QA criteria. This approach was enforced accurately to confirm the correctness of each included primary study.

In internal validity , the relationship between the variables of interest and the results is discussed. In this study, the applications of the stakeholders during the requirements elicitation and prioritization are discussed in the context of the recommendation systems. Few methods have been applied only for the small datasets of SRs with a main focus on the local software development rather than considering the global software development. As a result, the right performance of the methods to deal with large projects using recommendation systems might not be evaluated. To alleviate this issue, multiple sources of publications of the same work have been considered, whenever possible.

The relationship between the application and theory is discussed in construct validity . One of the threats to the construct validity arises from the elimination of possible related primary studies. To curb this threat, the search procedure was defined and all the studies related to the recommendation systems-based SRs elicitation and prioritization domain were included. The grey studies related to the domain were excluded.

Finally, the threats to external validity include the questions that limit the capability to generalize the findings of SLR outside the scope of the study. In this study, the grey and non-English studies are excluded. We believe that the review protocol used in this study helped us to choose a typical set of studies related to domain knowledge. The results of this study are more concerned with the requirements elicitation using recommendation systems domain from the industrial as well as the academic perspectives.

Results and discussion

In this SLR, the 50 primary studies are identified based on the review protocol. These studies have been published from 2009 to 2022. The year-wise distribution of the primary studies is depicted in Fig.  4 .

RQ-1: What are the different activities of the SRs elicitation methods?

Elicitation is the group of activities that helps in gathering information for a specific problem domain [S1, S2], and errors occurring in this phase hamper the success of the proposed software system [S3, S4]. The requirements elicitation, carried out by following group of activities, is governed by different factors like scope and objectives of the project, type and place of the organization, and scale of the system. These factors enhance the clarity, consistency, effectiveness, and unambiguity of the requirements [S5, S6].

Based on our analysis, it is found that some of the activities are common in requirements elicitation methods. For example, Pohl [S7] described three broad categories of the requirements elicitation activities that are commonly performed. Sandhu and Weistroffer [S8] highlighted the influence of requirements elicitation in the fulfillment of the needs of the organization and the stakeholders. The authors listed five primary tasks in requirements elicitation and comprehensively reviewed the importance and the challenges linked with the tasks. Mulla and Girase [S9] proposed a compendious study and focused on five different activities for the requirements elicitation. Sharma and Pandey [S10] discussed the need for a more detailed description of the requirements elicitation. In their work, the authors have considered ten activities of the requirements elicitation and identified various concerning challenges associated with them. Bani-Salameh and Aljawabrah [S5] also proposed 10 different activities in their requirements elicitation model. The authors through their model tried to generate all the correct requirements for a given software project. Jalil et al. [S11] divided the requirements elicitation processes into 5 phases. Wong and Mauricio [S12] proposed seven different activities embedded in the requirements elicitation process. The authors also identified the factors influencing these elicitation activities. The “software engineering body of knowledge” (SWEBOK) [S13] has proposed two activities that are considered the most important in SRs elicitation.

On the basis of our review, Fig.  5 summarizes the requirements elicitation activities which are common in the existing elicitation methods. From Fig.  5 , it is clear that the following activities are common in most of the requirements elicitation techniques: identifying the application domain, stakeholder identification, and analysis and identifying the sources of requirements, documentation, and refinement. In addition to these activities, some activities have also been introduced to handle vagueness and imprecision in the elicitation methods. For example, Sadiq [S14] proposed a fuzzy-based approach for stakeholders’ analysis so that key stakeholders can be identified based on the importance of SRs. Among various SRs elicitation techniques, goal-oriented methods have also been given due importance. In such methods, the goals of the stakeholders are broken down into sub-goals to get the functional requirements (FRs) and non-functional requirements (NFRs) of the system [S15]. Mohammad et al. [S16] developed a fuzzy-based method for SRs analysis in the goal-oriented domain. To address the issues of fuzzy-based methods, Sadiq and Devi [S17] proposed a method using rough-set theory for the prioritization of SRs. In another study, Sadiq and Devi [S18] developed a method for the selection of requirements of an IES using fuzzy-soft set approach. In these studies, small and medium datasets have been applied for the explanation of the proposed methodologies. Amaral and Elias [S19] proposed a risk-driven multiple objective evolutionary method for SRs selection. In a recent study, Nazim et al. [S20] discussed different types of datasets used in SRs selection and prioritization research. In their study, fuzzy AHP and fuzzy TOPSIS have been compared based on the datasets of an IES.

RQ-2: What are the applications of recommendation systems in the identification of the SRs?

There are different activities associated with the requirements elicitation process as shown in Fig.  5 . These activities are very essential to gather the correct requirements from different stakeholders using different techniques. When performed manually, these elicitation activities can be extensively time-consuming and prone to errors [S21]. Recommendation systems help in solving this problem.

In requirements elicitation methods, recommendation systems are used to recommend the prospective stakeholders. The recommendation systems keep the stakeholders updated by providing information about the project. The consensus among the stakeholders can also be achieved through the recommendation systems if the points of view of the stakeholders are efficiently considered [S22, S23].

Dumitru et al. [S24] proposed a recommendation system that deploys an incremental clustering approach for domain analysis. The approach emphasized the specifications of the available software products to recommend features that are viable for the software product under development using Association Rule Mining (ARM) and k-Nearest-Neighbor (k-NN). There are some studies in which recommendation systems have been combined with social networks so that the requirements of large-scale project can be identified. For example, Lim et al. [S25] developed a StakeNet methodology for eliciting a large set of SRs. The authors have employed social network measures for automating stakeholder analysis so that stakeholders can recommend each other as per the necessities of the project. One of the key steps of the StakeNet methodology is to collect profiles of the stakeholders and gather information in order to generate a prioritized list of the SRs.

Various other studies have also discussed the applications of recommendation systems during the requirements elicitation process. For example, a context-aware recommendation system proposed by Roher and Richardson [S26] addresses activities like exploring the application domain, identifying the goals of the organization, and channelizing these goals. The proposed recommender system also takes into consideration the place of deployment of the project to assist in integrating sustainability. Ninaus et al. [S27] developed an approach to assist elicitation activities that includes supporting stakeholders’ identification, prioritization and quality assurance of requirements, requirements reuse, and planning the software release. This approach is referred to as INTELLIREQ, which utilizes the advantages of different recommendation techniques, e.g., content-based recommendation using Dice coefficient, group recommendation using Majority voting, knowledge-based recommendation using preference matrix, etc., to make the requirements model more consistent and proactive. Iqbal et al. [S28] discussed the effects and applications of machine learning to automate the different requirements engineering tasks, e.g., requirements elicitation and discovery and requirements specification. The authors pointed out that machine learning provides better decision-making for a software project dealing with a large dataset with high degree of imprecision and ambiguity. Ahmad and Sadiq [S29] proposed a recommendation systems-based approach for prioritizing the requirements of an IES. In their work, the authors have identified a list of stakeholders and their requirements. Consequently, the elicited FRs and NFRs were selected by using the “ \({{\text{L}}}^{-1}{{\text{R}}}^{-1}\) inverse function arithmetic principle and graded mean integration” representation. Lunarejo [S30] has proposed a semi-automatic multi-criteria approach to address scalability and lack of automation in requirements identification for FRs and NFRs of software products. The proposed approach has been evaluated using real web-based geographic information systems (GIS). Mohebzada et al. [S31] in their systematic mapping focused on the applications of the recommendation systems in recommending stakeholders, priority of requirements, similar requirements, etc. It was observed that collaborative filtering has been used to generate the recommendations.

From the literature, it is evident that recommendation systems play a prominent role during requirements elicitation process and the automation tools and techniques based on artificial intelligence, and machine learning has received much attention during the requirements elicitation process. Table 2 summarizes the different activities of SRs elicitation where the recommendation system facilitates the identification of SRs. But simultaneously there are certain challenges of recommender systems as well during the elicitation activities, e.g., different types of stakeholders’ feedback and relationships between requirements [S32].

RQ-3: How the recommendation systems can facilitate the identification of stakeholders in requirements elicitation process?

People from different domains are engaged in requirements elicitation for a software project. These individuals or group of individuals are the stakeholders. The success and the failure of software project are influenced by the involvement of varied stakeholders [S9]. Identifying the stakeholders has always been intensive during the implementation and requires great deal of effort and time to finalize the complete list of stakeholders for a software project. Recommendation systems facilitate these tasks and collects information that are valuable for the proposed software system [S36, S37] as depicted in Fig.  6 . Thus, many studies have focused on simplifying and automating the stakeholders’ identification process. Mulla and Girase [S9] in their study have identified various social network measures that can be used for the recommendation and also for the prioritization of stakeholders. The study suggested to build a social network whose nodes depict stakeholders. The links of the network are the stakeholder’s recommendations. Here, the stakeholders recommend other stakeholders for the identification and prioritization. Castro-Herrera et al. [S38] analyzed the downside of the traditional techniques in large software projects with substantial number of stakeholders. The authors further highlighted the significance of the recommendation system in facilitating the recognition of key stakeholders.

Stakeholders recommendation process in a software project

In [S38], the authors have suggested a hybrid recommendation system to identify potential users to solve the unattended threads in open-source forums. Lim et al. [S25] developed StakeNet, which identifies, recommends, and prioritizes the stakeholders associated with a software project using social networks. Hujainah et al. [S39] proposed a new semi-automated technique termed StakeQP that facilitates stakeholder quantification and prioritization during software requirements elicitation. This method was evaluated by using the RALIC dataset to show its relevance and effectiveness in facilitating the recommendation of stakeholders and the subsequent requirements. Palomares et al. [S40] developed the OpenReq approach to propose recommendations of relevant stakeholders for the requirements elicitation. The approach utilizes the collaborative filtering recommendation mechanism by examining the contributions of the stakeholders in the earlier software projects, analyzing the strength and the interest of the stakeholders in a software project domain and personal availability of the stakeholders. Hariri et al. [S41] in their work suggested the identification and recommendation of stakeholders with proficiency in the software project under consideration. The authors have proposed the implementation of a hybrid recommender system to recommend three categories of stakeholders, i.e., direct stakeholders, indirect stakeholders, and inferred stakeholders. The work by Castro-Herrera et al. [S42] concentrates on identifying and bringing together different stakeholders into relevant online forums and discussion groups. The authors have incorporated both data mining and machine learning in their framework for providing semi-automated assistance to manage these requirements forums. To identify and recommend potential experts for a domain, the approach by Castro-Herrera and Cleland-Huang [S43] automatically examines the contribution of different stakeholders. The approach then uses machine learning to identify, examine, and categorize these contributions into different domains. The profiles of the stakeholders thus created help in classifying the stakeholders. Milano et al. [S44] in their analysis of multi-stakeholder recommendation systems suggest the conclusive advantages of multi-stakeholder approach over the traditional user-centric perspective of recommendation systems. The authors have highlighted and suggested to study the impact of recommendation systems on the benefits of different stakeholders. The authors have also credited the role of recommendation systems in facilitating the interactivity of a large number of stakeholders in online forums. Felfernig et al. [S45] in their study observed a growing need for smart software systems to automate the support to stakeholders. The authors have channelized the importance of social networks in the identification and recommendation of stakeholders and clustering of SRs for identifying the dependency among them.

Based on our review, we observed that stakeholders, being a prominent factor in the success of a software project, have received due consideration in the recommendation systems. Researchers and academicians have put forward various models to automate the activities pertaining to the stakeholders, i.e., identification and analysis of stakeholders, recommending stakeholders, providing recommendations to stakeholders, etc. Stakeholder identification methods have mainly focused on different types of projects, techniques used for its recommendation, and levels of automation; see Table  3 .

RQ-4: What are the ways to automate the selection of requirements elicitation techniques?

The requirements elicitation techniques are employed to examine the stakeholders’ needs in determining the requirements of the software under consideration. These techniques may be grouped into traditional techniques, group elicitation techniques, cognitive techniques, contextual techniques, goal-oriented techniques, quality function deployment methods, package-oriented requirements elicitation, etc. One of the challenges which is faced by facing requirements’ engineer during SRs elicitation is the selection of an appropriate elicitation technique [S46].

Darwish et al. [S47] listed a variety of factors that decide the techniques to be selected for a given software project. These factors include the level of criticality of the project under consideration, size of the project, and degree of project complexity. In their work, a hybrid machine learning model to select elicitation techniques has been proposed using 3-component approach. The approach begins with a literature review to identify common elicitation techniques and the factors affecting them. The second step is the identification of the factors affecting the technique selection using a multiple regression model. Finally, the required elicitation techniques are selected using a proposed Artificial Neural Network model. Tiwari et al. [S48] focused on selecting the methods for SRs elicitation. The authors underlined how the lack of recommendation system to select the elicitation techniques forced the stakeholders to use traditional company practices or individual experience. Hussein et al. [S46] in their work listed 19 factors which have been categorized into 4 broad categories, i.e., elicitor, stakeholder, project, and elicitation process that influence the specification and selection of requirements elicitation techniques. The authors have developed a prototype to assist the requirements engineers in performing the task. While most of these selections are done manually using the expertise of the requirement engineers, Ibrahim et al. [S49] proposed a model to automate the process. In their proposed model, the authors have used machine learning approach to automate the selection of elicitation technique. The authors have used k-NN algorithms to select the most appropriate technique to assist the requirement engineers in planning for the new project accounting for the requirements complexity characteristics. Moreover, Dafaalla et al. [S50] proposed a deep learning-based decision-making model for automating requirements elicitation technique selection. The authors, through their model, intend to reduce human errors thus enhancing the efficiency of the requirements elicitation in the development of a software project.

Based on our review, it is found that very few studies have focused on the automation of the requirements elicitation techniques selection. In most of the studies, theoretical and heuristic approaches have been explored for selecting elicitation techniques for the project under consideration. But few recent studies have proposed machine learning models to automate the requirements elicitation techniques selection.

Comparative study

In this section, we have compared the SLR between the recommendations systems-based SRs elicitation process and other selected methods based on the following criteria: area of SLR, year of publication, RQs, and support of recommendation system. The result is exhibited in Table  4 . On the basis of the comparative study, it is found that requirements elicitation is a key process of software development. There are different aspects of requirements elicitation like stakeholders’ identification [ 8 ], maturity of the requirements elicitation techniques [ 1 ], application of data-driven concepts in requirements elicitations [ 11 ], and SRs selection [ 16 ]. Different issues related to the requirements elicitation techniques have been discussed in the existing SLRs [ 1 , 8 , 9 , 10 , 11 , 12 , 13 ]. We could not find any relevant study from 2009 to 2022 that presents an SLR in the area of recommendation systems-based requirements elicitation process. Therefore, in this paper, an attempt has been made to fill this research gap.

Conclusions, challenges and future work

This paper presents an SLR of the recommendation system-based SRs elicitation process. In this SLR, a review protocol was constructed to formulate the RQs. The results of this SLR present the different activities involved in the identification of the SRs, applications of the recommendation systems during the SRs elicitation process, and also the identification of the stakeholders, and finally, it discusses the ways to automate the selection of SRs elicitation techniques.

For SRs elicitation process, elicitation activities are indispensable for selecting a comprehensive, complete, and consistent list of requirements for project under consideration. Our SLR finds that identifying the application domain, stakeholder identification, identification of the sources of requirements, stakeholder analysis, and selection of tools, techniques, and approaches for elicitation are the prominent elicitation activities that have been used in real-life applications. Most of these elicitation activities were performed manually in the form of brainstorming, questionnaires, discussions, meetings, etc., but in recent times there has been an incremental rise in automating the tasks. Thus, our SLR further underlines the importance of various recommendation techniques in requirements elicitation and in particular the recommendation and identification of stakeholders and their requirements.

Collaborative recommendation techniques and social network measures are the most implemented methodologies to automate the stakeholders’ recommendation process for a software project. On the other hand, less attention is given to recommendation-based approaches for analysis of risks, software cost estimation, requirements tracking, and identification of NFRs. The findings revealed that existing SRs elicitation techniques have some limitations and associated challenges. For example, for a large-scale system with a very large number of stakeholders or for system where stakeholders change their opinions very frequently, there may be discordances in the stakeholders’ requirements and subsequent goals. These discordances are addressed by negotiation between the stakeholders which are mostly performed heuristically. Categorization of these discordances and automating the negotiations to increase the quality of the software needs further research. Additionally, in goal-oriented requirements elicitation method, the main emphasis is on the analysis of SRs using AND/OR graphs. Integrating the recommendation systems with goal-oriented methods for analyzing the goals and stakeholders’ discordances is an important issue that needs to be addressed in the future.

Availability of data and materials

Not applicable.

Abbreviations

Systematic literature reviews

Software requirements

Quality assessment

Institute Examination System

Replacement Access, Library, and ID Card

Geographic Information Systems

Association Rule Mining

K-Nearest-Neighbor

Pacheco C, Garcia I, Reyes M (2018) Requirements elicitation techniques: a systematic literature review based on the maturity of the techniques. IET Softw 12(4):365–378

Article   Google Scholar  

Mobasher B, Cleland-Huang J (2011) Recommender systems in requirements engineering. AI Mag 32(3):81–89

Google Scholar  

Lim SL, Finkelstein A (2012) StakeRare: using social networks and collaborative filtering for large-scale requirements elicitation. IEEE Trans Softw Eng 38(3):707–735

Hassan T, Mohammad CW, Sadiq M (2020) StakeSoNet: analysis of stakeholders using social networks. In: 2020 IEEE 17th India council international conference. pp 1–6

Shambour QY, Abu-Alhaj MM, Al-Tahrawi MM (2020) A hybrid collaborative filtering recommendation algorithm for requirements elicitation. Int J Comput Appl Technol 63(1–2):135–146

Lauesen S (2020) IT project failures, causes and cures. IEEE Access 8:72059–72067

Roy D, Dutta M (2022) A systematic review and research perspective on recommender systems. J Big Data 9(1):59

Pacheco C, Garcia I (2012) A systematic literature review of stakeholder identification methods in requirements elicitation. J Syst Softw 85(9):2171–2181

Hujainah F, Bakar RBA, Abdulgabber MA, Zamli KZ (2018) Software requirements prioritisation: a systematic literature review on significance, stakeholders, techniques and challenges. IEEE Access 6:71497–71523

Aldave A, Vara JM, Granada D, Marcos E (2019) Leveraging creativity in requirements elicitation within agile software development: a systematic literature review. J Syst Softw 157:110396

Lim S, Henriksson A, Zdravkovic J (2021) Data-driven requirements elicitation: a systematic literature review. SN Comput Sci 2(16):1–35

Horkoff J, Aydemir FB, Cardoso E et al (2016) Goal-oriented requirements engineering: a systematic literature map. In: IEEE 24th International Requirements Engineering Conference (RE) 2016. pp 106–115

Wong LR, Mauricio D, Rodriguez GD (2017) A systematic literature review about software requirements elicitation. J Eng Sci Technol 12(2):296–317

Kitchenham BA, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering, technical report EBSE-2007-01, school of computer science and mathematics. Keele University, UK

Princeton University "About WordNet." WordNet. Princeton University. 2010. https://wordnet.princeton.edu/

Sadiq M, Parveen A, Jain SK (2021) Software requirements selection with incomplete linguistic preference relations. Bus Inf Syst Eng 63:669–688

Download references

Acknowledgements

The authors did not receive any funding from any organization/institution.

Author information

Authors and affiliations.

Department of Computer Engineering, Faculty of Engineering and Technology, Jamia Millia Islamia, A Central University, New Delhi, 110025, India

Faiz Akram & Tanvir Ahmad

Software Engineering Lab., Computer Engineering Section, UPFET, Jamia Millia Islamia, A Central University, New Delhi, 110025, India

Mohd. Sadiq

You can also search for this author in PubMed   Google Scholar

Contributions

All the authors have equal contributions in the research study.

Corresponding author

Correspondence to Faiz Akram .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Akram, F., Ahmad, T. & Sadiq, M. Recommendation systems-based software requirements elicitation process—a systematic literature review. J. Eng. Appl. Sci. 71 , 29 (2024). https://doi.org/10.1186/s44147-024-00363-4

Download citation

Received : 18 July 2023

Accepted : 03 January 2024

Published : 02 February 2024

DOI : https://doi.org/10.1186/s44147-024-00363-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Requirements engineering
• Goal-oriented requirements engineering
• Recommendation system
• Requirements elicitation
• Stakeholders
• Systematic literature review

• Open access
• Published: 13 May 2024

What are the strengths and limitations to utilising creative methods in public and patient involvement in health and social care research? A qualitative systematic review

• Olivia R. Phillips 1 , 2   na1 ,
• Cerian Harries 2 , 3   na1 ,
• Jo Leonardi-Bee 1 , 2 , 4   na1 ,
• Holly Knight 1 , 2 ,
• Lauren B. Sherar 2 , 3 ,
• Veronica Varela-Mato 2 , 3 &
• Joanne R. Morling 1 , 2 , 5  

Research Involvement and Engagement volume  10 , Article number:  48 ( 2024 ) Cite this article

103 Accesses

2 Altmetric

Metrics details

There is increasing interest in using patient and public involvement (PPI) in research to improve the quality of healthcare. Ordinarily, traditional methods have been used such as interviews or focus groups. However, these methods tend to engage a similar demographic of people. Thus, creative methods are being developed to involve patients for whom traditional methods are inaccessible or non-engaging.

To determine the strengths and limitations to using creative PPI methods in health and social care research.

Electronic searches were conducted over five databases on 14th April 2023 (Web of Science, PubMed, ASSIA, CINAHL, Cochrane Library). Studies that involved traditional, non-creative PPI methods were excluded. Creative PPI methods were used to engage with people as research advisors, rather than study participants. Only primary data published in English from 2009 were accepted. Title, abstract and full text screening was undertaken by two independent reviewers before inductive thematic analysis was used to generate themes.

Twelve papers met the inclusion criteria. The creative methods used included songs, poems, drawings, photograph elicitation, drama performance, visualisations, social media, photography, prototype development, cultural animation, card sorting and persona development. Analysis identified four limitations and five strengths to the creative approaches. Limitations included the time and resource intensive nature of creative PPI, the lack of generalisation to wider populations and ethical issues. External factors, such as the lack of infrastructure to support creative PPI, also affected their implementation. Strengths included the disruption of power hierarchies and the creation of a safe space for people to express mundane or “taboo” topics. Creative methods are also engaging, inclusive of people who struggle to participate in traditional PPI and can also be cost and time efficient.

‘Creative PPI’ is an umbrella term encapsulating many different methods of engagement and there are strengths and limitations to each. The choice of which should be determined by the aims and requirements of the research, as well as the characteristics of the PPI group and practical limitations. Creative PPI can be advantageous over more traditional methods, however a hybrid approach could be considered to reap the benefits of both. Creative PPI methods are not widely used; however, this could change over time as PPI becomes embedded even more into research.

Plain English Summary

It is important that patients and public are included in the research process from initial brainstorming, through design to delivery. This is known as public and patient involvement (PPI). Their input means that research closely aligns with their wants and needs. Traditionally to get this input, interviews and group discussions are held, but this can exclude people who find these activities non-engaging or inaccessible, for example those with language challenges, learning disabilities or memory issues. Creative methods of PPI can overcome this. This is a broad term describing different (non-traditional) ways of engaging patients and public in research, such as through the use or art, animation or performance. This review investigated the reasons why creative approaches to PPI could be difficult (limitations) or helpful (strengths) in health and social care research. After searching 5 online databases, 12 studies were included in the review. PPI groups included adults, children and people with language and memory impairments. Creative methods included songs, poems, drawings, the use of photos and drama, visualisations, Facebook, creating prototypes, personas and card sorting. Limitations included the time, cost and effort associated with creative methods, the lack of application to other populations, ethical issues and buy-in from the wider research community. Strengths included the feeling of equality between academics and the public, creation of a safe space for people to express themselves, inclusivity, and that creative PPI can be cost and time efficient. Overall, this review suggests that creative PPI is worthwhile, however each method has its own strengths and limitations and the choice of which will depend on the research project, PPI group characteristics and other practical limitations, such as time and financial constraints.

Peer Review reports

Introduction

Patient and public involvement (PPI) is the term used to describe the partnership between patients (including caregivers, potential patients, healthcare users etc.) or the public (a community member with no known interest in the topic) with researchers. It describes research that is done “‘with’ or ‘by’ the public, rather than ‘to,’ ‘about’ or ‘for’ them” [ 1 ]. In 2009, it became a legislative requirement for certain health and social care organisations to include patients, families, carers and communities in not only the planning of health and social care services, but the commissioning, delivery and evaluation of them too [ 2 ]. For example, funding applications for the National Institute of Health and Care Research (NIHR), a UK funding body, mandates a demonstration of how researchers plan to include patients/service users, the public and carers at each stage of the project [ 3 ]. However, this should not simply be a tokenistic, tick-box exercise. PPI should help formulate initial ideas and should be an instrumental, continuous part of the research process. Input from PPI can provide unique insights not yet considered and can ensure that research and health services are closely aligned to the needs and requirements of service users PPI also generally makes research more relevant with clearer outcomes and impacts [ 4 ]. Although this review refers to both patients and the public using the umbrella term ‘PPI’, it is important to acknowledge that these are two different groups with different motivations, needs and interests when it comes to health research and service delivery [ 5 ].

Despite continuing recognition of the need of PPI to improve quality of healthcare, researchers have also recognised that there is no ‘one size fits all’ method for involving patients [ 4 ]. Traditionally, PPI methods invite people to take part in interviews or focus groups to facilitate discussion, or surveys and questionnaires. However, these can sometimes be inaccessible or non-engaging for certain populations. For example, someone with communication difficulties may find it difficult to engage in focus groups or interviews. If individuals lack the appropriate skills to interact in these types of scenarios, they cannot take advantage of the participation opportunities it can provide [ 6 ]. Creative methods, however, aim to resolve these issues. These are a relatively new concept whereby researchers use creative methods (e.g., artwork, animations, Lego), to make PPI more accessible and engaging for those whose voices would otherwise go unheard. They ensure that all populations can engage in research, regardless of their background or skills. Seminal work has previously been conducted in this area, which brought to light the use of creative methodologies in research. Leavy (2008) [ 7 ] discussed how traditional interviews had limits on what could be expressed due to their sterile, jargon-filled and formulaic structure, read by only a few specialised academics. It was this that called for more creative approaches, which included narrative enquiry, fiction-based research, poetry, music, dance, art, theatre, film and visual art. These practices, which can be used in any stage of the research cycle, supported greater empathy, self-reflection and longer-lasting learning experiences compared to interviews [ 7 ]. They also pushed traditional academic boundaries, which made the research accessible not only to researchers, but the public too. Leavy explains that there are similarities between arts-based approaches and scientific approaches: both attempts to investigate what it means to be human through exploration, and used together, these complimentary approaches can progress our understanding of the human experience [ 7 ]. Further, it is important to acknowledge the parallels and nuances between creative and inclusive methods of PPI. Although creative methods aim to be inclusive (this should underlie any PPI activity, whether creative or not), they do not incorporate all types of accessible, inclusive methodologies e.g., using sign language for people with hearing impairments or audio recordings for people who cannot read. Given that there was not enough scope to include an evaluation of all possible inclusive methodologies, this review will focus on creative methods of PPI only.

We aimed to conduct a qualitative systematic review to highlight the strengths of creative PPI in health and social care research, as well as the limitations, which might act as a barrier to their implementation. A qualitative systematic review “brings together research on a topic, systematically searching for research evidence from primary qualitative studies and drawing the findings together” [ 8 ]. This review can then advise researchers of the best practices when designing PPI.

Public involvement

The PHIRST-LIGHT Public Advisory Group (PAG) consists of a team of experienced public contributors with a diverse range of characteristics from across the UK. The PAG was involved in the initial question setting and study design for this review.

Search strategy

For the purpose of this review, the JBI approach for conducting qualitative systematic reviews was followed [ 9 ]. The search terms were (“creativ*” OR “innovat*” OR “authentic” OR “original” OR “inclu*”) AND (“public and patient involvement” OR “patient and public involvement” OR “public and patient involvement and engagement” OR “patient and public involvement and engagement” OR “PPI” OR “PPIE” OR “co-produc*” OR “co-creat*” OR “co-design*” OR “cooperat*” OR “co-operat*”). This search string was modified according to the requirements of each database. Papers were filtered by title, abstract and keywords (see Additional file 1 for search strings). The databases searched included Web of Science (WoS), PubMed, ASSIA and CINAHL. The Cochrane Library was also searched to identify relevant reviews which could lead to the identification of primary research. The search was conducted on 14/04/23. As our aim was to report on the use of creative PPI in research, rather than more generic public engagement, we used electronic databases of scholarly peer-reviewed literature, which represent a wide range of recognised databases. These identified studies published in general international journals (WoS, PubMed), those in social sciences journals (ASSIA), those in nursing and allied health journals (CINAHL), and trials of interventions (Cochrane Library).

Inclusion criteria

Only full-text, English language, primary research papers from 2009 to 2023 were included. This was the chosen timeframe as in 2009 the Health and Social Reform Act made it mandatory for certain Health and Social Care organisations to involve the public and patients in planning, delivering, and evaluating services [ 2 ]. Only creative methods of PPI were accepted, rather than traditional methods, such as interviews or focus groups. For the purposes of this paper, creative PPI included creative art or arts-based approaches (e.g., e.g. stories, songs, drama, drawing, painting, poetry, photography) to enhance engagement. Titles were related to health and social care and the creative PPI was used to engage with people as research advisors, not as study participants. Meta-analyses, conference abstracts, book chapters, commentaries and reviews were excluded. There were no limits concerning study location or the demographic characteristics of the PPI groups. Only qualitative data were accepted.

Quality appraisal

Quality appraisal using the Critical Appraisal Skills Programme (CASP) checklist [ 10 ] was conducted by the primary authors (ORP and CH). This was done independently, and discrepancies were discussed and resolved. If a consensus could not be reached, a third independent reviewer was consulted (JRM). The full list of quality appraisal questions can be found in Additional file 2 .

Data extraction

ORP extracted the study characteristics and a subset of these were checked by CH. Discrepancies were discussed and amendments made. Extracted data included author, title, location, year of publication, year study was carried out, research question/aim, creative methods used, number of participants, mean age, gender, ethnicity of participants, setting, limitations and strengths of creative PPI and main findings.

Data analysis

The included studies were analysed using inductive thematic analysis [ 11 ], where themes were determined by the data. The familiarisation stage took place during full-text reading of the included articles. Anything identified as a strength or limitation to creative PPI methods was extracted verbatim as an initial code and inputted into the data extraction Excel sheet. Similar codes were sorted into broader themes, either under ‘strengths’ or ‘limitations’ and reviewed. Themes were then assigned a name according to the codes.

The search yielded 9978 titles across the 5 databases: Web of Science (1480 results), PubMed (94 results), ASSIA (2454 results), CINAHL (5948 results) and Cochrane Library (2 results), resulting in 8553 different studies after deduplication. ORP and CH independently screened their titles and abstracts, excluding those that did not meet the criteria. After assessment, 12 studies were included (see Fig.  1 ).

PRISMA flowchart of the study selection process

Study characteristics

The included studies were published between 2018 and 2022. Seven were conducted in the UK [ 12 , 14 , 15 , 17 , 18 , 19 , 23 ], two in Canada [ 21 , 22 ], one in Australia [ 13 ], one in Norway [ 16 ] and one in Ireland [ 20 ]. The PPI activities occurred across various settings, including a school [ 12 ], social club [ 12 ], hospital [ 17 ], university [ 22 ], theatre [ 19 ], hotel [ 20 ], or online [ 15 , 21 ], however this information was omitted in 5 studies [ 13 , 14 , 16 , 18 , 23 ]. The number of people attending the PPI sessions varied, ranging from 6 to 289, however the majority (ten studies) had less than 70 participants [ 13 , 14 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 ]. Seven studies did not provide information on the age or gender of the PPI groups. Of those that did, ages ranged from 8 to 76 and were mostly female. The ethnicities of the PPI group members were also rarely recorded (see Additional file 3 for data extraction table).

Types of creative methods

The type of creative methods used to engage the PPI groups were varied. These included songs, poems, drawings, photograph elicitation, drama performance, visualisations, Facebook, photography, prototype development, cultural animation, card sorting and creating personas (see Table  1 ). These were sometimes accompanied by traditional methods of PPI such as interviews and focus group discussions.

The 12 included studies were all deemed to be of good methodological quality, with scores ranging from 6/10 to 10/10 with the CASP critical appraisal tool [ 10 ] (Table  2 ).

Thematic analysis

Analysis identified four limitations and five strengths to creative PPI (see Fig.  2 ). Limitations included the time and resource intensity of creative PPI methods, its lack of generalisation, ethical issues and external factors. Strengths included the disruption of power hierarchies, the engaging and inclusive nature of the methods and their long-term cost and time efficiency. Creative PPI methods also allowed mundane and “taboo” topics to be discussed within a safe space.

Theme map of strengths and limitations

Limitations of creative PPI

Creative ppi methods are time and resource intensive.

The time and resource intensive nature of creative PPI methods is a limitation, most notably for the persona-scenario methodology. Valaitis et al. [ 22 ] used 14 persona-scenario workshops with 70 participants to co-design a healthcare intervention, which aimed to promote optimal aging in Canada. Using the persona method, pairs composed of patients, healthcare providers, community service providers and volunteers developed a fictional character which they believed represented an ‘end-user’ of the healthcare intervention. Due to the depth and richness of the data produced the authors reported that it was time consuming to analyse. Further, they commented that the amount of information was difficult to disseminate to scientific leads and present at team meetings. Additionally, to ensure the production of high-quality data, to probe for details and lead group discussion there was a need for highly skilled facilitators. The resource intensive nature of the creative co-production was also noted in a study using the persona scenario and creative worksheets to develop a prototype decision support tool for individuals with malignant pleural effusion [ 17 ]. With approximately 50 people, this was also likely to yield a high volume of data to consider.

To prepare materials for populations who cannot engage in traditional methods of PPI was also timely. Kearns et al. [ 18 ] developed a feedback questionnaire for people with aphasia to evaluate ICT-delivered rehabilitation. To ensure people could participate effectively, the resources used during the workshops, such as PowerPoints, online images and photographs, had to be aphasia-accessible, which was labour and time intensive. The author warned that this time commitment should not be underestimated.

There are further practical limitations to implementing creative PPI, such as the costs of materials for activities as well as hiring a space for workshops. For example, the included studies in this review utilised pens, paper, worksheets, laptops, arts and craft supplies and magazines and took place in venues such as universities, a social club, and a hotel. Further, although not limited to creative PPI methods exclusively but rather most studies involving the public, a financial incentive was often offered for participation, as well as food, parking, transport and accommodation [ 21 , 22 ].

Creative PPI lacks generalisation

Another barrier to the use of creative PPI methods in health and social care research was the individual nature of its output. Those who participate, usually small in number, produce unique creative outputs specific to their own experiences, opinions and location. Craven et al. [ 13 ], used arts-based visualisations to develop a toolbox for adults with mental health difficulties. They commented, “such an approach might still not be worthwhile”, as the visualisations were individualised and highly personal. This indicates that the output may fail to meet the needs of its end-users. Further, these creative PPI groups were based in certain geographical regions such as Stoke-on-Trent [ 19 ] Sheffield [ 23 ], South Wales [ 12 ] or Ireland [ 20 ], which limits the extent the findings can be applied to wider populations, even within the same area due to individual nuances. Further, the study by Galler et al. [ 16 ], is specific to the Norwegian context and even then, maybe only a sub-group of the Norwegian population as the sample used was of higher socioeconomic status.

However, Grindell et al. [ 17 ], who used persona scenarios, creative worksheets and prototype development, pointed out that the purpose of this type of research is to improve a certain place, rather than apply findings across other populations and locations. Individualised output may, therefore, only be a limitation to research wanting to conduct PPI on a large scale.

If, however, greater generalisation within PPI is deemed necessary, then social media may offer a resolution. Fedorowicz et al. [ 15 ], used Facebook to gain feedback from the public on the use of video-recording methodology for an upcoming project. This had the benefit of including a more diverse range of people (289 people joined the closed group), who were spread geographically around the UK, as well as seven people from overseas.

Creative PPI has ethical issues

As with other research, ethical issues must be taken into consideration. Due to the nature of creative approaches, as well as the personal effort put into them, people often want to be recognised for their work. However, this compromises principles so heavily instilled in research such as anonymity and confidentiality. With the aim of exploring issues related to health and well-being in a town in South Wales, Byrne et al. [ 12 ], asked year 4/5 and year 10 pupils to create poems, songs, drawings and photographs. Community members also created a performance, mainly of monologues, to explore how poverty and inequalities are dealt with. Byrne noted the risks of these arts-based approaches, that being the possibility of over-disclosure and consequent emotional distress, as well as people’s desire to be named for their work. On one hand, the anonymity reduces the sense of ownership of the output as it does not portray a particular individual’s lived experience anymore. On the other hand, however, it could promote a more honest account of lived experience. Supporting this, Webber et al. [ 23 ], who used the persona method to co-design a back pain educational resource prototype, claimed that the anonymity provided by this creative technique allowed individuals to externalise and anonymise their own personal experience, thus creating a more authentic and genuine resource for future users. This implies that anonymity can be both a limitation and strength here.

The use of creative PPI methods is impeded by external factors

Despite the above limitations influencing the implementation of creative PPI techniques, perhaps the most influential is that creative methodologies are simply not mainstream [ 19 ]. This could be linked to the issues above, like time and resource intensity, generalisation and ethical issues but it is also likely to involve more systemic factors within the research community. Micsinszki et al. [ 21 ], who co-designed a hub for the health and well-being of vulnerable populations, commented that there is insufficient infrastructure to conduct meaningful co-design as well as a dominant medical model. Through a more holistic lens, there are “sociopolitical environments that privilege individualism over collectivism, self-sufficiency over collaboration, and scientific expertise over other ways of knowing based on lived experience” [ 21 ]. This, it could be suggested, renders creative co-design methodologies, which are based on the foundations of collectivism, collaboration and imagination an invalid technique in the research field, which is heavily dominated by more scientific methods offering reproducibility, objectivity and reliability.

Although we acknowledge that creative PPI techniques are not always appropriate, it may be that their main limitation is the lack of awareness of these methods or lack of willingness to use them. Further, there is always the risk that PPI, despite being a mandatory part of research, is used in a tokenistic or tick-box fashion [ 20 ], without considering the contribution that meaningful PPI could make to enhancing the research. It may be that PPI, let alone creative PPI, is not at the forefront of researchers’ minds when planning research.

Strengths of creative PPI

Creative ppi disrupts power hierarchies.

One of the main strengths of creative PPI techniques, cited most frequently in the included literature, was that they disrupt traditional power hierarchies [ 12 , 13 , 17 , 19 , 23 ]. For example, the use of theatre performance blurred the lines between professional and lay roles between the community and policy makers [ 12 ]. Individuals created a monologue to portray how poverty and inequality impact daily life and presented this to representatives of the National Assembly of Wales, Welsh Government, the Local Authority, Arts Council and Westminster. Byrne et al. [ 12 ], states how this medium allowed the community to engage with the people who make decisions about their lives in an environment of respect and understanding, where the hierarchies are not as visible as in other settings, e.g., political surgeries. Creative PPI methods have also removed traditional power hierarchies between researchers and adolescents. Cook et al. [ 13 ], used arts-based approaches to explore adolescents’ ideas about the “perfect” condom. They utilised the “Life Happens” resource, where adolescents drew and then decorated a person with their thoughts about sexual relationships, not too dissimilar from the persona-scenario method. This was then combined with hypothetical scenarios about sexuality. A condom-mapping exercise was then implemented, where groups shared the characteristics that make a condom “perfect” on large pieces of paper. Cook et al. [ 13 ], noted that usually power imbalances make it difficult to elicit information from adolescents, however these power imbalances were reduced due to the use of creative co-design techniques.

The same reduction in power hierarchies was noted by Grindell et al. [ 17 ], who used the person-scenario method and creative worksheets with individuals with malignant pleural effusion. This was with the aim of developing a prototype of a decision support tool for patients to help with treatment options. Although this process involved a variety of stakeholders, such as patients, carers and healthcare professionals, creative co-design was cited as a mechanism that worked to reduce power imbalances – a limitation of more traditional methods of research. Creative co-design blurred boundaries between end-users and clinical staff and enabled the sharing of ideas from multiple, valuable perspectives, meaning the prototype was able to suit user needs whilst addressing clinical problems.

Similarly, a specific creative method named cultural animation was also cited to dissolve hierarchies and encourage equal contributions from participants. Within this arts-based approach, Keleman et al. [ 19 ], explored the concept of “good health” with individuals from Stoke-on Trent. Members of the group created art installations using ribbons, buttons, cardboard and straws to depict their idea of a “healthy community”, which was accompanied by a poem. They also created a 3D Facebook page and produced another poem or song addressing the government to communicate their version of a “picture of health”. Public participants said that they found the process empowering, honest, democratic, valuable and practical.

This dissolving of hierarchies and levelling of power is beneficial as it increases the sense of ownership experienced by the creators/producers of the output [ 12 , 17 , 23 ]. This is advantageous as it has been suggested to improve its quality [ 23 ].

Creative PPI allows the unsayable to be said

Creative PPI fosters a safe space for mundane or taboo topics to be shared, which may be difficult to communicate using traditional methods of PPI. For example, the hypothetical nature of condom mapping and persona-scenarios meant that adolescents could discuss a personal topic without fear of discrimination, judgement or personal disclosure [ 13 ]. The safe space allowed a greater volume of ideas to be generated amongst peers where they might not have otherwise. Similarly, Webber et al. [ 23 ], , who used the persona method to co-design the prototype back pain educational resource, also noted how this method creates anonymity whilst allowing people the opportunity to externalise personal experiences, thoughts and feelings. Other creative methods were also used, such as drawing, collaging, role play and creating mood boards. A cardboard cube (labelled a “magic box”) was used to symbolise a physical representation of their final prototype. These creative methods levelled the playing field and made personal experiences accessible in a safe, open environment that fostered trust, as well as understanding from the researchers.

It is not only sensitive subjects that were made easier to articulate through creative PPI. The communication of mundane everyday experiences were also facilitated, which were deemed typically ‘unsayable’. This was specifically given in the context of describing intangible aspects of everyday health and wellbeing [ 11 ]. Graphic designers can also be used to visually represent the outputs of creative PPI. These captured the movement and fluidity of people and well as the relationships between them - things that cannot be spoken but can be depicted [ 21 ].

Creative PPI methods are inclusive

Another strength of creative PPI was that it is inclusive and accessible [ 17 , 19 , 21 ]. The safe space it fosters, as well as the dismantling of hierarchies, welcomed people from a diverse range of backgrounds and provided equal opportunities [ 21 ], especially for those with communication and memory difficulties who might be otherwise excluded from PPI. Kelemen et al. [ 19 ], who used creative methods to explore health and well-being in Stoke-on-Trent, discussed how people from different backgrounds came together and connected, discussed and reached a consensus over a topic which evoked strong emotions, that they all have in common. Individuals said that the techniques used “sets people to open up as they are not overwhelmed by words”. Similarly, creative activities, such as the persona method, have been stated to allow people to express themselves in an inclusive environment using a common language. Kearns et al. [ 18 ], who used aphasia-accessible material to develop a questionnaire with aphasic individuals, described how they felt comfortable in contributing to workshops (although this material was time-consuming to make, see ‘Limitations of creative PPI’ ).

Despite the general inclusivity of creative PPI, it can also be exclusive, particularly if online mediums are used. Fedorowicz et al. [ 15 ], used Facebook to create a PPI group, and although this may rectify previous drawbacks about lack of generalisation of creative methods (as Facebook can reach a greater number of people, globally), it excluded those who are not digitally active or have limited internet access or knowledge of technology. Online methods have other issues too. Maintaining the online group was cited as challenging and the volume of responses required researchers to interact outside of their working hours. Despite this, online methods like Facebook are very accessible for people who are physically disabled.

Creative PPI methods are engaging

The process of creative PPI is typically more engaging and produces more colourful data than traditional methods [ 13 ]. Individuals are permitted and encouraged to explore a creative self [ 19 ], which can lead to the exploration of new ideas and an overall increased enjoyment of the process. This increased engagement is particularly beneficial for younger PPI groups. For example, to involve children in the development of health food products, Galler et al. [ 16 ] asked 9-12-year-olds to take photos of their food and present it to other children in a “show and tell” fashion. They then created a newspaper article describing a new healthy snack. In this creative focus group, children were given lab coats to further their identity as inventors. Galler et al. [ 16 ], notes that the methods were highly engaging and facilitated teamwork and group learning. This collaborative nature of problem-solving was also observed in adults who used personas and creative worksheets to develop the resource for lower back pain [ 23 ]. Dementia patients too have been reported to enjoy the creative and informal approach to idea generation [ 20 ].

The use of cultural animation allowed people to connect with each other in a way that traditional methods do not [ 19 , 21 ]. These connections were held in place by boundary objects, such as ribbons, buttons, fabric and picture frames, which symbolised a shared meaning between people and an exchange of knowledge and emotion. Asking groups to create an art installation using these objects further fostered teamwork and collaboration, both at an individual and collective level. The exploration of a creative self increased energy levels and encouraged productive discussions and problem-solving [ 19 ]. Objects also encouraged a solution-focused approach and permitted people to think beyond their usual everyday scope [ 17 ]. They also allowed facilitators to probe deeper about the greater meanings carried by the object, which acted as a metaphor [ 21 ].

From the researcher’s point of view, co-creative methods gave rise to ideas they might not have initially considered. Valaitis et al. [ 22 ], found that over 40% of the creative outputs were novel ideas brought to light by patients, healthcare providers/community care providers, community service providers and volunteers. One researcher commented, “It [the creative methods] took me on a journey, in a way that when we do other pieces of research it can feel disconnected” [ 23 ]. Another researcher also stated they could not return to the way they used to do research, as they have learnt so much about their own health and community and how they are perceived [ 19 ]. This demonstrates that creative processes not only benefit the project outcomes and the PPI group, but also facilitators and researchers. However, although engaging, creative methods have been criticised for not demonstrating academic rigour [ 17 ]. Moreover, creative PPI may also be exclusive to people who do not like or enjoy creative activities.

Creative PPI methods are cost and time efficient

Creative PPI workshops can often produce output that is visible and tangible. This can save time and money in the long run as the output is either ready to be implemented in a healthcare setting or a first iteration has already been developed. This may also offset the time and costs it takes to implement creative PPI. For example, the prototype of the decision support tool for people with malignant pleural effusion was developed using personas and creative worksheets. The end result was two tangible prototypes to drive the initial idea forward as something to be used in practice [ 17 ]. The use of creative co-design in this case saved clinician time as well as the time it would take to develop this product without the help of its end-users. In the development of this particular prototype, analysis was iterative and informed the next stage of development, which again saved time. The same applies for the feedback questionnaire for the assessment of ICT delivered aphasia rehabilitation. The co-created questionnaire, designed with people with aphasia, was ready to be used in practice [ 18 ]. This suggests that to overcome time and resource barriers to creative PPI, researchers should aim for it to be engaging whilst also producing output.

That useable products are generated during creative workshops signals to participating patients and public members that they have been listened to and their thoughts and opinions acted upon [ 23 ]. For example, the development of the back pain resource based on patient experiences implies that their suggestions were valid and valuable. Further, those who participated in the cultural animation workshop reported that the process visualises change, and that it already feels as though the process of change has started [ 19 ].

The most cost and time efficient method of creative PPI in this review is most likely the use of Facebook to gather feedback on project methodology [ 15 ]. Although there were drawbacks to this, researchers could involve more people from a range of geographical areas at little to no cost. Feedback was instantaneous and no training was required. From the perspective of the PPI group, they could interact however much or little they wish with no time commitment.

This systematic review identified four limitations and five strengths to the use of creative PPI in health and social care research. Creative PPI is time and resource intensive, can raise ethical issues and lacks generalisability. It is also not accepted by the mainstream. These factors may act as barriers to the implementation of creative PPI. However, creative PPI disrupts traditional power hierarchies and creates a safe space for taboo or mundane topics. It is also engaging, inclusive and can be time and cost efficient in the long term.

Something that became apparent during data analysis was that these are not blanket strengths and limitations of creative PPI as a whole. The umbrella term ‘creative PPI’ is broad and encapsulates a wide range of activities, ranging from music and poems to prototype development and persona-scenarios, to more simplistic things like the use of sticky notes and ordering cards. Many different activities can be deemed ‘creative’ and the strengths and limitations of one does not necessarily apply to another. For example, cultural animation takes greater effort to prepare than the use of sticky notes and sorting cards, and the use of Facebook is cheaper and wider reaching than persona development. Researchers should use their discretion and weigh up the benefits and drawbacks of each method to decide on a technique which suits the project. What might be a limitation to creative PPI in one project may not be in another. In some cases, creative PPI may not be suitable at all.

Furthermore, the choice of creative PPI method also depends on the needs and characteristics of the PPI group. Children, adults and people living with dementia or language difficulties all have different engagement needs and capabilities. This indicates that creative PPI is not one size fits all and that the most appropriate method will change depending on the composition of the group. The choice of method will also be determined by the constraints of the research project, namely time, money and the research aim. For example, if there are time constraints, then a method which yields a lot of data and requires a lot of preparation may not be appropriate. If generalisation is important, then an online method is more suitable. Together this indicates that the choice of creative PPI method is highly individualised and dependent on multiple factors.

Although the limitations discussed in this review apply to creative PPI, they are not exclusive to creative PPI. Ethical issues are a consideration within general PPI research, especially when working with more vulnerable populations, such as children or adults living with a disability. It can also be the case that traditional PPI methods lack generalisability, as people who volunteer to be part of such a group are more likely be older, middle class and retired [ 24 ]. Most research is vulnerable to this type of bias, however, it is worth noting that generalisation is not always a goal and research remains valid and meaningful in its absence. Although online methods may somewhat combat issues related to generalisability, these methods still exclude people who do not have access to the internet/technology or who choose not to use it, implying that online PPI methods may not be wholly representative of the general population. Saying this, however, the accessibility of creative PPI techniques differs from person to person, and for some, online mediums may be more accessible (for example for those with a physical disability), and for others, this might be face-to-face. To combat this, a range of methods should be implemented. Planning multiple focus group and interviews for traditional PPI is also time and resource intensive, however the extra resources required to make this creative may be even greater. Although, the rich data provided may be worth the preparation and analysis time, which is also likely to depend on the number of participants and workshop sessions required. PPI, not just creative PPI, often requires the provision of a financial incentive, refreshments, parking and accommodation, which increase costs. These, however, are imperative and non-negotiable, as they increase the accessibility of research, especially to minority and lower-income groups less likely to participate. Adequate funding is also important for co-design studies where repeated engagement is required. One barrier to implementation, which appears to be exclusive to creative methods, however, is that creative methods are not mainstream. This cannot be said for traditional PPI as this is often a mandatory part of research applications.

Regarding the strengths of creative PPI, it could be argued that most appear to be exclusive to creative methodologies. These are inclusive by nature as multiple approaches can be taken to evoke ideas from different populations - approaches that do not necessarily rely on verbal or written communication like interviews and focus groups do. Given the anonymity provided by some creative methods, such as personas, people may be more likely to discuss their personal experiences under the guise of a general end-user, which might be more difficult to maintain when an interviewer is asking an individual questions directly. Additionally, creative methods are by nature more engaging and interactive than traditional methods, although this is a blanket statement and there may be people who find the question-and-answer/group discussion format more engaging. Creative methods have also been cited to eliminate power imbalances which exist in traditional research [ 12 , 13 , 17 , 19 , 23 ]. These imbalances exist between researchers and policy makers and adolescents, adults and the community. Lastly, although this may occur to a greater extent in creative methods like prototype development, it could be suggested that PPI in general – regardless of whether it is creative - is more time and cost efficient in the long-term than not using any PPI to guide or refine the research process. It must be noted that these are observations based on the literature. To be certain these differences exist between creative and traditional methods of PPI, direct empirical evaluation of both should be conducted.

To the best of our knowledge, this is the first review to identify the strengths and limitations to creative PPI, however, similar literature has identified barriers and facilitators to PPI in general. In the context of clinical trials, recruitment difficulties were cited as a barrier, as well as finding public contributors who were free during work/school hours. Trial managers reported finding group dynamics difficult to manage and the academic environment also made some public contributors feel nervous and lacking confidence to speak. Facilitators, however, included the shared ownership of the research – something that has been identified in the current review too. In addition, planning and the provision of knowledge, information and communication were also identified as facilitators [ 25 ]. Other research on the barriers to meaningful PPI in trial oversight committees included trialist confusion or scepticism over the PPI role and the difficulties in finding PPI members who had a basic understanding of research [ 26 ]. However, it could be argued that this is not representative of the average patient or public member. The formality of oversight meetings and the technical language used also acted as a barrier, which may imply that the informal nature of creative methods and its lack of dependency on literacy skills could overcome this. Further, a review of 42 reviews on PPI in health and social care identified financial compensation, resources, training and general support as necessary to conduct PPI, much like in the current review where the resource intensiveness of creative PPI was identified as a limitation. However, others were identified too, such as recruitment and representativeness of public contributors [ 27 ]. Like in the current review, power imbalances were also noted, however this was included as both a barrier and facilitator. Collaboration seemed to diminish hierarchies but not always, as sometimes these imbalances remained between public contributors and healthcare staff, described as a ‘them and us’ culture [ 27 ]. Although these studies compliment the findings of the current review, a direct comparison cannot be made as they do not concern creative methods. However, it does suggest that some strengths and weaknesses are shared between creative and traditional methods of PPI.

Strengths and limitations of this review

Although a general definition of creative PPI exists, it was up to our discretion to decide exactly which activities were deemed as such for this review. For example, we included sorting cards, the use of interactive whiteboards and sticky notes. Other researchers may have a more or less stringent criteria. However, two reviewers were involved in this decision which aids the reliability of the included articles. Further, it may be that some of the strengths and limitations cannot fully be attributed to the creative nature of the PPI process, but rather their co-created nature, however this is hard to disentangle as the included papers involved both these aspects.

During screening, it was difficult to decide whether the article was utilising creative qualitative methodology or creative PPI , as it was often not explicitly labelled as such. Regardless, both approaches involved the public/patients refining a healthcare product/service. This implies that if this review were to be replicated, others may do it differently. This may call for greater standardisation in the reporting of the public’s involvement in research. For example, the NIHR outlines different approaches to PPI, namely “consultation”, “collaboration”, “co-production” and “user-controlled”, which each signify an increased level of public power and influence [ 28 ]. Papers with elements of PPI could use these labels to clarify the extent of public involvement, or even explicitly state that there was no PPI. Further, given our decision to include only scholarly peer-reviewed literature, it is possible that data were missed within the grey literature. Similarly, the literature search will not have identified all papers relating to different types of accessible inclusion. However, the intent of the review was to focus solely on those within the definition of creative.

This review fills a gap in the literature and helps circulate and promote the concept of creative PPI. Each stage of this review, namely screening and quality appraisal, was conducted by two independent reviewers. However, four full texts could not be accessed during the full text reading stage, meaning there are missing data that could have altered or contributed to the findings of this review.

Research recommendations

Given that creative PPI can require effort to prepare, perform and analyse, sufficient time and funding should be allocated in the research protocol to enable meaningful and continuous PPI. This is worthwhile as PPI can significantly change the research output so that it aligns closely with the needs of the group it is to benefit. Researchers should also consider prototype development as a creative PPI activity as this might reduce future time/resource constraints. Shifting from a top-down approach within research to a bottom-up can be advantageous to all stakeholders and can help move creative PPI towards the mainstream. This, however, is the collective responsibility of funding bodies, universities and researchers, as well as committees who approve research bids.

A few of the included studies used creative techniques alongside traditional methods, such as interviews, which could also be used as a hybrid method of PPI, perhaps by researchers who are unfamiliar with creative techniques or to those who wish to reap the benefits of both. Often the characteristics of the PPI group were not included, including age, gender and ethnicity. It would be useful to include such information to assess how representative the PPI group is of the population of interest.

Creative PPI is a relatively novel approach of engaging the public and patients in research and it has both advantages and disadvantages compared to more traditional methods. There are many approaches to implementing creative PPI and the choice of technique will be unique to each piece of research and is reliant on several factors. These include the age and ability of the PPI group as well as the resource limitations of the project. Each method has benefits and drawbacks, which should be considered at the protocol-writing stage. However, given adequate funding, time and planning, creative PPI is a worthwhile and engaging method of generating ideas with end-users of research – ideas which may not be otherwise generated using traditional methods.

Data availability

No datasets were generated or analysed during the current study.

Abbreviations

Critical Appraisal Skills Programme

The Joanna Briggs Institute

National Institute of Health and Care Research

Public Advisory Group

Public and Patient Involvement

Web of Science

National Institute for Health and Care Research. What Is Patient and Public Involvement and Public Engagement? https://www.spcr.nihr.ac.uk/PPI/what-is-patient-and-public-involvement-and-engagement Accessed 01 Sept 2023.

Department of Health. Personal and Public Involvement (PPI) https://www.health-ni.gov.uk/topics/safety-and-quality-standards/personal-and-public-involvement-ppi#:~:text=The Health and Social Care Reform Act (NI) 2009 placed,delivery and evaluation of services . Accessed 01 Sept 2023.

National Institute for Health and Care Research. Policy Research Programme – Guidance for Stage 1 Applications https://www.nihr.ac.uk/documents/policy-research-programme-guidance-for-stage-1-applications-updated/26398 Accessed 01 Sept 2023.

Greenhalgh T, Hinton L, Finlay T, Macfarlane A, Fahy N, Clyde B, Chant A. Frameworks for supporting patient and public involvement in research: systematic review and co-design pilot. Health Expect. 2019. https://doi.org/10.1111/hex.12888

Article   PubMed   PubMed Central   Google Scholar  

Street JM, Stafinski T, Lopes E, Menon D. Defining the role of the public in health technology assessment (HTA) and HTA-informed decision-making processes. Int J Technol Assess Health Care. 2020. https://doi.org/10.1017/S0266462320000094

Article   PubMed   Google Scholar  

Morrison C, Dearden A. Beyond tokenistic participation: using representational artefacts to enable meaningful public participation in health service design. Health Policy. 2013. https://doi.org/10.1016/j.healthpol.2013.05.008

Leavy P. Method meets art: arts-Based Research Practice. New York: Guilford; 2020.

Google Scholar  

Seers K. Qualitative systematic reviews: their importance for our understanding of research relevant to pain. Br J Pain. 2015. https://doi.org/10.1177/2049463714549777

Lockwood C, Porritt K, Munn Z, Rittenmeyer L, Salmond S, Bjerrum M, Loveday H, Carrier J, Stannard D. Chapter 2: Systematic reviews of qualitative evidence. Aromataris E, Munn Z, editors. JBI Manual for Evidence Synthesis JBI. 2020. https://synthesismanual.jbi.global . https://doi.org/10.46658/JBIMES-20-03

CASP. CASP Checklists https://casp-uk.net/images/checklist/documents/CASP-Qualitative-Studies-Checklist/CASP-Qualitative-Checklist-2018_fillable_form.pdf (2022).

Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Res Psychol. 2006. https://doi.org/10.1191/1478088706qp063oa

Article   Google Scholar  

Byrne E, Elliott E, Saltus R, Angharad J. The creative turn in evidence for public health: community and arts-based methodologies. J Public Health. 2018. https://doi.org/10.1093/pubmed/fdx151

Cook S, Grozdanovski L, Renda G, Santoso D, Gorkin R, Senior K. Can you design the perfect condom? Engaging young people to inform safe sexual health practice and innovation. Sex Educ. 2022. https://doi.org/10.1080/14681811.2021.1891040

Craven MP, Goodwin R, Rawsthorne M, Butler D, Waddingham P, Brown S, Jamieson M. Try to see it my way: exploring the co-design of visual presentations of wellbeing through a workshop process. Perspect Public Health. 2019. https://doi.org/10.1177/1757913919835231

Fedorowicz S, Riley V, Cowap L, Ellis NJ, Chambers R, Grogan S, Crone D, Cottrell E, Clark-Carter D, Roberts L, Gidlow CJ. Using social media for patient and public involvement and engagement in health research: the process and impact of a closed Facebook group. Health Expect. 2022. https://doi.org/10.1111/hex.13515

Galler M, Myhrer K, Ares G, Varela P. Listening to children voices in early stages of new product development through co-creation – creative focus group and online platform. Food Res Int. 2022. https://doi.org/10.1016/j.foodres.2022.111000

Grindell C, Tod A, Bec R, Wolstenholme D, Bhatnagar R, Sivakumar P, Morley A, Holme J, Lyons J, Ahmed M, Jackson S, Wallace D, Noorzad F, Kamalanathan M, Ahmed L, Evison M. Using creative co-design to develop a decision support tool for people with malignant pleural effusion. BMC Med Inf Decis Mak. 2020. https://doi.org/10.1186/s12911-020-01200-3

Kearns Á, Kelly H, Pitt I. Rating experience of ICT-delivered aphasia rehabilitation: co-design of a feedback questionnaire. Aphasiology. 2020. https://doi.org/10.1080/02687038.2019.1649913

Kelemen M, Surman E, Dikomitis L. Cultural animation in health research: an innovative methodology for patient and public involvement and engagement. Health Expect. 2018. https://doi.org/10.1111/hex.12677

Keogh F, Carney P, O’Shea E. Innovative methods for involving people with dementia and carers in the policymaking process. Health Expect. 2021. https://doi.org/10.1111/hex.13213

Micsinszki SK, Buettgen A, Mulvale G, Moll S, Wyndham-West M, Bruce E, Rogerson K, Murray-Leung L, Fleisig R, Park S, Phoenix M. Creative processes in co-designing a co-design hub: towards system change in health and social services in collaboration with structurally vulnerable populations. Evid Policy. 2022. https://doi.org/10.1332/174426421X16366319768599

Valaitis R, Longaphy J, Ploeg J, Agarwal G, Oliver D, Nair K, Kastner M, Avilla E, Dolovich L. Health TAPESTRY: co-designing interprofessional primary care programs for older adults using the persona-scenario method. BMC Fam Pract. 2019. https://doi.org/10.1186/s12875-019-1013-9

Webber R, Partridge R, Grindell C. The creative co-design of low back pain education resources. Evid Policy. 2022. https://doi.org/10.1332/174426421X16437342906266

National Institute for Health and Care Research. A Researcher’s Guide to Patient and Public Involvement. https://oxfordbrc.nihr.ac.uk/wp-content/uploads/2017/03/A-Researchers-Guide-to-PPI.pdf Accessed 01 Nov 2023.

Selman L, Clement C, Douglas M, Douglas K, Taylor J, Metcalfe C, Lane J, Horwood J. Patient and public involvement in randomised clinical trials: a mixed-methods study of a clinical trials unit to identify good practice, barriers and facilitators. Trials. 2021 https://doi.org/10.1186/s13063-021-05701-y

Coulman K, Nicholson A, Shaw A, Daykin A, Selman L, Macefield R, Shorter G, Cramer H, Sydes M, Gamble C, Pick M, Taylor G, Lane J. Understanding and optimising patient and public involvement in trial oversight: an ethnographic study of eight clinical trials. Trials. 2020. https://doi.org/10.1186/s13063-020-04495-9

Ocloo J, Garfield S, Franklin B, Dawson S. Exploring the theory, barriers and enablers for patient and public involvement across health, social care and patient safety: a systematic review of reviews. Health Res Policy Sys. 2021. https://doi.org/10.1186/s12961-020-00644-3

National Institute for Health and Care Research. Briefing notes for researchers - public involvement in NHS, health and social care research. https://www.nihr.ac.uk/documents/briefing-notes-for-researchers-public-involvement-in-nhs-health-and-social-care-research/27371 Accessed 01 Nov 2023.

Download references

Acknowledgements

With thanks to the PHIRST-LIGHT public advisory group and consortium for their thoughts and contributions to the design of this work.

The research team is supported by a National Institute for Health and Care Research grant (PHIRST-LIGHT Reference NIHR 135190).

Author information

Olivia R. Phillips and Cerian Harries share joint first authorship.

Authors and Affiliations

Nottingham Centre for Public Health and Epidemiology, Lifespan and Population Health, School of Medicine, University of Nottingham, Clinical Sciences Building, City Hospital Campus, Hucknall Road, Nottingham, NG5 1PB, UK

Olivia R. Phillips, Jo Leonardi-Bee, Holly Knight & Joanne R. Morling

National Institute for Health and Care Research (NIHR) PHIRST-LIGHT, Nottingham, UK

Olivia R. Phillips, Cerian Harries, Jo Leonardi-Bee, Holly Knight, Lauren B. Sherar, Veronica Varela-Mato & Joanne R. Morling

School of Sport, Exercise and Health Sciences, Loughborough University, Epinal Way, Loughborough, Leicestershire, LE11 3TU, UK

Cerian Harries, Lauren B. Sherar & Veronica Varela-Mato

Nottingham Centre for Evidence Based Healthcare, School of Medicine, University of Nottingham, Nottingham, UK

Jo Leonardi-Bee

NIHR Nottingham Biomedical Research Centre (BRC), Nottingham University Hospitals NHS Trust, University of Nottingham, Nottingham, NG7 2UH, UK

Joanne R. Morling

You can also search for this author in PubMed   Google Scholar

Contributions

Author contributions: study design: ORP, CH, JRM, JLB, HK, LBS, VVM, literature searching and screening: ORP, CH, JRM, data curation: ORP, CH, analysis: ORP, CH, JRM, manuscript draft: ORP, CH, JRM, Plain English Summary: ORP, manuscript critical review and editing: ORP, CH, JRM, JLB, HK, LBS, VVM.

Corresponding author

Correspondence to Olivia R. Phillips .

Ethics declarations

Ethics approval and consent to participate.

The Ethics Committee of the Faculty of Medicine and Health Sciences, University of Nottingham advised that approval from the ethics committee and consent to participate was not required for systematic review studies.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

40900_2024_580_MOESM1_ESM.docx

Additional file 1: Search strings: Description of data: the search strings and filters used in each of the 5 databases in this review

Additional file 2: Quality appraisal questions: Description of data: CASP quality appraisal questions

40900_2024_580_moesm3_esm.docx.

Additional file 3: Table 1: Description of data: elements of the data extraction table that are not in the main manuscript

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Phillips, O.R., Harries, C., Leonardi-Bee, J. et al. What are the strengths and limitations to utilising creative methods in public and patient involvement in health and social care research? A qualitative systematic review. Res Involv Engagem 10 , 48 (2024). https://doi.org/10.1186/s40900-024-00580-4

Download citation

Received : 28 November 2023

Accepted : 25 April 2024

Published : 13 May 2024

DOI : https://doi.org/10.1186/s40900-024-00580-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Public and patient involvement
• Creative PPI
• Qualitative systematic review

Research Involvement and Engagement

ISSN: 2056-7529

• General enquiries: [email protected]

Advertisement

A systematic literature review of requirements engineering education

• Original Article
• Open access
• Published: 19 May 2022
• Volume 28 , pages 145–175, ( 2023 )

Cite this article

You have full access to this open access article

• Marian Daun   ORCID: orcid.org/0000-0002-9156-9731 1 ,
• Alicia M. Grubb   ORCID: orcid.org/0000-0002-3552-3165 2 ,
• Viktoria Stenkova   ORCID: orcid.org/0000-0002-4936-1873 1 &
• Bastian Tenbergen   ORCID: orcid.org/0000-0002-0145-4800 3  

9287 Accesses

8 Citations

Explore all metrics

Requirements engineering (RE) has established itself as a core software engineering discipline. It is well acknowledged that good RE leads to higher quality software and considerably reduces the risk of failure or budget-overspending of software development projects. It is of vital importance to train future software engineers in RE and educate future requirements engineers to adequately manage requirements in various projects. To this date, there exists no central concept of what RE education shall comprise. To lay a foundation, we report on a systematic literature review of the field and provide a systematic map describing the current state of RE education. Doing so allows us to describe how the educational landscape has changed over the last decade. Results show that only a few established author collaborations exist and that RE education research is predominantly published in venues other than the top RE research venues (i.e., in venues other than the RE conference and journal). Key trends in RE instruction of the past decade include involvement of real or realistic stakeholders, teaching predominantly elicitation as an RE activity, and increasing student factors such as motivation or communication skills. Finally, we discuss open opportunities in RE education, such as training for security requirements and supply chain risk management, as well as developing a pedagogical foundation grounded in evidence of effective instructional approaches.

Similar content being viewed by others

Exploring the factors that influence the career decision of STEM students at a university in South Africa

Global state of the art of teaching life cycle assessment in higher education

Forschungsdesigns für die qualitative Sozialforschung

Avoid common mistakes on your manuscript.

1 Introduction

Requirements engineering (RE) is commonly accepted as the foundation of high-quality software [ 132 ]. Requirements engineering education (REE) must not only deal with teaching students how to specify formal and informal requirements but also how to elicit and negotiate requirements involving multiple sources—particularly human stakeholders. Thus, REE must make students aware of socio-technical challenges and teach human-related aspects, which poses significant challenges for REE in higher education.

Furthermore, students must be adequately prepared to take on industrial challenges [ 172 , 195 ], while incorporating the theoretical concepts underlying RE [ 30 ]. REE is at best an afterthought in many university software engineering curricula [ 179 ], focusing on lecture-style instruction with few or no realistic examples. In many cases [ 64 , 67 ], RE is not instructed in dedicated courses, but instructed as part of a generic software engineering course. The problem with this situation is twofold: on the one hand, graduates only gain a rudimentary understanding of the minimal RE knowledge required by accreditation standards [ 1 ], standard curricula [ 74 , 75 , 90 ], and bodies of knowledge [ 20 ]. On the other hand, the opportunity is lost to give students enough experience to pick the right RE tools for each development project. In consequence, it is left to the industry to adequately train their staff to be effective in RE.

We need a clear understanding of what to teach (i.e., learning objectives), as well as what educational approaches are the most effective, who the learners are, and what learning outcomes to strive for. Herein, we contribute a systematic literature review of the field of Requirements Engineering Education. Our work allows researchers to gain an overview of the current state of the art and provides educators with insights on how to teach which RE technique.

We consider three research goals:

Goal 1: develop a systematic map of the current state of REE research;

Goal 2: report on current practices and their learning outcomes; and

Goal 3: evaluate how REE has changed over the previous decade.

We further elaborate on these goals in Sect.  3 .

This paper is structured as follows. Section  2 discusses meta-studies on the field of REE. Section  3 details our research questions and methodology. In Sects.  4 – 6 , we discuss results pertaining to each of our research goals (goal 1–goal 3). Section  7 concludes this paper.

2 Background and related work

2.1 challenges in mastering requirements engineering.

Requirements Engineering (RE) is a socio-technical, iterative process to elicit, document, and manage the requirements of a system under development [ 56 ]. RE bridges the gap between human users, developers, and managers, i.e., between people with and without software engineering expertise. RE helps to understand what problem needs to be solved by a (software) system. In addition, it helps to discover who needs to be involved in the engineering process (i.e., stakeholders) and how the problem could be solved by exploring trade-offs and alternatives [ 186 ]. RE requires analysis of both the problem space (i.e., context analysis) and solution space (i.e., the intervention). This is accomplished through a variety of requirements discovery or requirements gathering techniques, including eliciting requirements by interviewing stakeholders or by analyzing existing systems, before documenting the requirements in the form of a specification.

For example, interview techniques alone demand careful selection, as stakeholders may respond differently depending on the mode of inquiry. Imagine a focus group for a new mobile app to allow children to self-monitor health symptoms. A focus group consisting of physicians and children might quickly arrive at decisions about the app’s medical goals, but neglect the children’s perspective because in this setting, the children themselves might be too intimidated to contribute. Documentation techniques require similar careful choice. Storyboards, personas, user interface mock-ups, and natural language requirements (constrained or not), are useful to communicate ideas quickly with a broad audience of non-technical stakeholders, but lack precision for safety-related applications. Formal methods are very precise; however, they require substantial technical expertise and are generally unfit for directly communicating design choices and alternative solutions to stakeholders.

Despite excellent work in the field, elicited and documented requirements artifacts are often incomplete, conflict with one another, and/or suffer from other inadequacies [ 55 , 120 ]. The quality of how the RE process is conducted immediately impacts the quality of the requirements, which in turn, impacts the quality of the system under development. The RE process must be iterative and perpetually monitored with regard to elicitation, documentation, and validation, as well as tracing [ 148 ] requirements from their “source” (e.g., stakeholders, but also laws and standards) to their “destination” (e.g., their refinement into more requirements, analysis results, or their implementation into code). These challenges motivate us to investigate the landscape of RE approaches as it relates to education and training.

Mastering Requirements Engineering is not only a monumental task for the learner, but also for the educator [ 39 ]. On one hand, the theory behind concepts, techniques, and ontologies is quite technical and demands a high amount of rote memorization [ 31 ]. On the other hand, most of the RE process is “learning by doing,” i.e., the learners must to experience it for themselves [ 65 ] before being able to appreciate (and with repeated exposure, eventually master) the RE process and develop a “feeling” when certain techniques are preferable over others. This dichotomy requires a carefully calibrated RE curriculum that balances theory instruction and process exposure.

2.2 Studies on the state of requirements engineering education

In a recent REFSQ conference keynote, Footnote 1 Martin Glinz provided a survey spanning the past several decades on RE Education literature. Indeed, over the past 20 years, a series of reports have been published into the state of the art of software engineering education that are more or less concerned with aspects of requirements engineering education. One of the earliest ones by Shaw [ 162 ] came at a time, where software engineering education was mostly done at the graduate level, aiming to prepare future PhD students. Shaw picked up the claim made in [ 183 ], where graduate and postgraduate software engineering education starts too late and should begin at the undergraduate level alongside traditional computer science education. To this end, Shaw identified “forces” impacting the software engineering industry and academia, and derived “aspirations” for higher education in software engineering to strive towards. Shaw took a wider view than RE education alone, and she aspires for software engineering education to include the need for novice software engineers to specialize into roles and sub-fields like requirements engineering, testing, and even safety assessment. Moreover, Shaw suggested that software engineering education takes an experience-based stance to allow the learner to put theory into practice and develop an intuition for the application of techniques.

By 2008 [ 105 ], software engineering curricula became relatively wide-spread at the undergraduate level, and with it came an increased focus on RE education. As pointed out by Regev et al. [ 146 ], undergraduate RE education was slow to address Shaw’s aspirations, due to discrepancies between typical project-based learning in higher education and actual industry experiences. According to Regev et al., academic classroom projects translate poorly to the industry because of their “sterile” nature, which inadequately reflect industrial practices. The authors attributed this discrepancy to the fact that academic projects must be narrowly scoped to be completed within one semester, by a few students who do not have prior knowledge of the application domain. Additionally, instructors must provide the same experiential opportunity regardless of student background and possible arising group conflicts. Regev et al.’s observations are consistent with views previously reported by a series of other authors, including [ 27 , 33 , 54 , 68 , 169 ], and later confirmed with an empirical study by Menon et al. [ 112 ].

Three systematic mapping studies were conducted between 2012 and 2020, which consist of the work by Malik and Zafar [ 105 ], the aforementioned work by Idri, Ouhbi, et al. [ 71 , 135 ], and the work by Cico et al. [ 24 ]. While the mapping studies by Malik and Zafar as well as by Cico et al. take a wide aim on software engineering education at large, the work by Idri, Ouhbi et al, focuses particularly on RE education. Interestingly, Malik and Zafar report that while some of the mapped primary studies are concerned with project-based learning, the vast majority are concerned with educational technology and tools. Moreover, none of the 70 studies mapped by Malik and Zafar could be easily classified into the knowledge area “Requirements Engineering” according to the reference curricula available then (i.e., “Knowledge Area A” in [ 2 ] or “Knowledge Area C” in [ 90 ]). This indicates that REE research was incongruent with reference curricula and software engineering education research largely ignored RE as a topic. The more focused mapping study conducted by Ouhbi et al. [ 71 , 135 ] reveals a similar trend: only 19 out of 79 mapped primary studies mention reference curricula. The vast majority of papers (77%, see [ 135 ]) present solution approaches with mostly graduate or undergraduate students, with only a minority describing some evaluation of existing approaches. Only few primary studies concerned with industrial training or industrial case studies were found (i.e., 16% and 6% or selected studies, respectively). Moreover, while Ouhbi et al. found that 16% of selected primary studies were written with industrial training consultants as co-authors, neither  [ 71 ] nor [ 135 ] report on industry-readiness of learners.

In summary, past studies investigating the state of the art of REE have been conducted and published in loose intervals. As the newest REE-specific study conducted by Ouhbi et al. [ 71 , 135 ] stems from 2012, we expect the field to have evolved in light of the strong evolution of the field driven by new technologies (cf. [ 193 ]). Therefore, in this paper we want to provide an up-to-date investigation of the current state of REE. We investigate how the field has changed since the investigation of Ouhbi et al., and whether needs posed by new technologies have already been considered in REE research. In addition, we derive common practices and provide guidelines for REE synthesized from the found studies, which has not been done so far. Thus, we review educational approaches that foster learning objectives suitable to the requirements-related problem to be instructed.

3 Research method

In this section, we first elaborate on our research goals introduced in Sect.  1 , and introduce the research questions explored in our systematic literature review (SLR). We then describe our SLR methodology in detail, including how we searched for relevant papers, extracted knowledge, and analyzed data.

3.1 Goals and research questions

As mentioned in Sect.  1 , this SLR complements the mapping study by Ouhbi et al. [ 135 ]. Ouhbi et al.’s work mainly investigated the type of contribution, without placing a clear focus on learning outcomes. We, therefore, provide an overview of existing research about the state of REE and its impact on students’ learning outcomes with the study at hand.

We define three overall goals for our study:

Goal 1: Provide a systematic map of the current state of research in requirements engineering education. Such a systematic map helps researchers in relating their own research to the state of the art and educators in selecting existing approaches for application and adaptation to their own needs.

Goal 2: Provide a synthesis of the current practices the studies reported in the systematic map (i.e., goal 1). This helps to identify approaches best suited for specific RE learning outcomes and challenges in teaching RE.

Goal 3: Evaluate how the state of REE has changed over the last decade since the investigation by Ouhbi et al. [ 135 ].

To fulfill these goals, we defined ten detailed research questions, that allow us to assess the state of REE research. The research questions are listed in Table  1 .

To achieve goal 1, our SLR contains a systematic map that adheres to established research questions for systematic maps as defined by Petersen et al. [ 142 ]. These research questions have been adapted to account for research on REE. As commonly done in systematic mapping studies, we are interested in the researchers involved in REE (RQ1-2), the major publications and venues in the area (RQ3-5), and how do authors conduct and describe their research in this area (RQ6-8). In addition, we defined research questions regarding the educational approaches used, learning outcomes addressed, and the RE techniques taught (RQ9-10).

To achieve goal 2, we relate answers from goal 1 with one another. This allows investigating the instructional theories underlying REE, with a focus on learning outcomes. Taking this as a starting point we synthesize the findings, contributions, benefits, and shortcomings of the papers in the so created sets.

To achieve goal 3, we defined the research questions to be investigated on the basis of the research questions used by Ouhbi et al. [ 135 ]. This allows us a fair comparison of our findings—particularly newer publications—with the findings of Ouhbi et al.. Thereby, it can be investigated whether the state of REE research has changed over the last decade.

3.2 Search procedure

The selected search method of an SLR may impact the found results considerably: manual search, database search, and snowball search may result in paper sets with significant disparities [ 21 ]. In order to avoid limiting the scope of investigation to selected venues (like in manual search), or getting “stuck” in local cliques of mutually referencing papers (like in snowball searches), we used a database search to cover the overall spectrum of possible approaches.

In this spirit, we also used broad search terms to lower the risk of missing relevant papers. Our defined search string is as follows:

TITLE-ABS-KEY (“Requirements Engineering” AND “Education”)

For database searches it is common to include synonyms in the search string; however, this was not appropriate in the case of our investigation. We excluded “training” and “learning” from the search string as pilot testing the search string yielded an extra-proportional number of machine learning and artificial intelligence approaches being included in the results, which are beyond the scope of this study. We restrained the string from including the different areas of RE as substitute for the term “requirements engineering.” Doing so would have led to a misrepresentation of the field as many techniques relevant for requirements engineering education are used in other fields. Instead, we wanted to represent what authors believe is requirements engineering education. Thus, we restricted the search to requirements engineering education literature.

In addition, we analyzed the search string using comparison by manual search for selected venues (as suggested as part of the quasi-gold standard, cf. [ 197 ]). Analysis showed high sensitivity of the search string.

We used Scopus for the search because it covers many publishers, including the most common publishers for computer science research (e.g., ACM, IEEE, Elsevier, Springer), and unlike Google Scholar allows filtering non-peer-reviewed publications. The search string was developed based on the literature review’s topic and research questions, as is commonly done in systematic literature reviews [ 89 , 142 ].

3.3 Study selection

The search was conducted by three different researchers who evaluated each paper based on the inclusion and exclusion criteria (see Table  2 ) on their own. We considered papers published at any time until December 31, 2020. Papers were included in the set of relevant papers of the respective literature review if all researchers found the paper relevant and excluded if all found the paper irrelevant. In cases of inconsistent perceptions of the paper’s relevance, the paper was discussed among the researchers until consensus was reached.

Study selection process

Data extraction process

Figure  1 shows the process of step-wise exclusion of studies to derive the final set of included studies. The studies were selected and excluded at different stages.

Automated search using the search string resulted in 671 publications to be considered.

In the first round of exclusion, 391 papers were excluded by Researcher A as they were very obviously of no relevance to the field, were non-peer-reviewed publications, not in English, or for other obvious violations of the exclusion criteria. This left 280 inclusion candidates.

In a second round, two other researchers applied the inclusion and exclusion criteria on the remaining 280 papers individually. The separate application of inclusion and exclusion criteria was chosen to improve the quality of the paper selection process. In cases of differences agreement was reached in later discussions. In this step, Researcher B included 55.2% and excluded 44.8% of the inclusion candidates, while Researcher C included 45.1% and excluded 54.9% of the inclusion candidates.

In this step, the difference in inclusions and exclusions between Researcher B and C were investigated in detail. We identified 105 common inclusions and 59 common papers where both researchers agreed on exclusion. This yielded an inter-rater agreement of 76.92%, \(k=0.5217\) (which is “fair agreement”). As this big difference was surprising, the situation was discussed between all researchers. It was noticed that most differences resulted from papers that were not about RE education in the first place but discussed RE education in the context of SE education or in more general curricula. A close investigation of these papers yielded the understanding that the papers did not provide sufficient detail on the particular aspects of RE education to be included in the study. In total, we excluded 43 REE-related papers that met this criterion. Consequently, after Step 4, 493 papers from the original 671 were excluded, yielding 178 papers as inclusion candidates. Of these, 36 papers (i.e., about 5.4% of the original 671 papers) remained undecided for a last step of conflict resolution.

In the final step, Researcher A investigated the undecided 36 papers, proposed a solution for inclusion and exclusion, and the final decision was reached by discussion among all three researchers. From the remaining 36 undecided papers, ten were included and 26 were excluded.

In summary, we investigated 671 papers, from which we excluded 519 papers. Resulting in the final set of 152 included publications. Footnote 2

3.4 Data extraction

The data extraction process is illustrated in Fig.  2 . To answer research questions RQ1-6, we extracted each paper’s meta-data from Scopus. For RQ7-9, each included paper was read carefully by three different researchers to extract data pertinent to the research questions. For RQ10, we grouped selected studies into common themes for synthesis. We used word-tags pertaining to the content of a study (e.g., “industry-centric,” “motivation,” or “completeness”) and discussed our findings. Where there was disagreement between any two researchers, a third researcher evaluated the paper. The final classification was reached through discussions among all three researchers.

3.5 Quality assessment

Recently, some SLRs assess the quality of included studies (e.g., [ 135 ]), but these assessments lack a common standard. For example, the application of qualitative quality assessment criteria may be seen as difficult and ambiguous particularly when conducted by researchers of diverse backgrounds (e.g., [ 97 ]), and may, therefore, result in the erroneous exclusion of study results from synthesis. In addition, as is the case with our search, SLRs do not require primary data (i.e., papers) to have been published with sufficient transparency and quality for application of further empirical methods. Thus, we follow a commonly suggested quantitative approach to quality assessment by only including publications that have been peer-reviewed. Hence, we elected to have the quality assessment criteria be reflected in the inclusion and exclusion criteria outlined in Table  2 , instead of conducting an additional subjective quality assessment.

3.6 Analysis and classification

In this section, we revisit our research questions RQ1-10 and describe how we applied the classification schemas. Table  3 presents an overview of this information.

For RQ6, we used a commonly accepted classification for research methods provided by Wieringa et al. [ 194 ]. In doing so, we distinguished between evaluation research, proposal of a solution, validation research, philosophical papers, opinion papers, and personal experience papers (see Table  4 ). Each paper was mapped to exactly one category. In some cases, the categorization of papers might not be obvious. For example, it can be difficult to distinguish between a proposal of a solution and validation research because these types differ in terms of completeness and rigor of their evaluation, which may not be fully described. In these cases, the classification was based only on the presentation of the paper. Other evaluation activities that were suggested but not explicitly reported in the paper were not considered. Each paper was then assigned to the category that fit best.

For RQ7, we adapted the scheme proposed by Petersen et al. [ 140 ], which has been reused in other mapping studies (e.g., [ 36 , 52 ]). However, as some papers did not fit well into any of the original categories, we added a category for other contributions. Table  5 lists each contribution type. Each paper was assigned to all categories that apply.

3.7 Validity evaluation

In this section, we discuss aspects of validity according to the classification scheme in [ 141 ], and the measures taken to mitigate these potential threats.

3.7.1 Descriptive validity

Descriptive validity deals with the accurateness and objectivity of an investigation. As threats to descriptive validity are considered more significant in qualitative investigations than in quantitative investigations, we assume that there are no major threats to descriptive validity. We did not use qualitative quality assessment but favored quantitative quality assessment, which supports descriptive validity. Misclassification of papers may have led to threats to descriptive validity for RQ10 in particular. We built our classification to a large extent on existing and accepted classification schemes. We classify papers as intended by the authors (e.g., type of research contribution, educational approach used), which have been substantiated in the peer review process, to avoid threats from misinterpretation. It cannot be completely ruled out that authors and reviewers of one paper might have accepted an erroneous classification. We assume this was rare enough in occurrence to not impact the descriptive validity (i.e., without misrepresenting the field).

3.7.2 Theoretical validity

Theoretical validity concerns whether the research questions can be answered with the study setup. A major threat in this category typically stems from selection bias. To avoid this bias, we defined objective inclusion and exclusion criteria and applied them rigorously. Inclusion and exclusion criteria were applied independently by two different researchers, with a third researcher validating the choices. Also, the classification was done by two researchers independently, again with a third conducting quality assurance. In case conflicts in the inclusion/exclusion or classification of a paper arose between any two researchers, another researcher was involved, and the conflict was solved by discussion among all researchers, switching roles between “classifier” and “validator” in order to help each individual maintain an objective point of view.

3.7.3 Generalizability

Generalizability of the findings deals with the question, whether the set of papers included into the systematic mapping study are representative and do not miss important aspects. Comparison with previous secondary studies on requirements engineering education (see also Sect.  3.8 ) indicates that we did not miss a considerable number of relevant primary studies to be included.

3.7.4 Interpretive validity

Interpretive validity is concerned with the validity of the conclusions drawn. Hence, researcher biases are a considerable threat. To avoid threats to interpretive validity inclusion and exclusion criteria as well as the classification scheme were not applied by one researcher alone. As outlined above, conflicts were resolved by discussion among at least three researchers that investigated the paper independently. This reduces the threat of researcher bias.

3.7.5 Repeatability

To ensure repeatability, we report the search and selection process as well as the inclusion and exclusion criteria in sufficient detail to enable other researchers to verify our work. Moreover, we make our data available online, particularly with regard to RQ10. Additionally, abstaining from applying qualitative exclusion criteria helps improve repeatability. However, it cannot be ruled out that different researchers might have classified some of the papers in some cases into different categories. This is a common threat in systematic mapping studies and systematic literature reviews. Yet, due to the large number of included publications, we are confident that this would not alter the implications of our findings.

3.8 Validity evaluation for goal 3

Regarding goal 3, it is important to ensure that we use common grounds with the study of Ouhbi et al. [ 135 ], since this study serves as a baseline for our comparison of how REE has changed over the past decade.

We identified 36 of the 79 studies selected by Ouhbi et al.. Two of these studies were considered the same contribution in our work (i.e., [ 84 ]) because the two papers were published in the same venue very close to one another. Of the 43 remaining studies reported by Ouhbi et al., we identified 32 that either did not meet our inclusion criteria (e.g., studies with a primary focus on RE education, rather than education at large using RE methods), or meet our exclusion criteria (most commonly studies that are less than four pages long or dealing with RE for engineering education, see Table  2 ). One study was unobtainable to us, but reported in Ouhbi et al. (i.e., [ 139 ], for which, in fact, we were unable to locate any publication record at all). The remaining seven studies identified by Ouhbi et al. were not identified by us using the process described above. These studies are [ 8 , 23 , 42 , 91 , 96 , 199 ] and [ 178 ].

Two of the contributions identified by us are in fact Ouhbi et al.’s work [ 71 , 135 ]. During Step 4 in Sect.  3.3 , we included 50 papers which were published in the time period reported by Ouhbi et al.. Of these, we selected 33 contributions that were not reported by Ouhbi et al.. These papers are [ 3 , 16 , 22 , 28 , 44 , 45 , 49 , 53 , 58 , 59 , 60 , 62 , 83 , 85 , 98 , 115 , 116 , 118 , 122 , 127 , 133 , 134 , 143 , 151 , 152 , 153 , 156 , 166 , 173 , 174 , 175 , 185 , 187 , 188 ] and [ 192 ]. Thus, we have an agreement of 67.19% with the work of Ouhbi et al., which yields a Cohen’s \(\kappa\) of 0.3586 (i.e., fair agreement ) [ 26 ].

When comparing results with Ouhbi et al., search strategy accounts for some of the differences between included studies. We relied on Scopus (as this already covers the established publishers in the field) to search for articles, while Ouhbi et al. used the publishers’ search engines and Google Scholar. We purposefully used a more general search string than Ouhbi et al., as outlined above to investigate what authors believe RE Education shall be concerned with. Additionally, we applied stricter exclusion criteria.

In summary, we found more candidate papers but also excluded more. Like Ouhbi et al., we were interested in metadata about the papers. Yet, they investigated which studies referred to reference curricula, while our investigation focused on educational approaches and learning outcomes regardless of reference curricula, and the change of REE research since Ouhbi et al.’s work. Thus, our work is complementary.

Returning to goal 3, since we have covered sufficient common ground with the work of Ouhbi et al., we can provide valid observations about how the field of REE has evolved over the past decade.

4 Results for goal 1

Automatically generated map of author networks. Red lines indicate connections between authors, who are part of two collaboration groups. The darker the hue, the more co-authored papers (Color figure online)

Fragment of author networks only including those with more than one collaboration

Publications and venues per year. Each bar represents one year, with cumulative counts of publications per year (RQ4) listed at the top of each bar. Bars are sub-divided by type of publication venue (RQ3) to illustrate changes in venue over time

In this section, we present our systematic literature map (goal 1) and explore each research question (RQ1-10) in detail.

4.1 Most active researchers (RQ1)

We begin by exploring who is most involved in REE activities. Table  6 shows the most prolific authors in the area of REE. A total of sixteen authors contributed at least four published papers. Most prolific is Didar Zowghi from the University of Technology Sydney, Australia with nine published papers. As can further be seen, authors regularly involved in REE stem from around the world with a strong focus on Europe. Nine researchers are affiliated with universities from European Union countries: Germany (5), Spain (2), Portugal (1), and Italy (1). Three authors are affiliated with Malaysia, two with Australia, and one with Chile or the United States. Thus, we can see that while 152 articles were selected in our study, the majority of the contributions do not appear to be the primary scientific focus of the publishing scholars (with the exception of the individuals from Table  6 ), as most authors have fewer than two contributions in this field.

4.2 Research networks (RQ2)

Using study metadata, we automatically generated Fig.  3 , which shows the existing networks of authors found in the included studies. This gives a high-level overview of how segmented the efforts are in REE. Rectangles visualize collaborations between individual authors. As can be seen there exists a variety of individual collaborations that are not connected to other authors. Thus, we can assume that the field is rather scattered without collaborations between different author clusters. The coloring indicates the number of collaborations. Most authors participate in only one collaboration (light blue color), the maximum amount of collaborations is four between two authors (dark blue color). To improve readability and further explore existing networks of authors, we isolated networks with more than one collaboration to create a fragment of our map in Fig.  4 . Overall, these findings suggest that most selected studies appear to be separate contributions without a systematic continuation of a research direction. A notable exception is the work by Zowghi, Spoletini, Ferreira, and Bano from recent years, which investigates the use of interviews to practice requirements elicitation [ 13 , 40 , 41 ] and inspections [ 11 ].

4.3 Top venues (RQ3)

Table 7 shows all venues where multiple papers on REE have been published. We found fourteen venues where researchers regularly publish REE research. Yet, there are only five venues where REE seems to be published on a regular basis (i.e., with more than five total publications). The most established venue for REE is the International Workshop on Requirements Engineering Education and Training (REET) with 32 publications. The most established conferences are the IEEE International Conference on Software Engineering Education and Training (CSEE&T, 16) and the IEEE International Requirements Engineering Conference (RE, 10). The most established journal is Requirements Engineering (REJ), yet carries only five publications (of 152 total selected studies). This indicates that so far many early ideas and problem descriptions are elaborated on, with more mature research on REE being rarely addressed in the three most representative venues of requirements engineering research. According to Daneva et al. [ 29 ], these are REJ, RE, and the Working Conference on Requirements Engineering: Foundations for Software Quality (REFSQ), excluding their workshops. Yet, RE and REJ carry only 15 publications (ca. 9.7% of all 152 selected studies), while REFSQ is not represented in this list at all. Moreover, REJ is merely in sixth place (shared with the iStar workshops). In conclusion, REE research seems to be primarily published in education-related venues that are not specific to requirements engineering as well as the REET workshop. We conclude that there may be a missing connection between non-education research in RE and research specific to RE education.

Citation network of articles (dots) being cited by other articles (edges, head pointing to cited article). As can be seen, only six articles are cited four or more times, suggesting no common foundation of RE education literature

4.4 Paper per year (RQ4)

We found an increasing trend of publications over the years. Figure  5 shows the distribution of publications by year and type of venue where a paper was published (see also Sect.  4.3 ). As can be seen, research on REE started slowly in the beginning with only four conference papers between 1988 and 1998, and one journal article. This was followed by a phase from 1999 to 2007, where papers were regularly published, however in small numbers each year, and only in conferences. Since 2007, REE-related workshops have appeared and are in part responsible for the increase in publications reaching a maximum of seventeen publications in 2018. Thus far, 2018 was the year with the largest number of published journal papers. These findings suggest that REE has gained more and more interest over the years and its importance is shown in still increasing publication numbers. More than half of the publications selected in our work were published after the work by Ouhbi et al. [ 135 ]. Ouhbi et al.’s work was conducted in 2012 (almost 10 years ago), which coincided with the beginning of a four-year hiatus of REET. The eighth installment of REET was in 2013 and ninth and tenth installments were in 2018 and 2020, respectively. This in turn coincides with a period of slightly decreased frequency of workshop contributions and contributions at the three top venues for RE-specific research [ 29 , 179 ]: the REFSQ conference, the RE conference, and the Requirements Engineering journal (see also Sect.  4.3 ).

4.5 Top cited publications (RQ5)

We generated a citation network to analyze citation cycles. Figure  6 shows an excerpt of the citation network, i.e., the set of papers that cite other papers from all included studies. Arrow heads point to papers citing another paper (i.e., can be thought of as an “import” relationship). First, it can be seen that only about a third of all selected studies cite any papers within our set of 152 selected papers at all; the two-thirds of papers not citing any other papers have been omitted from Fig.  6 . Second, no paper is cited more than four times (see outgoing arrows in Fig.  6 ). Most papers cite merely one or two other papers and only five papers cites at least as many other papers (see ID 1001, 1005, 1058, 1008, and 1043 in Fig.  6 ). These are typically review papers. For example, the paper with the ID 1001 is the review paper by Ouhbi et al. [ 135 ]. Thus, we can conclude that no considerable citation cycles do exist. This means that neither is there as standard reference for REE accepted by the community. Thus, we found that most (i.e., at least two-thirds of our selected studies) REE research happens in “a vacuum,” without relying heavily on other findings in the field.

4.6 Research methods (RQ6)

We evaluated the papers based on the presented type of research, i.e., its underlying research method. Table  8 (left-hand side) shows the results separated by year. As can be seen the vast majority of papers are either solution proposals or experience reports. In contrast, evaluation research and validation research are only sparsely conducted. This means that while there exists a plethora of approaches aiming at improving REE and a variety of personal experience reports, more thorough empirical investigations of the field either by exploratory evaluation studies or by thoroughly validated solutions are missing. We conclude from this that the maturity of the field must be considered rather low. This is in line with our findings from RQ5, as indications of high overall maturity would be indicated by common, frequently cited references.

4.7 Contributions (RQ7)

For the contributions of the included studies, we mapped the publications according to the classification scheme proposed by Petersen et al. [ 140 ]. Table  8 (right-hand side) shows the results separated by year. Most publications propose a method, followed by tools to be introduced in REE. This is in line with findings by Malik and Zafar [ 105 ] (see also Sect.  2.2 ). In addition, we found a large number of papers classified as “other” . These mostly result from the large number of experience reports, which typically do not propose any kind of contribution in the sense of the categories in [ 140 ]. Nevertheless, we classified them according to their common theme, as shown in Fig.  7 . As can be seen, many papers classified as “other” in Table  8 report on limitations, pitfalls, or constraints, yet without specifying concrete solutions (17 in total). A total of 12 papers are concerned with involving real or realistic stakeholders (e.g., through role playing), while six papers propose a course design (without explicitly proposing it as a solution to a specific problem). Six more papers propose education research case studies and/or examples (often conflating the two terms), while again six studies report on empirical studies with students, surveys, or other types of investigations, yet without validating or evaluating a proposed solution (see Table  8 ). We infer from this that unlike non-education fields of software engineering, REE is fairly diverse, yet centers around proposing specific methods or approaches, or involving specific tools. This is in-line with our finding that most contributions are solution proposals (see RQ6). Although this further indicates low maturity of the field, it also means that a diverse set of contributions and solution avenues exist to teach RE, thereby suggesting a rich (albeit unsystematic) “toolbox” of educational approaches.

“Other” contributions from Table  8

4.8 Keywords (RQ8)

Table 9 shows the ten most frequently used keywords. As can be seen most keywords are basic terms. Beside these keywords, other keywords are used five times or less often. Thus, this indicates that—beside the topic of requirements elicitation—there seems to be no specific area of interest in requirements engineering that education research particularly focuses on. The frequent use of the term “requirements elicitation” indicates that for this specific area of RE there may be a particular interest in how to teach this topic. Yet, other areas of RE may not receive as much attention. This may make it difficult for educators interested in the field to find a solution to an instructional problem they are faced with, without being intimately familiar with the many solution proposals that exist in the field (see RQ6 and RQ7).

4.9 Learners (RQ9)

Figure  8 shows the distribution of the emphasized audience of teaching approaches as stated by the included publications. The vast majority of papers (120) clearly address university students. Of these, three papers consider postgraduate students, 21 focus on graduate students, 44 on undergraduates, and 52 do not further specify the level of the learner. Only 17 papers address teaching industry professionals. Thirteen papers omit the audience (“not mentioned” in Fig.  8 ) or generically speak of “students” (“unknown” in Fig.  8 ). We assume some of these address university education and find this sufficiently obvious that authors do not deem it important to specify this further. One paper places emphasis on RE education at the high-school level and another one investigates RE knowledge in alumni. This seems to show that Shaw’s “aspiration” [ 162 ] was in part answered, as a substantial number of approaches target aspiring software engineers in very early stages (i.e., at the undergraduate level) to instruct role-specific skills related to requirements engineering. Yet, by comparison, industry training is currently not a key focus in REE research.

Type of learners addressed

Topics of interest

4.10 Learning outcomes (RQ10)

To gain insights into what the included studies propose or investigate—and thus on the question what the current state of research in REE deals with from a content-related point of view—we identified broad recurring themes. Figure  9 shows these themes and their frequency.

Teaching requirements engineering activities. Most papers (i.e., 73) are concerned with teaching different RE activities. Recurring activities are elicitation, negotiation, specification, requirements validation, management, and modeling. In addition, specific activities as safety analyses or requirements tracing are concern of some publications.

Teaching soft skills. Forty-nine included studies focus on teaching soft skills when teaching RE. Targeted soft skills are typically closely related to the work profile of a requirements engineer. Papers commonly focus on communication skills, teamwork and collaboration skills, conflict resolution skills, interviewing techniques, or technical writing.

Improving student-related factors. In this category, 32 papers aim at improving the learning of students by increasing student motivation, enthusiasm for the subject matter, coping with overwhelmed students, or aim to improve students’ ability to explore problems and deal with solution uncertainty.

Improving industry readiness. A total of 29 publications aim at improving industry readiness of the students to cope with real RE problems. This is typically done by involving real stakeholders in a course, using or investigating real requirements specifications, or applying industry-realistic examples in the classroom.

Teaching requirements quality. In total 20 papers, focus on improving students’ sensitivity to high-quality requirements. Requirements quality properties mainly include consistency and correctness of requirements and requirements specification documents, but also ambiguity, and completeness.

Raising awareness for integrated RE processes. Although these eleven papers were included as they place particular emphasis on teaching RE, their focus lies on doing so as part of a broader development context, e.g., dealing with real customers’ needs.

Adaptability to professional environments. Nine papers propose specific educational settings to foster professional RE skills. For instance, this includes distributed global settings to mimic spatial separation of teams or teaching computer science students together with students from other disciplines to raise awareness of multidisciplinary issues.

This list shows that teaching requirements engineering activities is only part of what REE is concerned with, as about half of the papers deal with non-core requirements engineering theory.

In summary, the results presented in this section constitute our systematic map, which addresses goal 1. It is notable that since Mary Shaw’s aspiration (i.e., to include more role-specific undergraduate software engineering education, see [ 162 ]) has been answered by the REE community. A vast plethora of approaches have been proposed, especially since 2012 and beyond. Yet, the field suffers from low overall maturity. Most research appears to be solution proposals, without suggesting a continuing research avenue, and without producing a core area of expertise, neither surrounding scholars, nor surrounding methods, nor surrounding specific contributions. Nevertheless, we found that successful requirements engineering instruction encompasses more than theory, i.e., student factors and soft skills, as well as industry-readiness. Therein lie core themes in the papers we have discussed. In the next section, we address goal 2 and discuss the most significant trends pertaining to learning outcomes, as well as the educational approaches to achieve them.

5 Results for goal 2

Next, we explore goal 2 of this paper, which investigates the current practices regarding pedagogical techniques and the learning outcomes they seek to achieve. We initially hoped to distill these practices based on data from validated approaches. However, as can be seen by the results of RQ6 and RQ7 (see Sects.  4.6 and 4.7 , respectively), research contributions are overwhelmingly solution proposals or experience reports, with only 21.7% being evaluation or validation research. While several proposed solutions provide at least minimal quantitative or qualitative evidence as to their efficacy, a systematic replication and investigation of their pedagogical benefits is (unsurprisingly [ 25 , 163 ]) largely missing.

Nevertheless, as can be seen by the results from RQ10 (see Sect.  4.10 ), there are some clear and promising trends. These trends can be summarized into the following topics:

Authenticity and industry-readiness

Teaching RE activities and requirements quality

Student factors and soft skill development

To give further context to our discussion of learning outcomes, we tagged the papers in our mapping based on their educational approach, as explained in Sect.  3.4 shown in Fig.  10 .

Type of educational approach

Publications per year proposing an industry-centric learning outcomes and educational approaches

Studies explicitly instructing RE activities

5.1 Authenticity and industry-readiness

The first trend that we observed is the prominence of work that focuses on industry-readiness and giving students an authentic RE experience. We found that 32 papers (see Fig.  10 ) used an industry-centric educational approach (e.g., by involving external stakeholders from real companies), and 29 papers (see Fig.  9 ) explicitly mention industry-readiness as a learning outcome. Figure  11 shows these studies over time. From this timeline, we can see that this research focus is comparatively young, as half of these approaches have been published in the past 10 years alone.

Further, this trend suggests that the community is moving away from instructor-centric approaches, which focus on rote memorization of theory and individual high-stakes problem solving. Instead, nearly two-thirds (61.84%) of the studies shown in Fig.  10 propose a non-instructor-centric approach to instruct RE. Note that in Fig.  10 , approaches could pertain to more than one category. Nevertheless, we found 94 individual studies. These include the 32 aforementioned industry-centric approaches, as well as student collaboration (29 studies), project-based (13 studies), problem-based (9 studies), and other inquiry-based paradigms (20 studies (e.g., games [ 5 , 50 ] or case studies [ 110 , 182 ]). Of the 32 industry-centric studies, seven studies do so by fostering student collaboration (i.e., [ 27 , 38 , 123 , 125 , 167 , 177 , 180 ]), four do so through project-based instruction (i.e., [ 9 , 16 , 32 , 180 ]), and four through some other form of inquiry-based instruction (i.e., [ 27 , 53 , 168 , 177 ]). Within these 32 studies, two paradigms around stakeholder involvement can be differentiated: on one hand, approaches involve external stakeholders in realistic projects (e.g., [ 48 , 61 , 103 , 137 , 138 ]); while on the other hand, approaches involve the instructor (or some other non-industry representative, e.g., [ 47 , 177 ]) to engage in role-playing to create an industry-realistic project experience (e.g., [ 98 , 125 , 130 , 168 , 176 ]). In both of these paradigms, stakeholders serve as a partner to help students with requirements activities to some degree (see Sect.  5.3 ).

However, achieving industry authenticity is not necessarily done by involving real or mimicked stakeholders alone. Other approaches include using industry-realistic case examples (e.g., [ 30 , 31 , 32 ]) or using geographically distributed teams working on the same project (e.g., [ 9 , 16 , 149 , 152 , 198 ]). These approaches are interesting because they address soft skills in addition to industry-readiness (also see Sect.  5.3 ).

Evidence on the effectiveness of improving industry-authenticity relies on experience reports (e.g., [ 98 , 138 , 171 ]). Quantitative data are mostly available by means of students’ course evaluations (see [ 147 ]), or exam results (see [ 32 ]). Perhaps this is because student evaluations, assignment sheets, and exams are the typical means of assessment of student performance; however, another way of assessing learning outcomes is to measure student performance against industry needs, such as through a graduate alumni surveys of preparedness. This was done in [ 184 ], where researchers found that perceived usefulness of instructed documentation formats (e.g., use cases or glossaries) seem to increase with graduates’ work service and project experience.

In summary, we consider it a positive development that educational approaches have taken a keen focus on improving students’ industry-readiness and are moving away from rote memorization in favor of formative learning. However, many of these approaches aim at doing so without consideration of industry needs. Few approaches report on providing requirements engineering training to practitioners, with the notable exception of Morales-Ramirez et. al’s work in [ 123 ]. For both, more studies providing evidence are desirable.

5.2 Teaching RE activities and requirements quality

Next, we investigate trends in selecting topics to include in RE training. About half of the investigated studies focus on specific RE activities (73 studies in total, see Fig.  9 ). We visualize our selected categories of this breakdown in Fig.  12 . Among the most common are elicitation (39.72% of studies, e.g., [ 53 , 69 , 86 , 102 , 150 , 174 , 176 ]), modeling (28.77% of studies, which includes “modeling syntax” [ 17 , 34 , 66 ] and “process modeling” [ 107 , 159 ]). Eleven studies (15.07%) explicitly aim to instruct the whole RE process, while validation, verification, or quality assurance are a topic in only eight studies (10.96%, e.g., [ 41 , 69 ]), and management in only five studies (6.85%, including “time management,” “project management,” or “process management,” i.e., [ 14 , 38 , 100 , 114 , 121 ]). Surprisingly rarely do studies investigate more rigorous RE activities. For example, we found only three studies that look at security requirements engineering (from a process perspective, not quality perspective, i.e., [ 110 , 111 , 143 ]), three studies investigating formal methods (i.e., [ 44 , 188 , 191 ]), and two studies on requirements tracing (i.e., [ 19 , 116 ]). Safety requirements engineering was merely the elementary instructional focus of a single study (i.e., [ 180 ]).

Looking closer at the most investigated RE activity, elicitation, the vast majority of included studies (i.e., 19 out of 29) do so by means of using interviews as the predominant technique (see Table  10 ). Of the remaining ten papers, six studies did not specifically emphasize any particular elicitation technique, while three used specific technique (e.g., workshops) and one used various elicitation techniques including interviews.

Additionally, we considered which instructional approaches were used in elicitation activities. As shown in Table  11 , papers that used role playing were the most predominant approach. Other approaches included using real stakeholders, games, and tools. The contribution by Sedelmaier and Landes [ 161 ] was particularly noteworthy in this respect, not only because it is one of the few studies that employ a specific pedagogical paradigm (i.e., “competence-oriented didactics” in Table  11 ). While two papers did not specify any approach, we did not find any papers that studied the use of competitor or market analyses.

It is also noteworthy that some requirements engineering activities that could be considered essential (e.g., negotiation or prioritization) are not specifically targeted by RE education at all, as shown in Fig.  12 . While these activities may be subsumed in those studies targeting the “whole process,” the respective authors did not explicitly list all activities they included.

We observed a mismatch between teaching requirements quality properties (i.e., completeness, consistency, and traceability) and work advocating for a project-based learning environment. We found a total of 20 studies that explicitly mention teaching students to be sensitive to requirements quality. Of these, only two also pertain to those included in Sect.  5.1 (i.e., [ 58 , 180 ]). Figure  13 shows the breakdown of quality requirements publications, where studies may target more than one quality. Of the remaining 18 studies (see Fig.  13 ), the predominant focus is on requirements “consistency” (i.e., [ 49 , 62 , 69 , 166 , 174 , 188 , 190 , 191 ]). “Correctness” is targeted by five studies (i.e., [ 7 , 49 , 58 , 87 , 188 ]); however in doing so, studies often conflate formal provability of requirements and the sense of adequacy with regard to stakeholder needs (see [ 55 , 57 ] for a discussion of the difference). Only one included study explicitly mentions “adequacy,” but this is specifically in the context of security requirements [ 131 ]. Similarly, “completeness” is only explicitly targeted by Westphal in [ 191 ], albeit in the context of formal modeling of requirements. Four studies do not limit the educational focus on individual qualities, but rather mention “quality as a whole.” These studies are [ 11 , 41 , 53 ] and [ 59 ]. A total of six studies implicitly target requirements quality (“others” in Fig.  13 ). While they explicitly state the need to instruct sensitivity to high-quality requirements, the educational approach therein is not specifically targeted to requirements artifacts, but rather activities to improve quality in requirements. In this sense, “complexity” or “abstraction” are mentioned by three studies (i.e., [ 37 , 93 , 114 ]). The remaining studies each mention one quality property: traceability [ 60 ], ambiguity [ 174 ], and understandability [ 175 ].

As outlined in Sect.  4.6 , many of the studies we surveyed are “solution proposals” (see Table  8 ). Of these, most advocate for project-centric collaborative approaches and a minority advocate for instructor-centric, theory-heavy instruction. This is consistent with our earlier finding (see Sect.  5.1 ) that most approaches advocating industry-readiness do so in a project-based setting, requiring students to experience the whole RE process, from elicitation to documentation, to management. However, only ten of 73 studies mention a specific RE activity in an industry-realistic setting as opposed to targeting the whole RE process or not mentioning RE activities at all (i.e., [ 28 , 32 , 48 , 50 , 100 , 106 , 126 , 128 , 180 , 184 ]).

In summary, we found very little overlap between studies mentioned in Sect.  5.1 with studies aiming to teach RE activities and a focus on requirements quality. This seems to suggest that by increasing industry-readiness comes at the expense of teaching specific RE activities and requirements quality. However, we do not believe this to be the case. Many of the studies mentioned in Sect.  5.1 aim to convey a feeling of the intricacies of the whole RE process to students, not just individual activities. Moreover, it is the whole process experience which highlights issues such as completeness of requirements through elicitation and documentation, adequacy/correctness of requirements through validation and verification, requirements consistency and the like. However, while not ignored, it seems that these intricacies are at best conveyed implicitly. We did not find any study explicitly investigating how industry-readiness may also foster requirements activity proficiency and sensitivity to requirements quality, and recommend this as an area for future research.

Studies explicitly instructing requirements quality

5.3 Student factors and soft skill development

A third theme we found in our analysis is that many of the included studies emphasize student factors and soft skill development. This means that the focus is on “how” to conduct requirements activities effectively, thereby increasing soft-skills such as communication (as in [ 177 ]) or customer-orientation (as in [ 174 ]) rather than solely teaching “that” requirements elicitation is necessary. We tagged the publications emphasizing soft skills and visualize our results in Fig.  14 . The most frequently addressed soft skills are teamwork, collaboration, or social interaction (30.61% of studies pertaining to soft skills, e.g., [ 16 , 31 , 32 , 66 , 118 , 138 , 147 , 153 , 180 , 187 ]), interviewing skills (24.49%, e.g., [ 12 , 13 , 28 , 35 , 156 , 176 , 198 ]), customer interaction or client-orientation (16.33%, e.g., [ 69 , 118 , 125 , 127 , 157 , 174 ]), and communication (also 16.33% of soft skill studies, e.g., [ 27 , 149 , 152 , 153 , 159 , 168 , 177 ]). Only two studies focused on agile development as a soft skill (namely, [ 67 , 102 ]); however, most studies focusing on collaboration and communication applied a project-centric and industry-realistic (see Sect.  5.1 ) learning environment in conjunction with agile methods. Like in Sect.  5.2 , the overlap to those studies in Sect.  5.1 is fairly low, as only four studies appear to explicitly involve authentic industrial settings to improve students’ soft skills, i.e.,  [ 28 , 67 , 118 , 180 ].

However, as introduced above, most of the contributions whose primary focus is on soft skill development do so in a collaborative and/or project-based setting. Of the studies that apply a formative instruction paradigm (i.e., project-based, problem-based, and/or collaboration-based instruction in Fig.  10 ) and of the 49 studies that aim to improve students’ soft skills (see Fig.  9 ) as their primary learning outcome, the overlap consists of 19 studies (i.e., 30.65% of formative methods studies from Fig.  10 ). These studies mainly focus on communication, interviewing, and team collaboration in project-based settings.

The overlap between the same formative instructional approaches from Fig.  10 and studies specifically aiming to improve student factors is much lower. We identified a total of 12 studies that employ a project-, problem-, or collaboration-based instructional method in combination with the explicit aim of improving student factors. These factors include enthusiasm and motivation (e.g.,  [ 31 , 98 , 108 ]), comprehension and understanding (e.g.,  [ 103 , 117 , 181 ]), learning and retention (e.g.,  [ 16 ]), and introspection (e.g.,  [ 43 , 94 ]), which is a 19.35% overlap with formative approaches.

In total, we found 32 studies that propose a diverse set of pedagogical strategies to improve student factors, which we show in Fig.  15 (note, studies may pertain to more than one student factor). The most commonly addressed student factors are motivation/enthusiasm (11 studies in total i.e.,  [ 5 , 31 , 32 , 34 , 50 , 51 , 94 , 98 , 117 , 138 , 180 ]), understanding/comprehension (8 studies, i.e., [ 44 , 63 , 103 , 117 , 159 , 170 , 181 , 188 ]), retention/learning (7 studies, i.e.,  [ 6 , 15 , 16 , 34 , 66 , 122 , 187 ]), and engagement/ interest (also 7 studies, i.e.,  [ 5 , 31 , 32 , 94 , 117 , 138 , 180 ]). The remaining six studies target a diverse, yet more abstract set of student factors, i.e., “combating students being overwhelmed” [ 190 ], “effort and aggravation” [ 58 ], “review effectiveness” [ 134 ], “acceptance of uncertainty” [ 14 ], “process competency” [ 129 ], and “introspection” into the validation process (i.e.,  [ 11 , 13 , 41 ], which for the purpose of this discussion, we consider one contribution).

Besides formative and industry-centric approaches as outlined above, studies aiming to improve student factors and soft skills propose a diverse set of strategies to fulfill their aim. In particular, the use of games or gamification (e.g., [ 5 , 6 , 34 , 50 , 99 , 156 , 170 , 187 , 196 ]), engaging case examples (e.g., [ 3 , 9 , 30 ]), or using low-stakes assignments (e.g., [ 18 , 28 , 196 ]) are promising approaches that emerge from the literature.

In summary, while proposals for teaching specific RE activities are separate from improving student factors and soft skills (see Sect.  5.2 ), we found that student factors and soft skills are a tangential learning outcome of this work. By comparison, industry-authenticity specifically adopts external stakeholders or role playing as an instructional mechanism in order to improve student motivation and enthusiasm (see Sect.  5.1 ). The REE literature recognizes that soft skills are critical for students’ success in future employment and that student factors are critical in improving student success in requirements engineering. Nevertheless, more work on how to successfully and holistically integrate theory instruction and student success is desirable.

Studies explicitly improving soft skills

Studies explicitly improving student factors

6 Results for goal 3

In this section, we address goal 3 by evaluating how REE has changed over the last decade. To accomplish this goal, we compare our findings to those from Ouhbi et al. [ 71 , 135 ]. In Sect.  3.8 , we compared our literature search methodology and results to those from Ouhbi et al. to contextualize our analysis. In the following subsections, we compare to Ouhbi et al.’s “implications and advice for instructors” and what REE research has contributed since the study concluded. Finally, we identify additional gaps in current REE literature and offer our own conclusions.

6.1 How literature addresses Ouhbi et al.’s “implications”

Following a detailed map of the REE field, Ouhbi et al. provide advice to REE instructors in the form of seven implications derived from their selected studies. In this section, we discuss these implications and contrast them with studies published after the period of investigation reported by Ouhbi et al., which allows us to consider the progress in the field since 2012. Furthermore, we expand on these implications with our own observations and recommendations for REE instructors.

6.1.1 Combating vague requirements

Ouhbi et al. recommend that instructors teach proper problem scoping in order to avoid vague requirements. The authors assert that certain personality traits improve team performance in this respect. Indeed, such a relationship exists [ 164 , 165 , 189 ] and as we have outlined above, student factors such as comprehension, effort, and enthusiasm are explicitly mentioned learning outcomes in 32 of our 152 selected studies. Eighteen of these studies fall into a time frame after Ouhbi et al.’s search completed. While none of these studies mention “vagueness” or “attention to detail” explicitly, several mention “introspection” (e.g., [ 13 ]) or “comprehension” (e.g., [ 117 ]). Unfortunately, “vagueness” or “level of detail” was not mentioned as a learning outcome in any of our selected studies. We conclude from this that instructors have recognized that student factors are crucial in educating students to become effective requirements engineers, yet student factors alone do not yield effective requirements specifications. We recommend instructors consider pedagogical techniques aimed to increase the level of detail and thereby combat vagueness in requirements specifications.

6.1.2 RE tool instruction

With hundreds of RE tools available on the market, Ouhbi et al. make a strong argument for the need to educate students into using these tools effectively. Indeed, 27 of our selected studies deal with tools or advocate using technology to improve learning and instruction. However, the overwhelming majority of these studies propose games (e.g., [ 5 , 50 , 51 , 70 , 156 , 170 , 187 , 196 ]) or simulation tools (e.g., [ 10 , 150 , 154 ]) to teach RE. They do not outline how to use tools during the RE process. A notable exception is [ 87 ], where requirements modeling using tools is taught as well as [ 106 ], which in part investigates the use of tools to conduct validation and verification of requirements. Our results show that RE tool instruction is still lacking, nearly 10 years after the conclusion of Ouhbi et al.’s survey. A focus here should be on industry-typical tools and tools that are likely to produce a tangible benefit to the RE process, for which current industry needs are unknown and must be assessed (see also Sect.  5.1 ). Nevertheless, using the right tools during RE also depends on the company-specific tool chains and may therefore be “on the job training,” rather than something that can (or should) be instructed at university level. Again, an industry perspective is required to answer this question.

6.1.3 Promote requirements modeling, validation and verification, and prototyping

Our results in Sect.  5.2 show that next to elicitation and RE process instruction, the most commonly addressed RE activities are modeling of any kind as well as quality assurance at large (a total of 29 studies, see Fig.  12 ). Most of these studies occurred before 2012 (i.e., when Ouhbi et al. concluded) with the exception of [ 17 , 66 , 87 , 117 , 125 , 184 ] and [ 190 ]. Ouhbi et al. were correct to point out that more instructional focus was required, as these 29 studies made up a mere 19.1% of all our selected studies (compared to 18.4% for “elicitation” alone). We agree that these activities (i.e., modeling and prototyping) should be promoted during RE instruction. Modeling and requirements validation have proven to be a key asset in the requirements engineer’s toolbox to bridge the gap between non-technical and technical stakeholders. Teaching non-technical skills has thus far mostly taken the form of soft skills (see Sect.  5.3 ), but even in this regard, the focus is on communication and interviewing due to the strong overlap with studies that focus on “elicitation” (see Fig.  14 ). Prototyping of requirements specifications has not been emphasized or made a key learning outcome in any of our selected studies. An opportunity here is lost in that students do not benefit from seeing the relationship between “theoretic” requirements specifications and their implementation. While we have reported an activity involving requirements prototyping in one of our selected studies [ 180 ], this was only a minor milestone in a RE project, burdened by other constraints in the timeline of the semester. We recommend practitioners to develop approaches such as [ 108 ] and incorporate requirements implementation as well.

6.1.4 Using industry-realistic projects

As outlined in Sect.  5.1 , delivering an authentic, industry realistic educational experience has consistently been a focus of REE literature since roughly 2005 (see Fig.  11 ). In Ouhbi et al.’s study, the focus was on REE approaches and their relationship to standard curricula, many which require industry-readiness as a student outcome (e.g., [ 1 ]). While this is a positive trend in the past, we concur with Ouhbi et al. that this remains an important educational outcome for future work in REE.

6.1.5 Promote global software development

Ouhbi et al. emphasize the need for REE approaches to meet the demands placed on software development through a consistent move toward distributed teams. In particular, in light of the recent events (i.e., the COVID-19 pandemic), we agree that video conferencing and distributed teamwork have become necessary skills for students, educators, and industry professionals to master, and will likely shape the landscape of software development for the coming years. Teaching effective RE in such a context may be easier going forward because learners may be accustomed to social distancing and working remotely. Nevertheless, only a minority of our selected studies consider distance learning or geographically separated teams, only one of which was published after 2012 (i.e., [ 16 , 108 , 149 , 152 , 198 ]). This must be a focus of REE approaches going forward, and these approaches could build off of the experiences from forced distancing during the COVID-19 pandemic.

6.1.6 Familiarize students with problem solving

Ouhbi et al. highlight the importance of problem solving skills to become effective requirements engineers and recommend REE literature to take a game-based approach to problem solving. While “problem solving” was only explicitly mentioned in one of our included studies (i.e., [ 17 ]) and while games-based instruction or gamification is the focus of several of our selected studies (e.g., [ 5 , 101 , 109 , 187 ]), we argue that these approaches are not the only strategies to teach effective problem solving. In fact, peer-learning [ 27 ], role-playing [ 4 , 98 , 168 ], fostering analytical thinking [ 64 ], and client-orientation [ 157 , 174 ] have been successfully applied to aspects of RE instruction. Problem solving is at the heart of RE. The key caveat seems to be to create a low-stakes environment, where students can “safely fail” (i.e., explore solution alternatives without grade penalty for being wrong or without threatening project success). Approaches that offer low-stakes learning experiences are quite common, both in individual-centered instruction (e.g., [ 113 , 158 ]) and collaborative instruction (e.g., [ 28 , 31 , 119 ]). In fact, 24 of our selected studies can be roughly categorized as employing some form of low-stakes problem solving experience; a trend that should continue in the future.

6.1.7 Use mobile devices as teaching tools

Ouhbi et al. made an argument to use mobile devices and online tools as a vehicle to teach RE. However, Ouhbi et al. did not articulate in what way REE, in particular, benefits from m-learning or e-learning. When examining our selected studies, only three mention some type of online platform or the use of mobile devices to teach RE (i.e., [ 88 , 124 , 144 ]). We conclude from this that the benefits of m-learning and e-learning to REE may still be largely unexplored, beyond the opportunity to prepare students for the challenges of global software development (see Sect.  6.1.5 ).

6.2 Gaps in current RE education literature

While industry-readiness, authenticity, and student soft skill development are important and encouraging trends in REE literature, in the following sections, we highlight the areas that have not received sufficient attention.

6.2.1 Safety and security requirements

Shockingly few studies (i.e., only three: [ 110 , 111 , 143 ]) deal with security requirements and only one study considers safety requirements explicitly [ 180 ]. Since software systems are increasingly entrusted with sensitive information and playing a mission-critical role, it is vital that students are exposed to these considerations at the earliest possible stage during their undergraduate curriculum. Further work is required to understand how to effectively instruct learners on the intricate notions of security requirements and their impact on the system under development. While some studies may incidentally involve safety and security requirements, a systematic educational approach is required.

6.2.2 Supply chain risk management and supplier/integrator relation

Most project-based approaches involving real or realistic stakeholders aim to convey the difficulty of managing conflicting requirements. However, these approaches may prime students towards an attitude of “document and forget” [ 32 ]. Requirements are rarely seen through to their implementation (see “prototyping” in Sect.  6.1 ). Moreover, typical software engineering projects emphasize software construction. The current literature largely ignores the need to systematically explore reuse of off-the-shelf components, the need to critically reflect on adopting components (e.g., libraries), or risk involved when adopting possible security-critical technologies. The decision to adopt a technology and risk its successful integration are inherently RE-related and must be systematically assessed on the basis of requirements. At present, students do not achieve this learning outcome with the approaches reported herein.

6.2.3 Pedagogy in RE education

Systematic application of pedagogy is largely ignored by contemporary REE literature. Merely two approaches make explicit use of Bloom’s taxonomy to guide their instruction [ 19 , 124 ] and only 10.5% of approaches (i.e., [ 4 , 7 , 11 , 19 , 40 , 41 , 53 , 92 , 104 , 112 , 127 , 145 , 155 , 158 , 160 , 161 , 176 ]) consider systematic pedagogy. Yet, with the exception of closely related studies such as [ 11 , 40 , 41 ], there seems to be no common pedagogical framework nor is there a common basis of systematically gathered evidence as to the effectiveness of teaching approaches given learning outcomes. In fact, to the best of our knowledge, the manuscript at hand is the first and thus far only systematic investigation into REE literature and students’ learning outcomes. We therefore declare a call to action for the REE community (and perhaps the software engineering education community at large) to produce a common, evidence-based pedagogical framework. We hope that the work at hand lays a suitable foundation for such an effort.

7 Discussion, conclusions, and future work

In this paper, we presented the results of a systematic literature review into learning outcomes portrayed in Requirements Engineering Education (REE) literature. We have selected 152 primary studies from 1988 to 2020, to provide three contributions: (goal 1) We provide a systematic map of the current state of REE research. (Goal 2) We review the current practices and educational approaches to achieve learning outcomes. (Goal 3) We show how REE has changed in the last decade and which topics remain unexplored in the literature.

Our main findings include the recent trend towards authentic and industry-realistic learning experiences to improve students’ knowledge, predominantly on topics such as requirements elicitation and modeling, but also with regards to students’ soft skills, collaboration, teamwork, and industry-readiness. To accomplish this, current trends involve real or realistic stakeholders and role playing in low-stakes collaborative project-based instruction scenarios. Theory-based instruction plays a subordinate role in REE, suggesting that knowing about theory is less emphasized than effectively applying theory in industry-realistic settings, ideally spanning all parts of the RE process.

Our findings further suggest that areas where REE approaches are currently lacking include instruction of safety and security requirements engineering, as well as supply chain risk management. Moreover, REE presently suffers from a lack of a common pedagogical basis and systematically gathered evidence. While a plethora of successful teaching methods have been proposed (e.g., game-based learning, new frameworks, and educational tools), for the most part, these contributions are in isolation and not part of a systematic attempt to propose methods that are tailored to student outcomes.

We contrast and complement findings from a previous mapping study by Ouhbi et al. [ 135 ]. While Ouhbi et al.’s work focuses on REE approaches and their consideration of standardized curricula, we place emphasis on synthesizing learning outcomes and educational approaches reported in the literature. We also highlight developments in the field since Ouhbi et al.’s study concluded in 2012. In part, we were able to replicate Ouhbi et al.’s results, differ in some findings, and provide additional findings not previously reported.

To our knowledge, a replication of a systematic literature review or mapping study has thus far not yet been completed in the discipline of software engineering. While it was not our aim to replicate Ouhbi et al.’s work, we believe that the work at hand sufficiently highlights areas of overlap. This produces a secondary outcome of our work, i.e., that differences between our findings can be explained by differences in search methodology and as well as rigor in inclusion and exclusion criteria.

In this paper, we lay a foundation for the REE community to produce a rich evidence-based understanding of effective pedagogical approaches. Given the vastness of our data set, we envision future work focusing on qualitative analysis of previous studies to uncover new insights. For example, we found that interviews for elicitation is well studied in the literature. Future work could look at which other elicitation techniques are taught (e.g., questionnaires, analyzing competitors). Similarly, other studies could investigate how requirements quality metrics (e.g., correctness, consistency) are instructed.

In addition to studying the level of learners (see Sect.  4.9 ), future work could study these educational approaches with respect to which approaches are taught as part of introductory, intermediate, or advanced courses in RE and SE, at both the bachelors and master levels. This would give greater insight into the depth of RE curriculum, and would be complementary to initial efforts  [ 64 , 67 ]. Supporting this line of inquiry, we also intend to survey educators to identify best practices and examine whether there are any instructional approaches that could be of relevance for RE education but have not been published.

As already introduced in Sect.  5 , we found 33 papers (i.e., 21.7% of selected studies) with validated approaches, which was insufficient for our intended analysis. Given the importance of evidence as to the effectiveness of pedagogy, we seek to complete an in-depth qualitative analysis of these papers as part of future work in order to provide insights to what works and what does not work. By looking more deeply at RE activities, we can assist new educators in understanding what is recommended.

In addition, as already discussed in the paper (see Sects.  5.1 ,  5.2 , and  6.1.4 ), we found the further research is required to explicitly investigate the relationship between industry-readiness and requirements proficiency among students. Finally, as proposed in Sect.  6.2.1 , we need a systematic educational approach to instruct students on the development and importance of security requirements.

Available at https://2021.refsq.org/details/refsq-2021-papers/3/The-Challenge-s-of-Teaching-Requirements-Engineering .

The data set is available here: https://doi.org/10.35482/csc.003.2021

ABET Engineering Accreditation Commission (2018) Criteria for accrediting engineering programs, 2018–2019

Abran A, Bourque P, Tripp LL (2004) Guide to the software engineering body of knowledge (SWEBOK(R)): version 3.0, 1st edn. IEEE Computer Society Press, Washington, DC

Google Scholar  

Adam S, Doerr J, Eisenbarth M (2009) Lessons learned from best practice-oriented process improvement in requirements engineering—a glance into current industrial RE application. In: Fourth international workshop on requirements engineering education and training, pp 1–5

Al-Ani B, Yusop N (2004) Role-playing, group work and other ambitious teaching methods in a large requirements engineering course. In: Proceedings of 11th IEEE international conference and workshop on the engineering of computer-based systems, pp 299–306

Alami D, Dalpiaz F (2017) A gamified tutorial for learning about security requirements engineering. In: Proceedings of IEEE 25th international requirements engineering conference (RE), pp 418–423

Alexander M, Beatty J (2008) Effective design and use of requirements engineering training games. In: Proceedings of seventh IEEE international workshop on requirements engineering education and training (REET)

Anil GR, Moiz SA (2017) A holistic rubric for assessment of software requirements specification. In: Proceedings of 5th national conference on E-learning and E-learning technologies (ELELTECH), 2017

Armarego J, Minor O (2005) Studio learning of requirements: towards aligning teaching to practitioner needs. In: REET’05 (1st international workshop on RE education and training). REET, 2005

Auriol G, Baron C, Fourniols J-Y (2008) Teaching requirements skills within the context of a physical engineering project. In: Proceedings of seventh IEEE international workshop on requirements engineering education and training (REET)

Babiceanu RF (2014) A software and systems integration framework for teaching requirements engineering. In: Proceedings of 121st ASEE annual conference and exposition, 2014

Bano M, Zowghi D, Ferrari A, Spoletini P (2020) Inspectors academy: pedagogical design for requirements inspection training. In: 2020 IEEE 28th international requirements engineering conference (RE), pp 215–226

Bano M, Zowghi D, Ferrari A, Spoletini P, Donati B (2018) Learning from mistakes: an empirical study of elicitation interviews performed by novices. In: Proceedings of IEEE 26th international requirements engineering conference (RE), pp 182–193

Bano M, Zowghi D, Ferrari A, Spoletini P, Donati B (2019) Teaching requirements elicitation interviews: an empirical study of learning from mistakes. Requir Eng 24(3):259–289

Article   Google Scholar  

Barnes RJ, Gause DC, Way EC (2008) Teaching the unknown and the unknowable in requirements engineering education. In: Proceedings of seventh IEEE international workshop on requirements engineering education and training (REET)

Bennaceur A, Lockerbie J, Horkoff J (2015) On the learnability of i*: experiences from a new teacher. In: 1st International iStar teaching workshop (iStarT 2015), vol 1370, pp 43–48

Berkling K, Geisser M, Hildenbrand T, Rothlauf F (2007) Offshore software development: transferring research findings into the classroom. Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics), vol. 4716 LNCS, pp 1–18

Berre AJ, Huang S, Murad H, Alibakhsh H (2018) Teaching modelling for requirements engineering and model-driven software development courses. Comput Sci Educ 28(1):42–64

Berry DM, Kaplan CS (2010) Planned programming problem gotchas as lessons in requirements engineering. In: Proceedings of 5th international workshop on requirements engineering education and training, pp 20–25

Bhowmik T, Niu N, Reese D (2014) Students vs. professionals in assisted requirements tracing: how could we train our students? In: Proceedings of 121st ASEE annual conference and exposition

Bourque P, Fairley RE, IEEE Computer Society (2014) Guide to the software engineering body of knowledge (SWEBOK(R)): version 3.0, 3rd edn. IEEE Computer Society Press, Washington, DC

Brings J, Daun M, Kempe M, Weyer T (2018) On different search methods for systematic literature reviews and maps: experiences from a literature search on validation and verification of emergent behavior. In: Proceedings of the 22nd international conference on evaluation and assessment in software engineering 2018, pp 35–45

Bubenko JA (1995) Challenges in requirements engineering. In: Proceedings of 1995 IEEE international symposium on requirements engineering (RE’95), pp 160–162

Callele D, Makaroff D (2006) Teaching requirements engineering to an unsuspecting audience. SIGCSE Bull 38(1):433–437

Cico O, Jaccheri L, Nguyen-Duc A, Zhang H (2020) Exploring the intersection between software industry and software engineering education—a systematic mapping of software engineering trends. J Syst Softw 172:110736

Cockburn A, Dragicevic P, Besançon L, Gutwin C (2020) Threats of a replication crisis in empirical computer science. Commun ACM 63(8):70–79

Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20(1):37–46

Connor AM, Buchan J, Petrova K (2009) Bridging the research-practice gap in requirements engineering through effective teaching and peer learning. In: Proceedings of sixth international conference on information technology: new generations, pp 678–683

Cybulski JL, Parker C, Segrave S (2006) Touch it, feel it and experience it: developing professional is skills using interview-style experiential simulations. In: Proceedings of 17th Australasian conference on information systems, 2006

Daneva M, Damian D, Marchetto A, Pastor O (2014) Empirical research methodologies and studies in requirements engineering: how far did we come? J Syst Softw 95:1–9

Daun M (2020) Teaching requirements engineering with industry case examples. In: Software engineering Unterricht and Hochschulen (SEUH), 2020

Daun M, Salmon A, Tenbergen B, Weyer T, Pohl K (2014) Industrial case studies in graduate requirements engineering courses: the impact on student motivation. In: Proceedings of IEEE 27th conference on software engineering education and training (CSEE&T), pp 3–12

Daun M, Salmon A, Weyer T, Pohl K, Tenbergen B (2016) Project-based learning with examples from industry in university courses: An experience report from an undergraduate requirements engineering course. In: Proceedings of IEEE 29th international conference on software engineering education and training (CSEE&T), pp 184–193

Davis AM, Hickey AM, Chamillard AT (2005) Moving beyond the classroom: integrating requirements engineering research & education to improve practice

de Pádua Albuquerque Oliveira A, Werneck VMB, do Prado Leite JCS, Cysneiros LM (2015) The monopoly game to teach ERi*c—intentional requirements engineering. In: 1st International iStar teaching workshop (iStarT 2015), vol 1370, pp 49–54

Donati B, Ferrari A, Spoletini P, Gnesi S (2017) Common mistakes of student analysts in requirements elicitation interviews. Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics), vol 10153. LNCS, pp 148–164

Engström E, Runeson P (2011) Software product line testing—a systematic mapping study. Inf Softw Technol 53(1):2–13

Feldgen M, Clua O (2015) Teaching effective requirements engineering for large-scale software development with scaffolding. In: Proceedings of IEEE frontiers in education conference (FIE), vol 2015-February, 2015

Fernandes JM, Machado RJ, Seidman SB (2009) A requirements engineering and management training course for software development professionals. In: 2009 22nd conference on software engineering education and training, February 2009, pp 20–25

Fernández DM, Franch X, Seyff N, Felderer M, Glinz M, Kalinowski M, Volgelsang A, Wagner S, Bühne S, Lauenroth K (2019) Do we preach what we practice? Investigating the practical relevance of requirements engineering syllabi—the IREB case

Ferrari A, Spoletini P, Bano M, Zowghi D (2019) Learning requirements elicitation interviews with role-playing, self-assessment and peer-review. In: 2019 IEEE 27th international requirements engineering conference (RE), pp 28–39

Ferrari A, Spoletini P, Bano M, Zowghi D (2020) SaPeer and ReverseSaPeer: teaching requirements elicitation interviews with role-playing and role reversal. Requir Eng 25(4):417–438

Ferrari R, Madhavji N (2005) Requirements engineering education for novice software architects. In: Proceedings of the 1st international workshop on requirements engineering education and training (REET), 2005

Ferreira VG, Canedo ED (2019) Using design sprint as a facilitator in active learning for students in the requirements engineering course: An experience report. In: Proceedings of the 34th ACM/SIGAPP symposium on applied computing (SAC19), pp 1852–1859

France RB, Larrondo-Petrie MM (1995) Understanding the role of formal specification techniques in requirements engineering. Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics), vol 895, pp 207–221

Fuji T (2005) Finding competitive advantage in requirements analysis education. In: 13th IEEE international conference on requirements engineering (RE’05), pp 493–494

Gabrysiak G, Giese H, Seibel A (2011) Why should I help you to teach requirements engineering? In: 2011 6th International workshop on requirements engineering education and training, August 2011, pp 9–13

Gabrysiak G, Giese H, Seibel A, Neumann S (2010) Teaching requirements engineering with virtual stakeholders without software engineering knowledge. In: Proceedings of 5th international workshop on requirements engineering education and training, pp 36–45

Gabrysiak G, Hebig R, Pirl L, Giese H (2013) Cooperating with a non-governmental organization to teach gathering and implementation of requirements. In: 2013 26th International conference on software engineering education and training (CSEE T), May 2013, pp 11–20

Garbers B, Periyasamy K (2006) A light-weight tool for teaching the development and evaluation of requirements documents. In: Annual conference of American Society of Engineering Education (ASEE), 2006

Garcia I, Pacheco C, León A, Calvo-Manzano JA (2019) Experiences of using a game for improving learning in software requirements elicitation. Comput Appl Eng Educ 27(1):249–265

García I, Pacheco C, León A, Calvo-Manzano JA (2020) A serious game for teaching the fundamentals of ISO/IEC/IEEE 29148 systems and software engineering—lifecycle processes—requirements engineering at undergraduate level. Comput Stand Interfaces 67:103377

Garousi V, Mesbah A, Betin-Can A, Mirshokraie S (2013) A systematic mapping study of web application testing. Inf Softw Technol 55(8):1374–1396

Gary KA (2009) Contextual requirements experiences within the software enterprise. In: Proceedings of fourth international workshop on requirements engineering education and training, pp 12–19

Gibson JP (2000) Formal requirements engineering: learning from the students. In: Proceedings 2000 Australian software engineering conference, pp 171–180

Glinz M (2000) Improving the quality of requirements with scenarios

Glinz M (2020) Standard glossary for the certified professional for requirements engineering (CPRE) studies and exam v2.0.0. Technical report, International Requirements Engineering Board e.V., October 2020

Glinz M, Fricker SA (2015) On shared understanding in software engineering: an essay. Comput Sci Res Dev 30(3):363–376

Goldsmith RF (2009) BAs will falter until they learn to discover REAL, business requirements. In: Proceedings of fourth international workshop on requirements engineering education and training (REET ’09), pp 6–11

Gotel O, Kulkarni V, Say M, Scharff C, Sunetnanta T (2009) Distributing responsibilities to engineer better requirements: leveraging knowledge and perspectives for students to learn a key skill. In: Fourth international workshop on requirements engineering education and training, pp 28–37

Gotel OCZ, Morris SJ (2012) Case-based stories for traceability education and training. In: Proceedings of seventh IEEE international workshop on requirements engineering education and training (REET), pp 1–8

Hagel G, Müller-Amthor M, Landes D, Sedelmaier Y (2018) Involving customers in requirements engineering education: mind the tgoals! In: Proceedings of 3rd European conference of software engineering education (ECSEE), pp 113–121

Hasson P, Cooper S (2004) A case study involving the use of Z to aid requirements specification in the software engineering course. In: Proceedings of 17th conference on software engineering education and training, vol 17, pp 84–89

Heimbürger A, Isomöttönen V (2019) Infographics as a reflective assignment method in requirements engineering e-course? In: 2019 IEEE frontiers in education conference (FIE), pp 1–5

Hertz K, Spoletini P (2018) Are requirements engineering courses covering what industry needs? A preliminary analysis of the United States situation. In: IEEE 8th international workshop on requirements engineering education and training (REET), pp 20–23

Hmelo-Silver CE (2004) Problem-based learning: what and how do students learn? Educ Psychol Rev 16:235–266

Horkoff J (2015) Observational studies of new i* Users: challenges and recommendations. In: 1st International iStar teaching workshop (iStarT 2015), vol 1370, pp 13–18

Horkoff J (2018) The influence of agile methods on requirements engineering courses. In: Proceedings of IEEE 8th international workshop on requirements engineering education and training (REET), pp 11–19

Huijs C, Sikkel K, Wieringa R (2005) Mission 2 solution: Requirements engineering education as central theme in the BIT programme

Iacob C, Faily S (2017) Using extreme characters to teach requirements engineering. In: Proceedings of IEEE 30th conference on software engineering education and training (CSEE&T), vol 2017-January, pp 107–111

Ibrahim Z, Soo MC, Soo MT, Aris H (2019) Design and development of a serious game for the teaching of requirements elicitation and analysis. In: 2019 IEEE international conference on engineering, technology and education (TALE), pp 1–8

Idri A, Ouhbi S, Fernández-Aléman JL, Toval A (2012) A survey of requirements engineering education. In: Proceedings of the 2012 IEEE global engineering education conference (EDUCON), pp 1–5

IEEE (1998) IEEE standard for conceptual modeling language syntax and semantics for IDEF1X/Sub 97/ (IDEF/Sub Object/). IEEE Std 1320.2-1998

IEEE (2014) IEEE standard for software quality assurance processes. IEEE Std 730-2014 (Revision of IEEE Std 730-2002), pp 1–138

IEEE & ACM JTFCC, Software Engineering (2004) Curriculum guidelines for undergraduate degree programs in software engineering. Technical report, IEEE & ACM; The Joint Task Force on Computing Curricula

IREB (2020) Certified professional for requirements engineering foundation level syllabus v3.0.1. Technical report, International Requirements Engineering Board e.V., October 2020

ISO/IEC (2002) ISO/IEC 15474-1:2002 information technology—CDIF framework—part 1: overview

ISO/IEC (2008) ISO/IEC 14102:2008 information technology—guideline for the evaluation and selection of CASE tools

ISO/IEC (2009) ISO/IEC 10746-2:2009 information technology—open distributed processing—reference model: foundations

ISO/IEC (2012) ISO/IEC 19500-2:2012 information technology—object management group—common object request broker architecture (CORBA)—part 2: interoperability

ISO/IEC (2014) ISO/IEC 25000:2014 systems and software engineering—systems and software quality requirements and evaluation (SQuaRE)—guide to SQuaRE

ISO/IEC (2015) ISO/IEC 2382:2015 information technology—vocabulary

ISO/IEC/IEEE (2017) ISO/IEC/IEEE Int. standard—systems and software engineering—vocabulary. ISO/IEC/IEEE 24765:2017(E)

Jagielska D, Wernick P, Wood M, Bennett S (2006) How natural is natural language?: how well do computer science students write use cases? In: Companion to the 21st ACM SIGPLAN symposium on object-oriented programming systems, languages, and applications, vol 2006, pp 914–924

Jamaludin NAA, Sahibuddin S, Hidayat NH (2012) Challenges of a project-based learning approach towards requirement engineering. Int J Comput Appl 50(3):66–71

Jiang Y, Li M, He Z, Zhao C (2009) Nine steps to shorten the distance between requirement theory and practice. In: First international workshop on education technology and computer science, vol 3, pp 694–698

Kakeshita T, Yamashita S (2015) A requirement management education support tool for requirement elicitation process of REBOK. In: 3rd International conference on applied computing and information technology/2nd international conference on computational science and intelligence, pp 40–45

Keller K, Neubauer A, Brings J, Daun M (2018) Tool-support to foster model-based requirements engineering for cyber-physical systems. In: Modellierung (workshops), vol 2060, pp 47–56

Kilicay-Ergin N, Laplante PA (2013) An online graduate requirements engineering course. IEEE Trans Educ 56(2):208–216

Kitchenham B, Brereton P (2013) A systematic review of systematic review process research in software engineering. Inf Softw Technol 55(12):2049–2075

Klapholtz D, McDonald J, Pyster A (2009) The graduate software engineering reference curriculum (gswerc). In: 2009 22nd Conference on software engineering education and training, pp 290–291

Knauss E, Schneider K, Stapel K (2008) A game for taking requirements engineering more seriously. In: 2008 Third international workshop on multimedia and enjoyable requirements engineering—beyond mere descriptions and with more fun and games, pp 22–26

Koch M, Landes D (2015) Making means-end-maps workable for recommending teaching methods. In: Proceedings of eighth international i* workshop, vol 1402, pp 85–90

Kramer J (2003) Abstraction—is it teachable? ‘the devil is in the detail’. In: 16th Conference on software engineering education and training, vol 2003-January, p 32

Kurkovsky S, Ludi S, Clark L (2019) Active learning with LEGO for software requirements. In: Proceedings of the 50th ACM technical symposium on computer science education, SIGCSE ’19, New York, NY, USA. Association for Computing Machinery, pp 218–224

Laiq M, Dieste O (2020) Chatbot-based interview simulator: a feasible approach to train novice requirements engineers. In: 2020 10th International workshop on requirements engineering education and training (REET), pp 1–8

Lami G (2005) Teaching requirements engineering in the small: an under-graduate course experience. In: Proceedings of the 1st international workshop on requirements engineering education and training (REET), 2005

Lavallee M, Robillard P-N, Mirsalari R (2013) Performing systematic literature reviews with novices: an iterative approach. IEEE Trans Educ 57(3):175–181

Liang P, De Graaf O (2010) Experiences of using role playing and wiki in requirements engineering course projects. In: Proceedings of 18th international IEEE requirements engineering conference, pp 1–6

Lima T, Campos B, Santos R, Werner C (2012) UbiRE: a game for teaching requirements in the context of ubiquitous systems. In: Proceedings of XXXVIII Conferencia Latinoamericana En Informatica (CLEI), 2012

Liu L, Jin Z (2008) Balancing academic and industrial needs in RE courses. In: Requirements engineering education and training, 2008

Lopes J (2020) Evaluating the students’ experience with a requirements elicitation and communication game. In: Proceedings of 23rd Ibero-American conference on software engineering (CIbSE 2020), ClbSE 2020

Lopez-Lorca A, Burrows R, Sterling L (2018) Teaching motivational models in agile requirements engineering. In: Proceedings of 8th international workshop on requirements engineering education and training (REET), pp 30–39

Ludi S (2007) Introducing accessibility requirements through external stakeholder utilization in an undergraduate requirements engineering course. In: Proceedings of 29th international conference on software engineering (ICSE’07), pp 736–743

Macaulay L, Mylopoulos J (1995) Requirements engineering: an educational dilemma. Autom Softw Eng 2(4):343–351

Malik B, Zafar S (2012) A systematic mapping study on software engineering education. Int J Educ Pedag Sci 6(11):3343–3353

Manohar P, Acharya S, Wu PY, Ansari AA, Schilling WW Jr (2015) Case study based educational tools for teaching software V&V course at undergraduate level. In: 122nd ASEE annual conference and exposition: making value for society, 2015

Marsicano G, Mendes FF, Fernandes MV, Freitas SAAD (2016) An integrated approach to the requirements engineering and process modelling teaching. In: IEEE 29th international conference on software engineering education and training (CSEET), pp 166–174

Marutschke DM, Kryssanov VV, Brockmann P (2020) Teaching distributed requirements engineering: simulation of an offshoring project with geographically separated teams. In: 2020 IEEE 32nd conference on software engineering education and training (CSEE T), pp 1–5

Mayr H (2015) Teaching better requirements engineering using LEGO® serious Play™. In: Proceedings of ACM conference on innovation and technology in computer science education (ITiCSE ’15), pp 126–131

Mead NR, Hough ED (2006) Security requirements engineering for software systems: case studies in support of software engineering education. In: Proceedings of 19th conference on software engineering education and training, vol 2006, pp 149–156

Mead NR, Shoemaker D, Ingalsbe J (2009) Teaching security requirements engineering using SQUARE. In: Proceedings of fourth international workshop on requirements engineering education and training, pp 20–27

Memon RN, Ahmad R, Salim SS (2010) Problems in requirements engineering education. In: Proceedings of 8th international conference on frontiers of information (FIT ’10), 2010

Memon RN, Ahmad R, Salim SS (2013) A direction framework to address problems in requirements engineering education. Malays J Comput Sci 26(4):294–311

Memon RN, Salim SS, Ahmad R (2012) Analysis and classification of problems associated with requirements engineering education: towards an integrated view. Arab J Sci Eng 39(3):1923–1935

Memon RN, Salim SS, Ahmad R (2012) Identifying research gaps in requirements engineering education: an analysis of a conceptual model and survey results. In: IEEE conference on open systems, 2012

Merten T, Schäfer T, Bürsner S (2012) Using RE knowledge to assist automatically during requirement specification. In: Seventh IEEE international workshop on requirements engineering education and training (REET), pp 9–13

Mich L (2014) Teaching requirements analysis: a student project framework to bridge the gap between business analysis and software engineering. In: Proceedings of 8th international workshop on requirements engineering education and training (REET), vol 1217, pp 20–25

Minocha S, Petre M, Roberts D (2008) Using wikis to simulate distributed requirements development in a software engineering course. Int J Eng Educ 24(4):689–704

Mkpojiogu EOC, Hussain A (2017) Assessing students’ performance in software requirements engineering education using scoring rubrics. In: AIP conference proceedings, vol 1891

Mohamed AH (2010) Facilitating tacit-knowledge acquisition within requirements engineering. In: Proceedings of the 10th WSEAS international conference on applied computer science, ACS’10, Stevens Point, Wisconsin, USA, 2010. World Scientific and Engineering Academy and Society (WSEAS), pp 27–32

Mohan S, Chenoweth S (2011) Teaching requirements engineering to undergraduate students. In: Proceedings of 42nd ACM technical symposium on computer science (SIGCSE ’11), pp 141–146

Moody DL, Sindre G (2003) Incorporating quality assurance processes into requirements analysis education. In: Proceedings of the 8th annual conference on innovation and technology in computer science education, vol 8, pp 74–78

Morales-Ramirez I, Alva-Martinez LH (2018) Requirements analysis skills: how to train practitioners? In; IEEE 8th international workshop on requirements engineering education and training (REET), pp 24–29

Moreira F, Ferreira MJ (2016) Teaching and learning requirements engineering based on mobile devices and cloud: a case study. In: Blended learning: concepts, methodologies, tools, and applications, vol 4. IGI Global, pp 1190–1217

Nakamura T, Kai U, Tachikawa Y (2015) Requirements engineering education using expert system and role-play training. In: Proceedings of IEEE international conference on teaching, assessment and learning for engineering (TALE), pp 375–382

Nakatani T (2008) Requirements engineering education for professional engineers. Front Artif Intell Appl 180(1):495–504

Nakatani T, Tsumaki T, Tamai T (2010) Instructional design of a requirements engineering education course for professional engineers. Smart Innov Syst Technol 3:119–151

Nakatani T, Tsumaki T, Tamai T (2010) Requirements engineering education for senior engineers: course design and its evaluation. In: Proceedings of 5th international workshop on requirements engineering education and training, pp 26–35

Nguyen L, Armarego J, Swatman P (2005) Understanding requirements engineering process: a challenge for practice and education. In: 5th International business information management conference, vol 1

Nkamaura T, Tachikawa Y (2017) Requirements engineering education using role-play training. In: Proceedings of 2016 IEEE international conference on teaching, assessment, and learning for engineering (TALE), pp 231–238

Noel R, Munoz R, Becerra C, Villarroel R (2017) Developing competencies for software requirements analysis through project based learning. In: Proceedings of 35th international conference of the Chilean Computer Science Society (SCCC)

Ochodek M, Kopczyńska S (2018) Perceived importance of agile requirements engineering practices—a survey. J Syst Softw 143:29–43

Ogata S, Matsuura S (2012) Training of requirements analysis modeling with UML-based prototype generation tool. In; 5th India software engineering conference, pp 105–108

Ott D, Raschke A (2012) Review improvement by requirements classification at Mercedes-Benz: limits of empirical studies in educational environments. In: Second IEEE international workshop on empirical requirements engineering (EmpiRE), pp 1–8

Ouhbi S, Idri A, Fernández-Alemán JL, Toval A (2015) Requirements engineering education: a systematic mapping study. Requir Eng 20(2):119–138

Paschoal LN, de Oliveira MM, Chicon PMM (2018) A chatterbot sensitive to student’s context to help on software engineering education. In: 2018 XLIV Latin American computer conference (CLEI), pp 839–848

Penzenstadler B, Mahaux M, Heymans P (2013) University meets industry: calling in real stakeholders. In: 2013 26th International conference on software engineering education and training (CSEE T), May 2013, pp 1–10

Penzenstadler B, Richardson D, Karlin B, Cook A, Callele D, Wnuk K (2014) Using non-profit partners to engage students in RE. In: Proceedings of 8th international workshop on requirements engineering education and training (REET 2014), vol 1217, pp 1–10

Periyasamy K, Qin X, He D (2011) A requirements editor for teaching requirements engineering. REET workshop, 2011

Petersen K, Feldt R, Mujtaba S, Mattsson M (2008) Systematic mapping studies in software engineering. In: 12th International conference on evaluation and assessment in software engineering (EASE) 12, pp 1–10

Petersen K, Gencel C (2013) Worldviews, research methods, and their relationship to validity in empirical software engineering research. In: 2013 Joint conference of the 23rd international workshop on software measurement and the 8th international conference on software process and product measurement. IEEE, pp 81–89

Petersen K, Vakkalanka S, Kuzniarz L (2015) Guidelines for conducting systematic mapping studies in software engineering: an update. Inf Softw Technol 64:1–18

Pfleeger SL, Pfleeger CP (2009) Harmonizing privacy with security principles and practices. IBM J Res Dev 53(2):6:1–6:12

Prihartini N, Soemitro HL, Hendradjaya B (2017) Identifying aspects of web e-learning in LMS-based for requirement engineering process modeling. In: International conference on data and software engineering (ICoDSE), 2017

Quintanilla Portugal RL, Engiel P, Pivatelli J, Do Prado Leite JCS (2016) Facing the challenges of teaching requirements engineering. In: Proceedings of 38th international conference on software engineering companion (ICSE ’16), pp 461–470

Regev G, Gause DC, Wegmann A (2008) Requirements engineering education in the 21st century, an experiential learning approach. In: 2008 16th IEEE international requirements engineering conference, pp 85–94

Regev G, Gause DC, Wegmann A (2008) Requirements engineering education in the 21st century, an experiential learning approach. In: 16th IEEE international requirements engineering conference, pp 85–94

Rempel P, Mäder P (2015) A quality model for the systematic assessment of requirements traceability. In: 2015 IEEE 23rd international requirements engineering conference (RE), pp 176–185

Romero M, Vizcaíno A, Piattini M (2008) Developing the skills needed for requirement elicitation in global software development. In: Proceedings of the tenth international conference on enterprise information systems, vol DISI, pp 393–396

Romero M, Vizcaíno A, Piattini M (2008) A simulator for education and training in global requirements engineering: a work in progress. In: Proceedings of eighth IEEE international conference on advanced learning technologies, pp 123–125

Romero M, Vizcaíno A, Piattini M (2008) Toward a definition of the competences for global requirements elicitation. In: 13th Annual conference on innovation and technology in computer science education, p 364

Romero M, Vizcaíno A, Piattini M (2008) Towards the definition of a multi-agent simulation environment for education and training in global requirements elicitation. In: Proceedings of 2008 conference on human system interactions, pp 48–53

Rosca D (2003) Developing teamwork and communication skills in a multidisciplinary experiment. In: Proceedings of 33rd annual frontiers in education conference, vol 3, pp S4C14–S4C17

Rosca D (2000) Active/collaborative approach in teaching requirements engineering. In: Proceedings of 30th annual frontiers in education conference, vol 1, pp T2C-9–T2C-12

Rupakheti CR, Hays M, Mohan S, Chenoweth S, Stouder A (2017) On a pursuit for perfecting an undergraduate requirements engineering course. In: Proceedings of IEEE 30th conference on software engineering education and training (CSEE&T), vol 2017-January, pp 97–106

Rusu A, Russell R, Cocco R (2011) Simulating the software engineering interview process using a decision-based serious computer game. In: Proceedings of 16th international conference on computer games (CGAMES), pp 235–239

Scepanovic S, Beus-Dukic L (2015) Teaching requirements engineering: EUROWEB experience. In: Proceedings of European conference on software architecture (ECSAW ’15), vol. 07-11-September-2015

Sedelmaier Y, Landes D (2014) A multi-level didactical approach to build up competencies in requirements engineering. In: Proceedings of 8th international workshop on requirements engineering education and training (REET 2014), vol 1217, pp 26–34

Sedelmaier Y, Landes D (2014) Using business process models to foster competencies in requirements engineering. In: Proceedings of IEEE 27th conference on software engineering education and training (CSEE&T), pp 13–22

Sedelmaier Y, Landes D (2017) Experiences in teaching and learning requirements engineering on a sound didactical basis. In: Proceedings of ACM conference on innovation and technology in computer science education (ITiCSE ’17), vol Part F128680, pp 116–121

Sedelmaier Y, Landes D (2018) Systematic evolution of a learning setting for requirements engineering education based on competence-oriented didactics. In: Proceedings of IEEE global engineering education conference (EDUCON), vol. 2018-April, pp 1062–1070

Shaw M (2000) Software engineering education: a roadmap. In: Proceedings of the conference on the future of software engineering, ICSE ’00, New York, NY, USA, 2000. Association for Computing Machinery, pp 371–380

Shepperd M, Ajienka N, Counsell S (2018) The role and value of replication in empirical software engineering results. Inf Softw Technol 99:120–132

Shuto M, Washizaki H, Kakehi K, Fukazawa Y, Yamato S, Okubo M, Tenbergen B (2017) Relationship between the five factor model personality and learning effectiveness of teams in three information systems education courses. In: 2017 18th IEEE/ACIS international conference on software engineering, artificial intelligence, networking and parallel/distributed computing (SNPD), pp 167–174

Shuto M, Washizaki H, Fukazawa Y, Yamato S, Okubo M, Tenbergen B (2018) Personality and learning effectiveness of teams in information systems education courses. EAI Endorsed Trans e-Learn 5(17):1–9

Sikkel K, Daneva M (2010) Teaching consistency of UML specifications. In: 5th International workshop on requirements engineering education and training, pp 17–19

Sikkel K, Daneva M (2011) Getting the client into the loop in information system modelling courses. In: Proceedings of 6th international workshop on requirements engineering education and training (REET), pp 1–4

Sindre G (2005) Teaching oral communication techniques in RE by student-student role play: initial experiences. In: 18th Conference on software engineering education and training (CSEET’05), pp 85–94

Smith R, Gotel O (2007) Re-o-poly: a game to introduce lightweight requirements engineering good practices

Soo MT, Aris H (2019) Game-based learning in requirements engineering: an overview. In: Proceedings of 2018 IEEE conference on e-Learning, e-Management and e-Services (IC3e), pp 46–51

Souza AF, Ferreira B, Valentim N, Conte T (2018) An experience report on teaching multiple design thinking techniques to software engineering students. In: XXXII Brazilian symposium on software engineering, pp 220–229

Spichkova M (2019) Industry-oriented project-based learning of software engineering. In: 2019 24th International conference on engineering of complex computer systems (ICECCS), pp 51–60

Suri D (2002) Introducing requirements engineering in an undergraduate engineering curriculum: lessons learnt. In: 2002 American Society for engineering education annual conference and exposition, pp 3175–3183

Suri D, Gassert J (2005) Gathering project requirements: a collaborative and interdisciplinary experience. In: Proceedings of American Society for engineering education annual conference and exposition, pp 6759–6764

Svahnberg M, Gorschek T, Borg A, Sandahl K, Eriksson M, Börster J, Loconsole A (2008) Perspectives on requirements understandability—for whom does the teacher’s bell toll? In: 2008 Requirements engineering education and training (REET)

Svensson RB, Regnell B (2017) Is role playing in Requirements Engineering Education increasing learning outcome? Requir Eng 22(4):475–489

Tachikawa Y, Nakamura T (2017) Education for requirements elicitation using group-work and role-play. In: Proceedings of 2017 IEEE global engineering education conference (EDUCON), pp 780–783

Takako N (2007) Improving engineering mind in eliciting requirements. In: REET workshop, 2007

Tenbergen B, Daun M (2019) Is requirements-engineering research delivering what it promised?: a review of its accomplishments and opportunities after 10 years. IEEE Softw 36(4):6–11

Tenbergen B, Daun M (2019) Industry projects in requirements engineering education: application in a university course in the US and comparison with Germany. In: Proceedings of Hawaii international conference on system sciences 2019, HICSS-53, 2019

Tiwari S, Ameta D, Singh P, Sureka A (2018) Teaching requirements engineering concepts using case-based learning. In: Proceedings of 2nd international workshop on software engineering education for millennials (SEEM’18), pp 8–15

Tiwari S (2020) Impact of CBL on student’s learning and performance: an experience report. In: Proceedings of 13th innovations in software engineering conference (ISEC 2020), ISEC 2020, New York, NY, USA, 2020. Association for Computing Machinery

Tomayko JE (1999) Forging a discipline: an outline history of software engineering education. Ann Softw Eng 6(1–4):3–18

Tort A, Olivé A, Pastor JA (2014) Former students’ views on the usefulness of conceptual modeling education. Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics), vol 8697, pp 237–246

Tuya J, Garcia-Fanjul J (1999) Teaching requirements analysis by means of student collaboration. In: Proceedings—frontiers in education conference, vol 1, pp 11b4–11–11b4–15

Van Lamsweerde A (2009) Requirements engineering: from system goals to UML models to software, vol 10. Wiley, Chichester

Vega K, Fuks H, Carvalho G (2009) Training in requirements by collaboration: branching stories in second life. In: Proceedings of Simpósio Brasilerio de Sistemas Colaborativos (SBSC 2009), pp 116–122

Von Konsky BR, Robey M, Nair S (2004) Integrating design formalisms in software engineering education. In: Proceedings of 17th conference on software engineering education and training, vol 17, pp 78–83

Washizaki H, Sunaga Y, Shuto M, Kakehi K, Fukazawa Y, Yamato S, Okubo M, Tenbergen B (2017) Combinations of personal characteristic types and learning effectiveness of teams. In: 2017 IEEE 41st annual computer software and applications conference (COMPSAC), vol 1, pp 456–457

Wei B, Delugach HS, Colmenares E, Stringfellow C (2016) A conceptual graphs framework for teaching UML model-based requirements acquisition. In: Proceedings of IEEE 29th international conference on software engineering education and training (CSEET), pp 71–75

Westphal B (2018) An undergraduate requirements engineering curriculum with formal methods. In: Proceedings of IEEE 8th international workshop on requirements engineering education and training (REET), pp 1–10

Wever A, Maiden N (2011) What are the day-to-day factors that are preventing business analysts from effective business analysis? In: 2011 IEEE 19th international requirements engineering conference, pp 293–298

Weyer T, Daun M, Tenbergen B (2020) The changing world and the adapting machine—how digital transformation changes requirements engineering in the embedded and cyber-physical systems industry. IEEE Softw 38(5):83–91

Wieringa R, Maiden N, Mead N, Rolland C (2006) Requirements engineering paper classification and evaluation criteria: a proposal and a discussion. Requir Eng 11(1):102–107

Wohlin C, Regnell B (1999) Achieving industrial relevance in software engineering education. In: Proceedings 12th conference on software engineering education and training (Cat. No.PR00131), pp 16–25

Yasin A, Liu L, Li T, Wang J, Zowghi D (2018) Design and preliminary evaluation of a cyber Security Requirements Education Game (SREG). Inf Softw Technol 95:179–200

Zhang H, Babar MA, Tell P (2011) Identifying relevant studies in software engineering. Inf Softw Technol 53(6):625–637

Zowghi D (2009) Teaching requirements engineering to the Bahá’í students in Iran who are denied of higher education. In: Proceedings of fourth international workshop on requirements engineering education and training (REET ’09), pp 38–48

Zowghi D, Yusop N, Mehboob Z (2007) The role of conducting stakeholder meetings in requirements engineering training. In: International workshop on the requirements engineering education and training. REET workshop, 2007

Download references

Open Access funding enabled and organized by Projekt DEAL.

Author information

Authors and affiliations.

paluno-The Ruhr Institute for Software Technology, University of Duisburg-Essen, 45127, Essen, Germany

Marian Daun & Viktoria Stenkova

Department of Computer Science, Smith College, Northampton, MA, 01063, USA

Alicia M. Grubb

Department of Computer Science, State University of New York, Oswego, NY, 13126, USA

Bastian Tenbergen

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Marian Daun .

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Daun, M., Grubb, A.M., Stenkova, V. et al. A systematic literature review of requirements engineering education. Requirements Eng 28 , 145–175 (2023). https://doi.org/10.1007/s00766-022-00381-9

Download citation

Received : 21 May 2021

Accepted : 01 April 2022

Published : 19 May 2022

Issue Date : June 2023

DOI : https://doi.org/10.1007/s00766-022-00381-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Requirements engineering
• Requirements engineering education
• Systematic literature review
• Learning outcomes
• Find a journal
• Publish with us
• Track your research