research for project based learning

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 08 December 2020

Project-based learning: an analysis of cooperation and evaluation as the axes of its dynamic

• Berta de la Torre-Neches   ORCID: orcid.org/0000-0001-7305-362X 1 ,
• Mariano Rubia-Avi 1 ,
• Jose Luis Aparicio-Herguedas 2 &
• Jairo Rodríguez-Medina   ORCID: orcid.org/0000-0002-6466-5525 3  

Humanities and Social Sciences Communications volume  7 , Article number:  167 ( 2020 ) Cite this article

11k Accesses

11 Citations

14 Altmetric

Metrics details

• Development studies

Project-based learning is an active method that develops the maximum involvement and participation of students in the learning process. It requires the teacher to energize the learning scenario by promoting the cooperation of students to investigate, make decisions and respond to the challenges of the project. It also requires activating an evaluation system that promotes awareness, reflexivity and a critical spirit, facilitating deeper learning. This case study aims to understand the functioning of cooperative work established during the application of the method, as well as to know how the evaluation process progresses in the perspective of a group of teachers of secondary education that set up this methodology in their classes. The data obtained from interviews with the teachers involved in the study, teachers’ notebooks, and open-question questionnaire applied to high-school students are analyzed. Although the students were organized in small groups in order to develop their collaborative skills, intragroup frictions and conflicts were not sufficiently addressed or supervised in time by the teachers, thus resulting in an incomplete development of the synergies and collaboration necessaries to the project. From the point of view of the evaluation, the importance of the implementation of training and shared evaluation systems is well recognized, although a more traditional evaluation model, which does not sufficiently address the project development process prevails, and the value of the qualification on the final product achieved still weights.

Similar content being viewed by others

Elementary school teachers’ attitudes towards project-based learning in China

Community-oriented engineering co-design: case studies from the Peruvian Highlands

Students’ structured conceptualizations of teamwork in multidisciplinary student teams using concept maps

Introduction.

As a result of the crisis scenario that began in Spain in 2007, the need to incorporate to the Secondary Education stage some subjects with economic contents, was posed in order to introduce and make students understand the socio-economic circumstances in the world. Simultaneously, teaching methods have been incorporating some learning methodologies that aim to make students able to solve, with involvement, the problems presented to them (Martín and Rodríguez, 2015 ). Some of these methods orient learning towards a competitive character such as cooperative methodologies, gamification or project-based learning (PBL) (Hernández March, 2006 ).

The PBL method is a methodological alternative that involves direct contact with the object of study and ends with the realization of a work project by the students initially proposed by the teacher (Bell, 2010 ), applying knowledge and skills and developing an attitude of commitment (Sánchez, 2018 ). In order to do this, students analyze the topic raised, think about it, organize themselves, search for information, work as a team and make decisions. It is, therefore, intended to promote knowledge of the contents as well as the management of skills and attitudes, learning to mobilize those resources said in situation and to solve problems (Perrenoud, 2008 ).

The experience carried out requires students to face real-life problem statements through activities that suit their interests (Krajcik and Blumenfeld, 2006 ), find and use tools to address them and act collaboratively to propose solutions through an action plan (Barret, 2005 ; Bender, 2012 ; Blumenfeld et al., 1991 ). Traditional training models are based in the premise that students have to know the content in order to apply it in solving a problem. The PBL reverses this order and considers that students obtain the knowledge while solving a problem (Jonassen, 2011 ), an aspect that results in a higher quality of the information they handle to solve it, since it is shared, discussed and applied in a concrete situation (Thomas, 2000 ).

Thus, through PBL, students plan, discuss, and implement projects that have real-world impact and are significant to them (Blank, 1997 ; Dickinson et al., 1998 ). They implement skills for the management of interpersonal and team relationships, the teacher acting as a guide and counselor during the learning process (Kolmos, 2012 ; Thomas, 2000 ). This allows students to think about their proposals, develop them and become aware of the process itself and everything that it implies beyond the results achieved (Brundiers and Wiek, 2013 ; García et al., 2010 ).

In this way, the acquisition of social skills, empathetic behavior, dialog and listening (Belland et al., 2006 ), the development of critical and reflective thinking (Mergendoller et al., 2006 ) is favored by activating competencies such as collaboration, decision-making, organization and group responsibility (Blank, 1997 ; Dickinson et al., 1998 ), contributing to the development of a more motivating and participatory learning climate (Lima et al., 2007 ).

This methodological aspect requires, in parallel, the review of the evaluation systems; it appears as necessary to leave behind the traditional cumulative models to introduce a new model of more formative, shared and authentic evaluation that is able to guarantee a greater involvement of the students in the development of their and their peer’s learning process (Brown and Race, 2013 ). An authentic evaluation offers the students opportunities to learn through the evaluation process planned and directed by the teacher. When the evaluation system is carefully designed to articulate with the learning results that are expected to be achieved, it is possible to obtain benefits in terms of greater participation and helps students to advance in the development of their knowledge, skills and attitudes (Brown, 2015 ).

Cooperation as the basis of project-based learning

One of the essential aspects of developing the PBL is the management of cooperation between the group participants, an aspect that must be guaranteed and supervised by offering sufficient feedback (Thomas, 2000 ). For Orlick ( 1986 ) cooperation is directly related to communication, cohesion, trust and skills development for positive social interaction.

However, Díaz-Barriga and Hernández ( 2002 ) consider that group work, which teachers frequently launch in project initiatives, does not necessarily implies true cooperation and there are many interpersonal problems that students face (Prince and Felder, 2006 ). This aspect prevents a real learning of collaboration and its application in action to address the shared phase of project management.

Burdett ( 2007 ) considers that, sometimes within the group, interpersonal relationships are strained since participation in group work involves much more than each member’s knowledge on a given subject: It involves listening, negotiating, giving in; ultimately, skills that favor the dynamics of group work. Such situations of tension and intragroup crises jeopardize the assignment to be developed and the effectiveness of group synergy, as established by Del Canto et al. ( 2009 ), Jhen and Mannix ( 2001 ), Kerr and Bruun ( 1983 ), Putnam ( 1997 ), and Velázquez ( 2013 ) and those are grouped around five critical dimensions: Differences in individual capacities to complete assignments, resulting in the stowaway effect ; imbalance in the functions to be performed; early abandonment in completing assignments due to unresolved discrepancies; struggle to make one’s own ideas prevail and lack of communicative skills.

Also for Kerr and Bruun ( 1983 ) and Slavin ( 2014 ) tensions arise from the lack of a follow-up by the teacher in the group work process entrusted to their students, not monitoring the performance and contribution of each member by thriving the aforementioned stowaway effect, imbalances in workloads borne by each member and unresolved crises in interpersonal relationships, not benefiting the task management, the project development and its fair evaluation.

Intragroup conflicts often cause widespread student complaints, lack of motivation, frustration, and occasionally, a preference for individual work that does seem to guarantee the fair evaluation of the assignment (Gámez and Torres, 2012 ; McConnell, 2005 ).

That is why establishing initial cooperative learning dynamics to learn how to collaborate, assume new responsibilities, communicate and assertively express ideas (Velázquez, 2013 ), is essential to get started in the PBL methodology. Johnson et al. ( 1999a ) define cooperative learning as a work-based methodology in small, usually heterogeneous groups in which students work together to improve their own and other member’s learning.

Several authors understand cooperative learning as an active methodology that favors the reflection of students while completing the assignment; not only des it allow to achieve academic goals, but also social objectives, it stimulates interaction through the proposal of small groups and guides the realization of a type of group work, structured and monitored, to favor the learning of all the members of the group without exception (Dyson, 2002 ; Johnson et al., 1999b ; Kagan, 2000 ; Pujolàs, 2009 ).

According to Johnson and Johnson ( 1999 ) the management of cooperative learning by teachers requires, for its effectiveness, guarantees in the management of positive interdependence, making the students understand that work benefits colleagues by prioritizing “us” over “I”, proactive interaction, individual responsibility, interpersonal skills, and group processing at the end of the work sessions performed.

The teacher establishes a structured process of true cooperation easing the development of academic objectives, but also other competitive objectives: cooperation, communication, social skills (Walberg and Paik, 2002 ).

It is important to note in this regard the role of the evaluation on the projects implemented, developed and presented. Pérez-Pueyo and López-Pastor ( 2017 ) propose a model of formative evaluation through the use of cooperative projects, in which a further step is taken in the autonomy of the students by fully involving them in the teaching process through shared tutoring, especially when the realization of projects that require a lot of involvement or levels of complexity in their realization is encouraged. In addition, the use of tools such as auto evaluations and group co-evaluations (Hamodi et al., 2015 ), allow the teacher to give more effective feedback during the process, based on the information provided by the students.

Based on the contributions of the various authors cited above, who understand cooperative learning as an active methodology that allows students to achieve not only academic goals but also social objectives, thus promoting the learning for all the students without exceptions, the present study aims to achieve the following objectives: Understanding the functioning of cooperative work present in the development of the operational dynamics of the PBL launched.

Knowing how the formative evaluation process develops in the operational dynamics of the PBL.

Participants and context

The study included 16 students on their fourth year of Secondary Education (with an average of 15 years old, 8 females and 8 males) attending Cristo Rey Polytechnic Institute in the city of Valladolid, and taking the elective subject of Economics. Also three male teachers and two female teachers (ages [35–57]) who teach at the same center and stage, in which they apply PBL as an active methodology. All procedures were in accordance with the ethical standards of the institutional and national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

During the development of the research, the ability of students to work through PBL was tested, applying the academic project entitled My Business Plan , throughout the subject of Economics in the compulsory secondary education stage. The students were arranged in groups of 4 to 5 members with different capacities and potentialities.

These heterogeneous groups allowed the development of various skills by the students, with the intention of improving them together with intragroup interpersonal relationships.

Data collection and information analysis tools

An in-depth interview was designed for teachers who were to some extent incorporating PBL as an active methodology in the development of their subjects. They thus form a representation of the faculty imparting subjects such as Economics, Geography and History, Biology and Geology, Physics and Chemistry and Philosophy. At the same time, an open-question questionnaire was designed for students. Finally, a reflexive diary was drafted in which observations were recorded from the experiences carried out in class.

In relation to the analysis of the information obtained, the ATLAS.ti software has been used, confectioning a work of textual analysis of the transcripts of teachers’ interviews, the answers on the open questions of the questionnaire answered by the students, alongside with the teacher’s own reflexive diary.

On the three primary documents, a coding process is carried out inductively and deductively through two cycles (Miles et al., 2014 ). Thus, during the process, a constant circular relationship between the codes already obtained and the new ones I created, refining the concepts, grouping them, to infer in higher-level constructs as groups of explanatory codes (Kalpokaite and Radivojevic, 2019 ).

The codes obtained during the first coding cycle were analyzed critically and independently by the four researchers participating in the study establishing a thoughtful debate. Continuous feedback between researchers and their ongoing participation in the regeneration and refinement of codes and groups of codes supported the credibility, reliability and transparency of the research (Neal et al., 2015 ).

It was considered that saturation had been reached at the time where comparisons between the data ceased to show new relationships and properties between them, depleting that representative wealth of a circular analytical process (Flick, 2007 ).

In order to address the credibility aspects of the research in relation to the interpretative difficulties of the phenomenon studied (Lincoln and Guba, 1985 ), a structure of prolonged over time experimentation was developed, with the presence of the researcher at the location, maintaining the same methodological order, establishing her figure as an observer teacher during the time of research; in the analysis of the data, a process of triangulation was developed from the three aforementioned sources of documentary data, this allowing the contrast of the discoveries.

Forty-one explanatory codes of the phenomenon under investigation were established and grouped around four categories: Learning, interaction-collaboration, motivation, organization.

The use of the ATLAS.ti software as a code co-coordinate tool was convenient, allowing to observe how four codes of the categories Learning and Interaction-Collaboration related to each other: cooperation, conflicts, evaluation and project. Their relational study allows to reflect critically on the several handicaps found and whose consideration is essential for the applicability of the practice.

Thus, to address the first objective of the study—knowing the functioning of cooperative work in the development of the operational dynamics of the PBL launched—taking as a starting point the perceptions of the teachers interviewed and the relationships they establish between PBL and cooperation, they show a formula of practical application using cooperative structures in the form of small groups, which they consider makes it easier for students to encourage communication, to develop skills for interpersonal relationships, as well as individual and group responsibility in the fulfillment of the assignments proposed.

(…) I mix it at first with cooperative work, with small groups, with cooperative structures because being such dense subjects (…) and at the end of the school year, the last quarter, we already work on the project (Male Teaching Interview. 4:69).

In the groups, the smaller the better they work, (I would recommend) four tops, like last year (…) this allows everyone to work, if they are too many, the tasks get diluted and if there are very few, and it also happens sometimes, if one is sick or misses class for some reason for too many days, the groups gets resented… then it rally allows to work on relationships and influences the quality of learning very clearly by what I say… one is good at one thing, the other is good at some other thing, and they end up learning from each other (Male Teaching Interview. 4:358).

The same teacher considers, in the application of the methodology, the creation of small working groups, defending this formula as very valuable to develop the communicative and negotiation abilities to reach agreements and coordinate with others, the students winning from an experiential point of view, in socialization and interaction resources.

I divided the class into 4 groups of 4 students each (…), they had ten minutes to explain in front of the rest of the classmates what their business model was by answering various questions. (…) the idea of the project is that they are the ones who work on this concept throughout the course and thus gradually become familiar with that environment and its vocabulary (Reflexive Diary. 3:394).

Through the PBL they work together, they talk more, they must agree on different aspects, and it requires coordination, that is, an effort of all of them, not depending so much on their individual abilities; this approach is very different from the master class, and I do believe that, from a social point of view, socialization develops more and better this way (Reflexive Diary. 4:412).

However, the same teachers interviewed acknowledge that, during the development of the methodology, applying group work strategies for cooperation, numerous frictions and interpersonal conflicts are often triggered within the working groups. A closer attention is put on those students who does not follow the intended pattern of behavior and unleash conflict because they do not assume or carry out their workload.

The most negative aspect are those students who do not want to participate, or find it difficult to participate, or do not get involved and seriously harm the group, and sometimes problems such as friction and conflicts can appear among them for this reason; working individually, logically, there is no such problem (Female Teacher Interview. 4:150).

That student who is a little lazier, they can take advantage of the group work situation so that others work a little for them (Male teacher interview. 4:343).

This aspect is also observed and recorded by the teacher in her reflexive diary, acknowledging incidents that are likely to occur in the groups, generating some interpersonal conflict and influence on group performance to carry out the tasks of the project.

There is a group of four boys who you have to tell off and who I do not intend to bring together in the future for the groups of the project (Reflexive Diary. 3:296).

Z (…) during group work he plays with the table, gets distracted by what other teammates do (…). I think he’s a boy who is too easily distracted and annoys his peers (Reflexive Diary. 3:160).

The students themselves consider that the project suffers when situations in which not all members of the group work in tune occur, creating imbalances in the effort made and in the management of the workloads and involvement assumed, which have an impact not only on the realization of the tasks and assignments and their final evaluation, but also on the intragroup climate.

I don’t like it when there’s someone in my group who doesn’t work and gets the same grade as me or we fail the project all because of him, because we don’t all work equally; sometimes I felt that if I didn’t tell them to do something, they wouldn’t do it (Student Questionnaire. 5:242)

There are groups where only one or two people work and it’s not fair. The rest of them get too comfortable and their work is minimal. I would try watching those who do not work, or not giving them the same grade (Student Questionnaire. 5:123)

When the members of the group do not work, the project can be a disaster; and if a person does not want to do their job then discussions arise; for me the experience is negative because I did work and I did it all by myself (Student Questionnaire. 6:134)

With regard to the second objective of the study, knowing how the formative evaluation process develops in the operational dynamics of the PBL, taking into account the teachers involved in the inclusion of PBL in their teaching practice, it seems to show a difficult development, recognizing the constant presence of tests and evaluations as a generalized tool of measurement of the acquired knowledge. However, it recognizes the value of other competence aspects that must necessarily be considered by applying tools that make it easier for students to raise awareness of the developed learnings, as well as the value of the teacher as a guide who oversees the learning process and controls and leads it.

Evaluation is a complex topic because if you base your work on projects and in the end you give them an exam you are giving more value to the contents and not so much to everything else; that is why for the final evaluation we are already working on taking into consideration the valuable opinions of each one, that of the classmates, the ones shared among students and teachers through auto evaluation practices, co-evaluation and heteroevaluation. In this way they develop their critical ability, their capability to value themselves and others (Male Teacher Interview. 4:323).

I like as a teacher to supervise how they perform the practice of PBL, if everyone works and contributes; then I believe that this work is done in front of them (Male teacher interview. 4:442).

When one works in a group within the classroom the relationship between the students and the teacher is reinforced because they are no longer seen as a figure of authority or a superior, but as a guide who knows, who helps, who collaborates with them and listens to them (Female Teacher Interview. 4:388).

The same teacher in her reflexive diary mentions the use of evaluation practices such as co-evaluation allowing the students to express themselves in order to participate and getting them involved through paper presentations and consequent evaluation between classmates; she also references the heteroevaluation allowing the time for student-teacher dialog based on the assignments and a proposal to solve the project addressed.

What I want is for them to work a little bit and, to make sure of that, as they develop the eight sections on their project, they must make a presentation in front of the rest of their classmates that will be evaluated by themselves and commented by the rest of us (Reflexive Diary. 3:701).

Once the presentations were completed, I gave each group a questionnaire to conduct a co-evaluation on the project addressed; for this evaluation, each group would evaluate the work presented by the other groups, grading representatively each of the sections of the project, so that we could have several grades to be used for the final evaluation of the project (Reflexive Diary. 3:335).

To conclude, the students recognize certain limitations in the evaluation of their work, mainly in a key of a non-follow-up of the process established in the classroom to address the project and the assignments required. They propose solutions to develop a greater control on those people in the group who do not contribute in the realization of the aforementioned assignments, as well as a better management of the final grade that, being the same for the whole group, is detrimental, in their perception, to the formation of a fair value in relation to the unequal effort made. Sometimes the proposed solutions are oriented in an opposite direction to the cooperative spirit that the PBL promotes.

The way I would solve the problem of those colleagues who take advantage of the work of others when working as a group is to set them alone to work; to do their own project; that way, at least they would control those who do not work (Student Questionnaire. 5:168).

As a positive experience, I find working with projects more enjoyable and entertaining; the most negative thing is that it is almost never worked equally, and approximately the same grade is received. It is better to grade individually instead of having a final group grade (Student Questionnaire. 3:356).

The problem with those classmates who take advantage of other’s work when working in a group I would solve by telling the teacher, and giving an individual grade on each assignment done by each group member, specifying who did what (Student Questionnaire. 5:206).

When teaching methods such as PBL are used, in which the teacher poses a question, a challenge or a specific problem connected with the reality that students have to solve (Bell, 2010 ), the degree of involvement of these students seems to increase. In the teaching-learning process, they become the protagonists when they are invited to seek, assess, interpret and share information with the rest of the group members, and they apply a more critical way of thinking, since they are constantly and mutually questioned about why and what are they studying for.

In this sense, the students participate collaboratively in all the proposed assignments: understanding and interpretation of data, collection of information, preparation of partial deliveries, writing of the final report, and oral presentation before others, assessing the problem or challenge proposed with the intention of being able to draw their own conclusions.

In the implementation of these formative dynamics as an alternative to more traditional methodological models, a new way of generating and developing learning is consequently activated, applying a cooperative work model, being the management of group activity to face the project a vital aspect.

In relation to the cooperative dynamics of operation of the PBL experiences developed, the implementation of a methodological model is observed; this model is based, as a starting point, on cooperative structures by which the students are intended to address the project. Such structures materialize in the form of small and heterogeneous groups that seek to guarantee communication between their members (Johnson et al., 1999a ), unleashing a strongly competency learning model (Perrenoud, 2008 ) in which students have to combine the knowledge, skills and attitudes that they learn, in a shared way with their classmates, to face the assignments and carry out the project proposed and presented by the teacher (Bell, 2010 ; Thomas, 2000 ).

In the same way, intentionally, the dynamics proposed by teachers through this methodology intend to trigger learning situations in which negotiation, compromise, listening, agreement-reaching and coordination to make decisions and solve problems are aspects of interaction and socialization necessarily to be encouraged, as established by Belland et al. ( 2006 ) and Bender ( 2012 ).

However, there is a general concern about the management in the classroom of the cooperative structures placed in order to develop the project. Friction, conflicts inherent in group life and the consequence of the cooperation dynamics applied to establish in a shared way the action plan to address the entrusted project are recognized. They identify in certain students a lack of willingness for cooperation and commitment, aspects that generate intragroup tension that for Slavin ( 2014 ) is necessary to keep track of by the teacher during the learning process, for example, paying special attention to those situations in which the stowaway effect occurs (Kerr and Bruun, 1983 ; Slavin, 2014 ).

In this matter, the students themselves describe occasional imbalances in the efforts made to carry out the assignments, the weight of the workloads assumed and, ultimately, a certain lack of harmony when relating to each other when it comes to getting involved in the project. For Del Canto et al. ( 2009 ), Jhen and Mannix ( 2001 ), Putnam ( 1997 ), and Velázquez ( 2013 ) cooperation requires attention on these critical aspects during its development, benefiting the group climate itself and thus, the performance on the assignments. For Gámez and Torres ( 2012 ) and McConnell ( 2005 ), intragroup conflict provokes generalized complaints, loss of enthusiasm and motivation for group members, a source of arguments and frustration, an aspect present in the study in the voice of the students involved.

At the same time, the teaching staff, in relation to the evaluation of the formative dynamics based on the PBL put in place, recognize the importance of paying attention to various competency aspects inherent to the cooperative learning process obtained.

This aspect, in line with what is suggested by Blank ( 1997 ), Dickinson et al. ( 1998 ), Mergendoller et al. ( 2006 ) and Belland et al. ( 2006 ), materializes in the attention to capacities such as empathy, listening, critical thinking, collaboration, decision-making, group responsibility, the teacher assuming a role of leader and guide of all these during the process of learning, as considered by Thomas ( 2000 ), Walberg and Paik ( 2002 ) and Kokotsaki et al. ( 2016 ), supporting the maintenance of a more motivating, participatory and facilitating group work climate (Lima et al., 2007 ).

Despite the use of traditional evaluation dynamics presenting a more finalist nature, such as the test or exam, the teaching staff recognize the value of formative and shared evaluation tools, such as self-evaluation, co-evaluation and heteroevaluation. In this sense, it is observed in the group, not without difficulties (Ertmer and Simons, 2005 ) a certain appreciation for the involvement of the students in the evaluation process, giving them a voice to express their own perception through dynamics such as the presentation of resulting works and shared evaluation in this regard. Paradoxically, the students involved consider a certain lack of follow-up by the teachers on the assignments they carry out and that are a part of the project, in correlation with a conflictive management of the grade in this regard. For Pérez-Pueyo and López-Pastor ( 2017 ) it is necessary to take further steps in the autonomy and personal initiative of the students and their involvement in the evaluation process, the teacher being able to apply techniques such as auto-evaluation, peer evaluation, shared evaluation, self-grading and dialogued grading. The same authors, for example, advocate for intervening in a Secondary Education classroom by applying cooperative projects and final presentations of group papers or events preparation, tutoring in a shared way with their students and involving them in their—and other’s—learning process; The teacher can also complete the methodological initiative by developing group auto-evaluations and co-evaluations, the students evaluating the process of effecting the group assignments or the actual completion of the final presentations. Some recommended instruments to lead the aforementioned evaluation techniques are the group class diary, the auto-evaluation reports and the evaluation scales (Hamodi et al., 2015 ; Hernando et al., 2017 ).

In short, the PBL experience carried out contains all the technical elements to facilitate a learning model of the competence type, which addresses both knowledge and skills to carry out the assignments and to offer solutions to the problems inherent to the given project, as well as the abilities to do so jointly and cooperatively. However, it shows that the methodological practice proposed still suffers from a real follow-up on the group process set, establishing feedback means in the action itself, neglecting the potential conflicts that arise and the smooth completion of the assignments.

In relation to evaluation, the importance of a more formative evaluation model is recognized among the teachers involved, appreciating practices that activate the participation and involvement of students, although the weight of the final products continues to be relevant to the process itself.

Data availability

All data generated or analyzed during this study are included in this published article.

Barret T (2005) What is problem-based learning. In: O´Neill G, Moore S, McMullin B (eds) Emerging issues in the practice of university learning and teaching. AISHE, Dublin, pp. 55–66

Google Scholar  

Bell S (2010) Project-based learning for the 21st century: skills for the future. Clear House 83:39–43. https://doi.org/10.1080/00098650903505415

Article   Google Scholar  

Belland BR, Ertmer PA, Simons KD (2006) Perceptions of the value of problem-based learning among student with special needs and their teachers. Int J Problem-based Learn 1(2):1–18. https://doi.org/10.7771/1541-5015.1024

Bender W (2012) Project-based learning: differentiating instruction for the 21st Century. Corwin, California

Blank W (1997) Authentic instruction. In: Blank WE, Harwell S (eds.) Promising practices for connecting high school to the real World. University of South Florida, Tampa, pp. 15–21

Blumenfeld PC, Soloway E, Marx R (1991) Motivating Project-based learning: sustaining the doing, supporting the learning. Educ Psychol 26:369–398. https://doi.org/10.1080/00461520.1991.9653139

Brown S (2015) Authentic assessment: using assessment to help students learn. Electr J Educ Res Assess Eval 21(2):1–8. https://doi.org/10.7203/relieve.21.2.7674

Brown S, Race P (2013) Using effective assessment to promote learning. In: Hunt L, Chalmers D (eds.) University teaching in focus. A learning-centred approach. Routledge, London, pp. 74–91

Brundiers K, Wiek A (2013) Do we teach what we preach? An international comparative appraisal of problem- and project-based learning courses in sustainability. Sustainability 5(4):1725–1746. https://doi.org/10.3390/su5041725

Burdett J (2007) Degrees of separation balancing intervention and independence in group work assignments. Aust Educ Res 34(1):55–71. https://doi.org/10.1007/BF03216850

Del Canto P, Gallego I, López JM, Mora J, Reyes A, Rodríguez E, Valero M (2009) Conflictos en el trabajo en grupo: cuatro casos habituales. Revista de Formación e Innovación Educativa Universitaria 2(4):344–359

Díaz-Barriga F, Hernández R (2002) Estrategias docentes para un aprendizaje significativo. McGraw-Hill, Barcelona

Dickinson KP, Soukamneuth S, Yu HC, Kimball M, D’Amico R, Perry R, Kingsley C, Curan SP (1998) Providing educational services in the Summer Youth Employment and Training Program [Technical assistance guide]. U.S. Department of Labor, Office of Policy and Research, Washington

Dyson B (2002) The implementation of cooperative learning in an elementary school physical education program. J Teach Phys Educ 22(1):69–85

Ertmer PA, Simons KD (2005) Scaffolding teachers’ efforts to implement problem based learning. Int J Learn 12(4):319–328. https://doi.org/10.18848/1447-9494/CGP/v12i04/46447

Flick U (2007) El diseño de investigación cualitativa. Morata, Madrid

Gámez MJ, Torres C (2012) Las técnicas de grupo como estrategia metodológica en la adquisición de la competencia de trabajo en equipo de los alumnos universitarios. J Educ Teach Train 4:14–25

García I, Gros B, Noguera I (2010) La relación entre las prestaciones tecnológicas y el diseño de las actividades de aprendizaje para la construcción colaborativa del conocimiento. Cultura y Educación 22(4):395–418. https://doi.org/10.1174/113564010793351867

Hamodi C, López-Pastor V, López-Pastor AT (2015) Medios, técnicas e instrumentos de evaluación formativa y compartida del aprendizaje en educación superior. Perfiles Educativos 37(147):146–161

Hernández March A (2006) Metodologías activas para la formación de competencias. Educatio Siglo XXI 24:35–56

Hernando A, Hortigüela D, Pérez-Pueyo A (2017) El proceso de evaluación formativa en la realización de un video tutorial de estiramientos en inglés en un centro bilingüe. In: López-Pastor V, Pérez-Pueyo A (eds) Buenas prácticas docentes: Evaluación formativa y compartida en educación: experiencias de éxito en todas las etapas educativas. Universidad de León, León, pp. 260–271

Jhen KA, Mannix EA (2001) The dynamic nature of conflict: a longitudinal study of intragroup conflict and group performance. Acad Manag J 44(2):238–251. https://www.jstor.org/stable/3069453

Johnson D, Johnson R (1999) Aprender juntos y solos. Aprendizaje cooperative, competitivo e individualista. Aique, Buenos Aires

Johnson D, Johnson R, Holubec E (1999a) El aprendizaje cooperativo en el aula. Paidós, Buenos Aires

Johnson D, Johnson R, Holubec E (1999b) Los círculos del aprendizaje. La cooperación en el aula y la escuela. Aique, Buenos Aires

Jonassen D (2011) Supporting problem solving in PBL. Interdiscip J Problem-Based Learn 5(2):95–119. https://doi.org/10.7771/1541-5015.1256

Kagan S (2000) L´apprendimento cooperativo: L´approccio strutturale. Edizioni Lavoro, Roma

Kalpokaite N, Radivojevic I (2019) Demystifying qualitative data analysis for novice qualitative researchers. Qual Rep 24(13):44–57

Kerr NL, Bruun SE (1983) Dispensability of member effort and group motivation losses: free-rider effects. J Pers Soc Psychol 44(1):78–94. https://doi.org/10.1037/0022-3514.44.1.78

Kokotsaki D, Menzies V, Wiggins A (2016) Project-based learning: a review of the literature. Improving Schools 19(3):267–277. https://doi.org/10.1177/1365480216659733

Kolmos A (2012) Changing the curriculum to problem based and project-based learning. In: Yusof KM, Azli NA, Kosnin AM, Syed Yusof SK, Yusof YM (eds) Outcome-based science, technology, engineering, and mathematics education: innovative practices. IGI Global, pp. 50–61

Krajcik JS, Blumenfeld PC (2006) Project-based learning. In: Sawyer K (ed) Cambridge handbook of the learning sciences. Cambridge University Press, pp. 317–334

Lima RM, Carvalho D, Flores M, Van Hattum-Janssen N (2007) A case study on project led education in engineering: student´s and teacher´s perceptions. Eur J Eng Educ 32(3):337–347

Lincoln YS, Guba EG (1985) Naturalistic inquiry. Sage, Beverly Hills

Book   Google Scholar  

Martín A, Rodríguez S (2015) Motivación en alumnos de Primaria en aulas con metodologías basadas en proyectos. Revista de Estudios e Investigación en Psicología y Educación Extr 1:58–62. https://doi.org/10.17979/reipe.2015.0.01.314

McConnell D (2005) Networked e-learning groups. Stud High Educ 30(1):25–42. https://doi.org/10.1080/0307507052000307777

Mergendoller JR, Maxwell NL, Bellisimo Y (2006) The effectiveness of problem-based instruction: a comparative study of instructional methods and student characteristics. Interdiscip J Problem-Based Learn 1(2):49–69. https://doi.org/10.7771/1541-5015.1026

Miles MB, Huberman AM, Saldaña J (2014) Qualitative data analysis. A methods sourcebook. Sage, London

Neal JW, Neal ZP, VanDyke E, Kornbluh M (2015) Expediting the analysis of qualitativedata in evaluation: a procedure for the rapid identification of themes fromaudio recordings (RITA). Am J Eval 36(1):118–132. https://doi.org/10.1177/1098214014536601

Orlick T (1986) Juegos y deportes cooperativos. Editorial Popular, Madrid

Pérez-Pueyo A, López-Pastor V (2017) El estilo actitudinal como propuesta metodológica vinculada a la evaluación formativa. In: López-Pastor V, Pérez- Pueyo A (eds) Evaluación formativa y compartida en educación: experiencias de éxito en todas las etapas educativas. Universidad de León, Secretariado de Publicaciones, León, pp. 240–259

Perrenoud P (2008) Construir las competencias, ¿es darle la espalda a los saberes? Revista de Docencia Universitaria 6(11):1–8

Prince MJ, Felder M (2006) Inductive teaching and learning methods: definitions, comparisons and research bases. J Eng Educ 95(2):123–138. https://doi.org/10.1002/j.2168-9830.2006.tb00884.x

Pujolàs P (2009) La calidad en los equipos de aprendizaje cooperativo. Algunas consideraciones para el cálculo del grado de cooperatividad. Revista de Educación 349:225–239

Putnam J (1997) Cooperative learning in diverse classrooms. Prentice-Hall, Upper Saddle River, NJ

Sánchez N (2018) Clase invertida y aprendizaje basado en proyectos en el aula de biología. Un proyecto de innovación para 1° de la ESO. Valoración de la experiencia. Enseñanza Teaching 36(1):81–11. https://doi.org/10.14201/et21836181110

Slavin RE (2014) Cooperative learning and academic achievement: why does groupwork work? Anales de Psicología 30(3):785–791. https://doi.org/10.6018/analesps.30.3.201201

Thomas JW (2000) A review of research in project-based learning. Autodesk Foundation, San Rafael, CA

Velázquez C (2013) La pedagogía de la cooperación en educación física. Kínesis, Armenia

Walberg H, Paik S (2002) Prácticas eficaces. Magisterio, Bogotá

Download references

Acknowledgements

We gratefully acknowledge the support of the teachers, school managers and the children that participated for their collaboration.

Author information

Authors and affiliations.

Department of Pedagogy, Faculty of Education and Social Work, University of Valladolid, Paseo de Belén, 1, 47011, Valladolid, Spain

Berta de la Torre-Neches & Mariano Rubia-Avi

Deparment of Physical Education, Faculty of Education, University of Valladolid, Plaza de la Universidad, 1, 40005, Segovia, Spain

Jose Luis Aparicio-Herguedas

Department of Methods of Research and Diagnosis in Education I, National Distance Education University, Juan del Rosal, 14, 28040, Madrid, Spain

Jairo Rodríguez-Medina

You can also search for this author in PubMed   Google Scholar

Contributions

The authors contributed equally to this work.

Corresponding author

Correspondence to Berta de la Torre-Neches .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

de la Torre-Neches, B., Rubia-Avi, M., Aparicio-Herguedas, J.L. et al. Project-based learning: an analysis of cooperation and evaluation as the axes of its dynamic. Humanit Soc Sci Commun 7 , 167 (2020). https://doi.org/10.1057/s41599-020-00663-z

Download citation

Received : 22 September 2020

Accepted : 05 November 2020

Published : 08 December 2020

DOI : https://doi.org/10.1057/s41599-020-00663-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

• Open access
• Published: 06 January 2022

The key characteristics of project-based learning: how teachers implement projects in K-12 science education

• Anette Markula 1 &
• Maija Aksela   ORCID: orcid.org/0000-0002-9552-248X 1  

Disciplinary and Interdisciplinary Science Education Research volume  4 , Article number:  2 ( 2022 ) Cite this article

42k Accesses

39 Citations

7 Altmetric

Metrics details

The aim of this multiple-case study was to research the key characteristics of project-based learning (PBL) and how teachers implement them within the context of science education. K-12 science teachers and their students’ videos, learning diaries and online questionnaire answers about their biology related PBL units, within the theme nature and environment, were analysed using deductive and inductive content analysis ( n  = 12 schools). The studied teachers are actively engaged in PBL as the schools had participated voluntarily in the international StarT programme of LUMA Centre Finland. The results indicate that PBL may specifically promote the use of collaboration, artefacts, technological tools, problem-centredness, and certain scientific practices, such as carrying out research, presenting results, and reflection within science education. However, it appeared that driving questions, learning goals set by students, students’ questions, the integrity of the project activities, and using the projects as a means to learn central content, may be more challenging to implement. Furthermore, although scientific practices had a strong role in the projects, it could not be defined how strongly student-led the inquiries were. The study also indicated that students and teachers may pay attention to different aspects of learning that happen through PBL. The results contribute towards a deeper understanding of the possibilities and challenges related to implementation of PBL and using scientific practices in classrooms. Furthermore, the results and the constructed framework of key characteristics can be useful in promoting research-based implementation and design of PBL science education, and in teacher training related to it.

Introduction

Project-based learning (PBL) can be a useful approach for promoting twenty-first century learning and skills in future-oriented K-12 science education. PBL refers to problem-oriented and student-centred learning that is organised around projects (Thomas, 2000 ). This means that the intended learning of new skills and content happens through the project that students carry out in groups (Condliffe et al., 2017 ; Parker et al., 2013 ; Thomas 2000 ). Thus , PBL can be described as a collaborative inquiry-based teaching method where students are integrating, applying and constructing their knowledge as they work together to create solutions to complex problems (Guo et al., 2020 ). It is important that students practice working like this at school, as future generations will need to be able to overcome global environmental problems. As such, science education has to equip students with deeper learning instead of simple memorising of facts; students need the ability to apply their scientific knowledge in situations requiring problem-solving and decision-making (Miller & Krajcik, 2019 ).

PBL relies on four significant ideas from learning sciences: learning is most effective when students (1) construct their understanding actively and (2) work collaboratively in (3) authentic learning environments, whilst being sufficiently scaffolded with (4) cognitive tools (Krajcik & Shin, 2014 ). Compared to traditional teacher-led instruction, PBL has been found to result in greater academic achievement (Chen & Yang, 2019 ; Balemen & Özer Keskin, 2018 ). Additionally, it has been shown to improve students’ skills in critical thinking and question-posing (Sasson et al., 2018 ). There is also some evidence that PBL might contribute to developing students’ intra- and interpersonal competencies (Kaldi et al., 2011 ).

Within science and technology education, one of the key benefits of PBL is arguably immersing students in using scientific practices, such as asking questions (Novak & Krajcik, 2020 ). Whilst various approaches can be taken to PBL, scientific practices are often considered as one of its key characteristics (see Table  1 for discussion about the key characteristics of PBL). The idea is that in PBL, students should participate in authentic research in which they use and construct their knowledge like scientists would (Novak & Krajcik, 2020 ). Using scientific practices has been found to contribute towards students’ engagement when learning science (Lavonen et al., 2017 ), and PBL does indeed appear to have a positive impact on students’ attitudes and motivation towards science and technology (Kortam et al., 2018 ; Hasni et al., 2016 ). PBL allows students to see and appreciate the connection between scientific practices and the real world, significance of learning, carrying out investigations and the open-endedness of the problems under investigation (Hasni et al., 2016 ).

Nevertheless, according to the review done by Condliffe et al. ( 2017 ), the efficacy of PBL in terms of student outcomes is not entirely clear. In a more recent review, however, Chen & Yang ( 2019 ) found more distinctive benefits to learning compared to previous studies. As they suggest, it may be that implementation of PBL has developed between 2000 and 2010, potentially owing to the better availability of training programmes and materials. Nonetheless, whilst Chen & Yang ( 2019 ) did find that PBL improves students’ academic achievement in STEM (science, technology, engineering and mathematics), they also found that the positive effect of PBL appeared to be somewhat bigger in social sciences compared to STEM subjects. Additionally, the various distinctions between different researchers for what makes PBL different from other closely related instructional approaches, such as inquiry-based and problem-based learning, make it challenging to confidently determine exactly how effective PBL is as an instructional method (Condliffe et al., 2017 ).

However, PBL is supported by governments, researchers, and teachers in many countries (Novak & Krajcik, 2020 ; Condliffe et al., 2017 ; Aksela & Haatainen, 2019 ; Annetta et al., 2019 ; Hasni et al., 2016 ) . Studies have found that teachers consider PBL as an approach that promotes both students’ and teachers’ learning and motivation, collaboration and a sense of community at school level, student-centred learning, connects theory with practice and brings versatility to teachers’ instruction (Viro et al., 2020 ; Aksela & Haatainen, 2019 ). However, regardless of teachers’ enthusiasm towards PBL, they can still struggle with its implementation (Tamim & Grant, 2013 ). PBL is a challenging method to use in practice, as it requires a fundamental understanding of its pedagogical foundations (Han et al., 2015 ), and it appears that teachers tend to have limited and differing conceptions about PBL (Hasni et al., 2016 ). For example, PBL is often defined through its distinct characteristics (Hasni et al., 2016 ; Thomas, 2000 ), but these tend to be unknown to teachers (Tamim & Grant, 2013 ). What is more, research has indicated that in order for PBL to be implemented as it is described by researchers, teachers require training and multiple years of practice with it (Mentzer et al., 2017 ). In fact, students display greater learning gains when their teacher is experienced with PBL (Capraro et al., 2016 ; Han et al., 2015 ), and it appears that partial or incorrect implementation of PBL may even have negative consequences for students’ academic performance (Capraro et al., 2016 ; Erdoğan et al., 2016 ).

Both Viro et al. ( 2020 ) and Aksela & Haatainen ( 2019 ) found that according to STEM teachers, the most challenging aspects of implementing PBL are project organisation (for example, time management), technical issues, resources, student-related challenges and collaboration (Viro et al., 2020 ; Aksela & Haatainen, 2019 ). As PBL requires students to study a certain phenomenon in detail by using scientific practices, it takes longer than more traditional approaches (Novak & Krajcik, 2020 ). Researchers have also reported that teachers consider irrelevance to subject teaching and an unfamiliar teaching style among the significant negative aspects of PBL (Viro et al., 2020 ). Implementation of PBL should focus on teaching twenty-first century skills, being student-centred, and building strong and personal interaction between students and teachers (Morrison et al., 2020 ). This requires both teachers and students to take on new roles. In PBL, teachers are often having to act simultaneously as designers, champions, facilitators and managers, and students are expected to be self-directed learners who are able to endure the ambiguity and open-endedness of PBL projects (Pan et al., 2020 ).

Despite the move towards student-centred approaches (for example, inquiry-based teaching) in many national curricula, such as in the United States (National Research Council, 2012 ), Finland (Lähdemäki, 2019 ) and throughout much of Europe (European Commission, 2007 ), there is a distinct lack of research about PBL that is initiated by teachers (Condliffe et al., 2017 ). There is very little research into how teachers understand and use PBL when they are not guided by university researchers, and the models they develop for its implementation (Hasni et al., 2016 ). It is also important to research what kinds of changes teachers make to PBL curricula to adapt them to their classes, and how this process could be supported (Condliffe et al., 2017 ). Often the reality in classrooms differs from the visions in curricula (Abd-El-Khalick et al., 2004 ), and simply reforming the science curricula does not mean that teachers understand how to implement the new concepts into their teaching (Severance & Krajcik, 2018 ). In order to gain a better understanding of how teachers implement PBL and the related possibilities and challenges in practice, and to promote the use of PBL in education, PBL units from K-12 schools were studied from the perspective of key characteristics of PBL. The studied schools were from several different countries and they all had participated in the international StarT programme ( https://start.luma.fi/en/ ) by LUMA Centre Finland (see ‘Participants’).

Key characteristics of PBL

Most projects done at schools are not considered to be PBL, as PBL is often defined more specifically through its distinct characteristics (Hasni et al., 2016 ; Thomas, 2000 ), also referred to as ‘design principles’ (Condliffe et al., 2017 ). However, there is still ambiguity among researchers about what the exact key characteristics or design principles of PBL are (Condliffe et al., 2017 ; Hasni et al., 2016 ). Krajcik & Shin ( 2014 ) propose the following six features as key characteristics of PBL: (1) driving question, (2) learning goals, (3) scientific practices, (4) collaboration, (5) using technological tools, and (6) creating an artefact. These characteristics, including their purpose and features, have been discussed based on the literature review in Table 1 .

In this study, the PBL units were researched by using the six key characteristics found in Table 1 as a framework (Krajcik & Shin, 2014 ). The categories in the content analysis (see Table  2 in ‘Methods’) were based on these characteristics. At the time of doing the analysis, the model proposed by Krajcik & Shin ( 2014 ) was the most recent and detailed description of the characteristics of PBL that allowed study into the quality of the PBL units in practice. Additionally, their framework is in line with the views of other authors who focused on the characteristics of PBL, including the recent systematic review by Hasni et al. ( 2016 ) into the characteristics of STEM PBL used by researchers, and with the reviews done by for example, Condliffe et al. ( 2017 ) and Thomas ( 2000 ). However, in order to study the quality of PBL units under each of the characteristics, the framework was developed further by using the most current literature. For example, the phases of inquiry-based learning (Pedaste et al., 2015 ) were used to study how scientific practices were carried out by the schools.

Most earlier science education studies have looked at teachers’ perceptions of PBL through questionnaires and interviews (Hasni et al., 2016 ), but this study analysed teachers and students’ reports of their projects in practice. Considering the widely recognised challenges in the implementation of PBL, and the shift in many national curricula towards PBL and similar approaches, there is an urgent need to understand how teachers are managing the change, and what kinds of models they are developing for the implementation of the new curricula in their classrooms. The aim of this study is to understand possibilities and challenges related to the implementation of PBL in practice through the key characteristics (Table 1 ). The detailed research questions are: (1) Which key characteristics of PBL do teachers implement in the projects? and (2) How do teachers implement these characteristics in practice?

This study was carried out as a multiple-case study (Yin, 2014 ) on schools that participated in the international StarT programme by LUMA Centre Finland from different countries. A multiple case study allows for comparison between the differences and similarities between the cases (Yin, 2014 ), and therefore to gain a preliminary idea of characteristics or issues that might be common across the schools. The PBL units of twelve K-12 schools were studied (see ‘Participants’ for further details on the selection criteria). The schools participated in the international StarT competition organised by LUMA Centre Finland ( https://start.luma.fi/en/ ) during the academic year of 2016–17 or 2017–18.

The StarT programme

StarT encourages teachers to share their best models for implementing PBL, and students to present the products and research they have done within their groups (StarT programme). The competition has two categories: teachers’ descriptions of the PBL units that were carried out by the schools (‘ best practices’ ), and ‘ students’ projects ’ that describe what individual student groups studied, created and learned during the school’s PBL unit. Each school was able to upload one entry to the teachers’ category, describing the implementation of the project unit from teachers’ perspective as a best practice for other schools, and an unlimited number of students’ projects related to this unit. As such, each ‘ student project ’ is part of the same PBL unit organised by the school, but it describes what one student group produced under the PBL unit implemented by the teachers. Depending on the school and how much freedom the students had in the PBL unit, the student groups might have had completely different research topics, or they might have just produced slightly different artefacts to the same problem.

To participate in each category, the schools needed to upload a three-minute-long video describing the best practice or the project and to answer questions on an online form. Additionally, student groups were required to upload a learning diary, the format of which could be freely chosen. As such, the schools had significant freedom in terms of what they wanted to report about their PBL units. At the time of the data collection, the participants did not receive any professional development training from StarT, but depending on how closely they followed the online channels of StarT, they had access to project ideas and videos from other participants via the programme website, and the programme also included voluntary webinars and newsletters. However, these materials were freely available to anyone on the internet, and participating in the competition did not require any other engagement with the StarT programme.

Content analysis

Deductive content analysis is suitable for research that aims to study an existing model or theory (Hsieh & Shannon, 2005 ). The key characteristics of PBL shown in Table 1 were used as a basis for the deductive and inductive content analysis, where it was determined which characteristics teachers implemented in the projects, and how they did this. In qualitative content analysis, data is analysed by reducing it to concepts that describe the studied phenomenon, for example, through pre-defined categories, whilst also acknowledging the themes rising from the data (Elo et al., 2014 ; Cohen et al., 2007 ). The final categories used in the deductive analysis, and discussion about decisions regarding them, can be seen in Table 2 . The data was looked at inductively within these categories (Marshall & Rossman, 2014 ). An example of the coding combining inductive and deductive content analysis is given in Table  3 .

The analysed materials ( n  = 12 project units and n  = 17 students’ projects; see details under ‘Participants’ and in Table  5 ) were written responses to questions on an online form, videos and learning diaries. The units considered in the analysis were words, sentences, and paragraphs from verbal communication. As the students’ projects were what individual student groups produced within the PBL unit of the school, all of the materials provided by an individual school were considered as an entity when studying how the school carried out PBL. Therefore, there was no differentiation between the source of the information (for example, learning diary or best practice video) but instead all materials from a single school were treated as equal evidence of how the characteristics of PBL were implemented (see Table 3 ). However, since two schools provided multiple student groups’ works as student projects, and there were differences in the approaches that different student groups took to carrying out their project work, also the number of student projects displaying each of the key characteristics is included in Table  6 under ‘Results’.

In order to see how the six key characteristics of PBL were distributed across the projects, the overall frequencies of characteristics displayed in a project unit (1 = present, 0 = not present) were counted. Table  4 displays the sections from the coding framework that were included in the frequency count. Each row in the second column was counted as ‘1’ if it was observed and as ‘0’ if it was not. Including these features in the frequency count allows a satisfactory picture of the distribution of the key characteristics across the studied schools to be drawn (See Fig.  1 and Table 6 ). Scientific practices are emphasised in the count due to their many subcategories, but this was deemed appropriate since they are a good indication of how inquiry-based and student-led the projects were. Learning goals and gains have a significant role too, but their role is similarly justified by their importance – they determine largely whether the projects have resulted in their intended purpose, learning. The results regarding the implementation and distribution of the key characteristics can be found under ‘Results’.

In order to improve the reliability and validity of the study, triangulation was employed (Turner et al., 2017 ) through the use of different types of materials as sources of information. This increases the reliability of studies looking at human behaviour (Cohen et al., 2007 ) and case studies (Yin, 2014 ), as that allows cues from different sources to be combined into a more representative image of a case (Baxter & Jack, 2008 ). Firstly, the materials consisted of three different types of media: written descriptions and answers to questions on an online form, videos, and a learning diary, the medium of which was not pre-defined for the participants. Secondly, the studied schools only consisted of learning communities that had participated in both the teacher category of StarT with a ‘best practice’ (a description of the PBL unit from teachers’ point of view) and the student category with at least one ‘student project’ (description of the work one student group did during the PBL unit). As such, this study includes the viewpoints of both teachers and students. Additionally, the results from coding were agreed upon by both of the authors.

Participants

The study analysed students’ projects and teachers’ best educational practices at K-12 school level ( n  = 12 project units and n  = 17 students’ projects; see Table 5 for details) that were implemented in 2016–2017 or 2017–2018. The projects were mostly ( n  = 9) created and implemented by teachers and students, and as such they reflect the reality of schools when it comes to implementing PBL. Only n  = 3 schools mentioned that they had participated in a (university-led) development programme. As such, the studied PBL units provide a plausible reflection of the reality of active teachers implementing PBL (see ‘Limitations’ for further discussion).

The studied PBL units within the theme ‘Nature and environment’ were chosen from the learning communities that participated in the international StarT programme in 2016–2017 and in 2017–2018. The other themes that the StarT participants could choose for their projects were ‘Technology around us’, ‘Mathematics around us’, ‘This works! A mobile toy’, ‘Stars and space’, ‘Well-being’, ‘Home, culture and internationality’. ‘Nature and environment’ was the most popular single theme during both years of data collection: n  = 132 learning communities from all n  = 277 learning communities indicated that they had done a project related to it in 2016–2017, and n  = 50 out of n  = 229 in 2017–2018. Whilst the studied projects focus on the theme ‘nature and environment’ in the context of biology education, the interdisciplinary nature of the theme makes the results largely applicable for other sciences. The decision to base the study on a single discipline was made in order to gain a more detailed understanding of the implications of STEM PBL for subject teaching; the case in this study focusing on teaching biology through PBL.

The first criteria in selecting the cases for this study was to include only PBL units implemented by K-12 school (ages 7 to 18). Additionally, only projects themed ‘Nature and environment’, where biology had a clear role, were included. Finally, only schools that had provided full sets of materials used in the analysis (written responses, videos and learning diaries) were included. Full sets of materials were required for both teachers’ descriptions of the PBL unit and students’ projects, either in English or Finnish (one school had to be excluded due to an insufficient level of English).

Table 5 presents participants and their school levels: 12 schools matched the criteria described above. In total, 12 project units and 17 students’ projects were analysed, with only two of the schools having provided more than one student project as a part of the project unit. 11 of the studied schools were from six different countries in Europe, and one school was from Southwest Asia. Schools D, E and F (Table 5 ) participated in the same PBL development programme implemented by a local university.

The participants gave permission for using their materials for research purposes upon their participation in StarT. However, as this study looks at the projects from an evaluative perspective, direct quotations or detailed descriptions of individual cases that could be used to identify the schools were not included.

The results for each of the research questions (see end of the chapter “Key characteristics of PBL”) will be presented separately.

(1) The key characteristics of PBL in the projects

The most frequently displayed key characteristics of PBL were collaboration, artefacts, technology, problem-centredness, and out of scientific practices, carrying out research, presenting results and reflection (see Table 6 for more detail). At least some form of collaboration (either between the students, between teachers or with outside partners) took place in all but one of the schools. Any interaction that the schools described as having taken place between different actors was considered as collaboration. Furthermore, technology was used as a part of the projects in all of the schools. Artefacts were also created in all of the studied projects. The results for each of the characteristics are summarised in Table 6 (research question 1), which also outlines how they were implemented (research question 2). As n  = 2 schools provided multiple projects by different student groups, the number of projects ( n  = 17) is higher than the number of schools ( n  = 12).

Regarding scientific practices that students participated in, presenting results (n = 12 schools), interpreting results ( n  = 11) and reflection ( n  = 10) were most commonly demonstrated. However, not all schools ( n  = 4) displayed clearly that students had done any research (such as searching for information, observation and collecting data). As testing hypotheses was not visible in any of the projects ( n  = 0), according to the definition of Pedaste et al. ( 2015 ), the research was considered as” exploration” ( n  = 8) instead of” experimentation” (n = 0). Only n = 4 schools included a mention of students having presented questions that had an impact on the course of the project or the investigations that were carried out.

Driving questions and learning goals were among the key characteristics that were not described well (Table 6 ). None of the twelve schools that were studied displayed evidence of having used a driving question in their projects. However, the majority of the schools (n = 8) did centre their projects around solving a single problem. According to PBL literature, this is not the same as having a driving question (see Table 1 for a more detailed description), but in the absence of driving questions it was considered useful to study whether the projects were at least centred around solving a single problem. Learning goals (goals with a reference to students’ development) were also not that commonly described; materials from n  = 6 schools displayed learning goals set by teachers, but none of the schools displayed learning goals set by students. However, students did appear to set practical goals (goals with no reference to students’ development) in the projects from n  = 3 schools, and teachers mentioned these in most schools too ( n  = 9). Furthermore, students’ descriptions of what they had learnt as a result of the projects were visible in the materials of n  = 10 schools, whereas teachers’ comments regarding that were only visible in those of half ( n  = 6) of the schools.

Figure  1 displays the distribution of the characteristics across the project units. The highest frequency values were for the schools E and F, which both had participated in the same development programme organised by a local university. However, although they did not receive help from researchers, schools A (f = 18), I (f = 17) and C (f = 16) still displayed a reasonably high count of PBL characteristics. In fact, school C had the same frequency of PBL characteristics as school D, which was the third school to participate in the university-led development programme. Figure 1 shows that there is a clear difference between schools whose PBL units were most closely in line with the PBL framework used in this study (f = 21, n  = 2) and the schools that provided project units with the least resemblance to it (f = 9, n = 2).

Frequency of the PBL characteristics demonstrated by the schools A-L ( n  = 12, see Tables  4 and 5 )

(2) Implementation of the key characteristics in the projects

The main results regarding the implementation of the key characteristics are summarised in Table 6 , together with their visibility. The detailed description about the implementation of each of the key characteristics of PBL can be found below: (1) driving question, (2) learning goals, (3) scientific practices, (4) collaboration, (5) using technological tools, and (6) creating an artefact.

Using central problems instead of driving questions did not stop schools from accomplishing some of the characteristics of a good driving question. In all of the schools where the project had a central problem, the problems were related to environmental issues, which meant that they were regarded as socio-scientific issues (Sadler, 2009 ). All of these schools also used local or familiar learning environments, which is another characteristic of a good driving question. For example, they researched everyday phenomena ( n  = 7 projects), used family or peers as audience ( n  = 6), created an impact on the local environment (n = 6) or studied it ( n  = 5). Some also visited local attractions ( n  = 2) or collaborated with students’ families (n = 2).

Interestingly, teachers and students seemed to report different kinds of learning gains; students focused on learning biology ( n  = 7 schools) more than teachers ( n  = 3), who paid attention to progress in learning social skills ( n  = 6), other twenty-first century skills ( n  = 2) and scientific practices ( n  = 2). Students reported these respectively in n  = 4, n  = 1 and n  = 0 schools. Furthermore, teachers did not mention students’ personal development (for example, new perspectives and experiences), which the students themselves noted in n  = 2 schools. Students also mentioned development of their environmental values more often ( n  = 4 compared to teachers in n  = 2 schools). ICT skills were mentioned in n  = 2 schools by students and n  = 1 by teachers.

When words that referred to the students’ development (for example, “develop”, “apply” or “learn”) were used in conjunction with the aims of the project, the goal was interpreted as a learning goal. However, when they were absent, the goal was interpreted as a concrete practical aim (for example, “creating an herb garden”). N  = 5 projects displayed practical goals set by students, all of which were related to biology too. However, none of the goals set by students were learning goals according to the definition described above; they all focused on the practical aims of the work instead. Learning goals set by teachers included learning related to biology ( n  = 5 schools), scientific practices ( n  = 4), social skills ( n  = 3), other twenty-first century skills ( n  = 1) and technical skills ( n  = 1). The learning goals related to biology could be divided into values ( n  = 5 schools), content ( n  = 3) and skills ( n  = 1).

The materials of the study did not allow extensive assumptions about what was teachers’ and what students’ viewpoint, but in terms of learning goals, it was deemed necessary to make a distinction based on the sentence structures. If a continuous part of the text displayed students as implementers and was written in third person (for example, “in this project students are expected to …” or “their goal is to …” ), the learning was interpreted as having been set by the teacher. However, if a continuous part of the text was presented in first person and the text clearly displayed that “we” referred to students, the part of the text that described learning was interpreted as students’ viewpoint to learning.

With regards to different scientific practices, it was not possible to identify how student-led the implementation was due to lack of teachers’ and students’ comments on this. Hypotheses were not presented in any of the projects, although n  = 8 projects included experiments that could have included a hypothesis. The three projects that did not show any signs of doing research and interpreting data were all from the same school and generally vaguely described; these projects did not show evidence of students drawing conclusions either. As all projects were presented to others at least through the video that was shared to StarT, all of them were considered as having presented the results of the project. However, all but one project described having done that in other ways as well, for example, by giving presentations for younger students and parents, and making posters.

Most of the projects were carried out in various learning environments and with a variety of partners. In terms of collaboration, three categories emerged: collaboration between students ( n  = 11 schools), collaboration between teachers ( n  = 9), and collaboration between the school and outside actors ( n  = 9). Collaboration between students was mostly group work ( n  = 16 projects) or presenting the work for other students ( n  = 9 projects). Teachers collaborated mostly with other teachers in the same school ( n  = 8 schools), and in some cases with teachers from another school ( n  = 4); however, n  = 3 of these schools participated in the same development programme of a local university, and this university organised the event where the collaboration happened. The materials did not provide information of how the teachers collaborated with each other or divided tasks. The outside partners were students’ parents ( n  = 9), universities ( n  = 5), media ( n  = 5), museums ( n  = 5), municipalities or other public agencies ( n  = 4), local people ( n  = 3), other experts ( n  = 3), and organisations ( n  = 2).

Technology used by students in their projects could be divided into two categories that emerged from the materials: ICT (information and communication technologies) and technology that was used as a scientific research tool. All technology that is commonly available and used at homes (and schools), such as editing videos, programming and text editing, and calculation programmes, was included in the ICT category. Any technology that is not commonly expected to be found at homes but that can be used to do scientific measurements and observations (for example, pH probes and nitrogen indicators, microscopes and voltage meters) was considered as scientific technology. According to this definition, students used scientific technology in n  = 6 projects and ICT in n  = 15 projects.

The artefacts included for example, reports, slideshows, lessons, webpages and miniature models. Multiple artefacts were created in majority of the projects ( n  = 14). Different categories emerged depending on what the role of these artefacts was in the project. In n  = 2 projects, the artefacts were part of a larger, final artefact. For example, one of the schools developed a webpage on climate change, and the contents of the webpage (for example, campaign videos and articles) were produced by separate student groups. Whilst multiple artefacts were created in many projects, it was more common for them to complement each other, meaning that they dealt with the same topic by answering it from a slightly different angle (n = 6 projects). In one of these projects, students had, for example, created both a video and a slide show on the same topic, or both a written report and a physical miniature model.

In the third category, in which multiple artefacts were made, students created artefacts that dealt with the same theme but did not directly attempt to answer the same question ( n  = 5). These artefacts were the result of multiple activities that were separate from each other. For example, in one project, students created weather maps, recorded air pressure, and made art related to weather. Although all of these activities were related to the same theme, they were clearly separate from one another, and they did not aim to solve a common problem. In the rest of the projects ( n  = 4), only one clear artefact was produced. In n  = 2 of these projects, the artefact was relatively simple, and the materials did not give evidence of students having had to carry out significant research or experimentation in order to create it. In the other n = 2 projects, the artefact was clearly a complex technical product, such as a miniature model of an energy-efficient house or an irrigation system for plants. These projects displayed evidence of the students having done smaller experiments to be able to create the final artefact. However, as the results of these experiments were not turned into clear artefacts, these artefacts were considered as separate from the first category (‘single artefacts form the final artefact’).

The main aim of this study was to understand the possibilities and challenges related to the implementation of key characteristics of PBL. These aims will be discussed in relation to each of the research questions below.

Key characteristics of PBL implemented by the teachers

This study shows that within the context of K-12 science education, using PBL creates opportunities for the implementation of the following key characteristics (Krajcik & Shin, 2014 ): collaboration, artefacts, technology, problem-centredness, and scientific practices (Table 6 ; carrying out research, presenting results, and reflection). However, it might also be true that these characteristics are generally commonly implemented at schools, or aspects of social constructivism or PBL familiar to teachers. For example, Viro et al. ( 2020 ) found that teachers saw development of teamwork skills among the most important characteristics of PBL. However, both Viro et al. ( 2020 ) and Aksela & Haatainen ( 2019 ) also found that teachers consider technical issues and collaboration as significant challenges in science PBL; as such, teachers’ attention may have been directed to describe the use of these practices in their project reports.

This study indicates that schools might struggle especially with implementing driving questions, using students’ questions, and having students set their own learning goals (see ‘Teachers’ implementation of the key characteristics’ for further discussion). Notably, the characteristics that were commonly visible in the studied PBL units were also well-aligned with the StarT format that promotes their implementation (StarT programme). As such, there might be potential in encouraging teachers to implement certain characteristics of PBL through a competition and its instructions and assessment criteria. For example, StarT does not mention driving questions, and although ¾ of the projects were centred around solving a problem, no driving questions were visible. Similar to this study, Haatainen & Aksela ( 2021 ) found that only half of the 12 StarT schools they studied included driving questions in their projects. Driving questions have previously been identified as the most challenging aspect of PBL (Mentzer et al., 2017 ), but it is likely that the studied teachers were not even familiar with the concept as there were no mentions of this ‘hallmark’ of PBL. Based on the results, it might be worthwhile to include the framework used in this study more visibly into the StarT programme in order to direct the teachers’ attention to the desired characteristics. However, although advocated for by StarT (StarT programme), students’ questions were hardly visible at all. Goals set by students were also rare ( n  = 3 schools), and none of them showed signs of learning goals set by students (see next section for further discussion).

Teachers’ implementation of the key characteristics

Artefacts and driving questions would seem to require further instruction. Nearly half of the schools produced single artefacts that resulted from separate activities only linked together through a common theme. Artefacts should, however, answer the driving question and draw the project together (for example, Mentzer et al., 2017 ). Although there were no driving questions, many of the projects that were centred around solving a problem still managed to demonstrate other characteristics of PBL and the qualities of a good driving question well (centred around solving a problem, use of socio-scientific issues, and local or familiar learning environments). This is in line with the findings of Morrison et al. ( 2020 ), who found that teachers are very aware of the importance of authenticity and working with real-world problems in PBL. However, although the driving question can be replaced with a central problem (Hasni et al., 2016 ), it has an important role in unifying the activities within a PBL unit (Thomas, 2000 ). Judging by the artefacts, many of the projects lacked the kind of unity described in literature, especially those with no central problem or one that was defined broadly. Therefore, the observations from this study support the views of Mentzer et al. ( 2017 ), Krajcik & Shin ( 2014 ) and Blumenfeld et al. ( 1991 ) on the importance of a driving question on unifying the PBL unit.

As only half of the schools displayed learning goals and many of the projects mentioned that they had been carried outside of regular lesson time, it seemed like most of the projects were not primarily used as a means to learn central concepts. According to Thomas ( 2000 ), this is not PBL, but Tamim & Grant ( 2013 ) suggest taking a broader outlook on what is considered PBL. Nevertheless, as collaboration, time and organisation of the projects have previously been found to be among the aspects of PBL that teachers find challenging (Viro et al., 2020 ; Aksela & Haatainen, 2019 ), it is not surprising that teachers would prefer to use PBL outside of regular lesson time and focus on developing students’ soft skills, rather than focusing on content acquisition. However, spending sufficient time and covering central content have been identified among the central variables for successful PBL teaching in science education (Tal et al., 2006 ), in addition to building strong teacher-student relationships (Morrison et al., 2020 ). This indicates that for PBL to be a truly useful method for teachers, the recent changes in curricula towards less content and covering more skills (Novak & Krajcik, 2020 ) need to be sustained, and these changes need to be reflected in the standardised tests too.

The learning goals mentioned by the teachers were well aligned with the learning gains associated with PBL (for example, scientific practices, social skills and other twenty-first century skills, environmental values), but this does not equal working with concepts central to their curricula. Furthermore, for students to benefit from the learning gains associated with PBL, the focus should be on learning rather than doing a project; the teachers’ attention should be on what the students can research and find out, instead of focusing on what students can create and do (Lattimer & Riordan, 2011 ). Mentzer et al. ( 2017 ) found that projects implemented by teachers who had used PBL for no longer than a year did not resemble a coherent research project, and that this changed only after two or three years of PBL implementation. The projects tended to be a collection of lessons that were poorly connected to each other, and that consisted of either highly structured activities that had the same pre-defined outcome for all students, or of activities in which the main purpose was to research without a clear outcome (Mentzer et al., 2017 ). Similarly, in this study, the projects were often a collection of separate activities tied together through a common theme. According to Blumenfeld et al. ( 1991 ), this could be solved with a good driving question which brings cohesion to the project and ensures that students are working with central concepts and problems.

Although scientific practices were represented generally well across the studied schools, students’ questions were hardly visible, and goals set by students were rare ( n  = 3 schools). As such, it remains unclear how student-led the projects were exactly. For example, Herranen & Aksela ( 2019 ) highlight the importance of training teachers to use students’ questions as the basis of classroom inquiries, as this has clear implications for how authentically the inquiry will resemble that of scientists. Teachers might see PBL as student-centred (Aksela & Haatainen, 2019 ) and use scientific practices in their projects, but the reality is that they can be employed in a highly teacher-led fashion too (Colley, 2006 ). Earlier research into StarT projects indicated that the projects varied from having “complete student autonomy” to having “teacher-led activities with little student choice” (Haatainen & Aksela, 2021 ).

Furthermore, Severance & Krajcik ( 2018 ) found that even with support from researchers, teachers struggled to understand the idea of using scientific practices in their teaching. Also, teachers themselves consider lack of support for PBL implementation, including teachers’ professional skills and motivation, among the most common hindrances to PBL implementation (Viro et al., 2020 ). In line with this, the n  = 3 schools in this study that received support for the implementation of PBL from a university, all displayed a higher count of PBL characteristics and scientific practices than most of the studied schools (Fig. 1 ). However, whilst two of them displayed the highest count of characteristics across all cases, one of them had a lower count, closer to the values of schools that did not receive help. This highlights the importance of providing additional support for the schools in terms of the pedagogy of PBL and implementing scientific practices, and the fact that even support from a university does not guarantee research-based implementation of PBL. Even when teachers implement PBL units designed by researchers, they can adapt the unit significantly when moulding it for their educational context (Condliffe et al., 2017 ). Depending on the teachers’ beliefs, it is likely that all of these adaptations are not beneficial for learning (Condliffe et al., 2017 ).

Additionally, teachers who intended to teach biology through the projects (5/12 schools) mainly focused on developing students’ values towards nature and environment. This can of course be expected as all projects aimed to solve environmental issues, but it should not give a reason to exclude goals related to subject-specific content and skills. Especially, as the data consisted of projects in which biology had a clear role, and the students frequently (7/12 schools) mentioned having learnt biology content. However, the teachers mentioned this in three schools only. The explanation could be that students had a more liberal idea of what constitutes as biology content, or that the teachers had not even attempted to teach core content through the projects, and thus did not pay particular attention to development in that area. Nevertheless, the different views between teachers and students in terms of perceived learning gains may be an interesting point to study in the future.

Overall, it seems like the teachers mainly used PBL for learning soft skills, which is commonly reported about PBL (Guo et al., 2020 ; Aksela & Haatainen, 2019 ). For instance, in a study of PBL in mathematics, Viro et al. ( 2020 ) found that less than half of the in- and pre-service teachers they surveyed ( n  = 64) considered learning mathematics among the three most important characteristics of a successful PBL unit. Other options that they considered as most important for a successful PBL unit in mathematics were all related to student motivation and learning of twenty-first century skills. In line with this, the results indicate a need to emphasise the importance of planning the PBL unit around the core curriculum so that in-depth subject teaching can occur (Grossman et al., 2019 ; Tal et al., 2006 ). Context-based and problem-based approaches to instruction are seen as useful for student learning in biology (Cabbar & Senel, 2020 ; Jeronen et al., 2017 ), but if the focus is not on central concepts, then it remains uncertain how useful the PBL units are from the perspective of academic performance.

Development of twenty-first century skills is vital for solving issues related to sustainability, which makes PBL an attractive approach for teaching topics related to it (Konrad et al., 2020 ). Using environmental issues as the starting point of PBL projects in science education has become increasingly popular, and there is a growing body of evidence of its usefulness as a way to implement STEM PBL (for example, Hugerat, 2020 ; Triana et al., 2020 ; Kricsfalusy et al., 2018 ). This study is in line with that as students stated that their environmental attitudes had developed in several schools ( n  = 4). Teachers mentioned developing students’ environmental values as learning goals of the projects in n  = 5 schools, and n  = 2 schools mentioned that the goal had been reached. However, as the participants of this study had a lot of freedom in terms of what they decided to report about their projects, teachers not explicitly mentioning the development of environmental values does not necessarily mean that the goal was not reached.

Limitations

Content analysis can only focus on what is visible in the materials (Cohen et al., 2007 ). As teachers and students have reported their project work to the StarT competition that searches good models for the implementation of PBL, it can be expected that the teachers would highlight (and instruct their students to highlight) the aspects of PBL that they consider important in the videos and written descriptions that they provided. Consequently, if a certain characteristic of PBL is not visible in their materials at all, it is likely that teachers are either not aware of it or do not consider it that important for the implementation of PBL. However, as participating in competitions such as StarT is usually extra work for the teachers, they might struggle to find the time to provide materials that accurately represent their views on what was essential for the project. Furthermore, the form of reporting was very open-ended (for example, videos and learning diaries). As such, it remains possible that if the instructions for reporting the PBL unit had included specific questions about certain characteristics, teachers might have been able to comment on them. Nevertheless, it remains true that in their reports, teachers would include what they valued and focused on most in their projects.

What is more, as participation in StarT is completely voluntary, it is likely that the sample of teachers and schools studied is limited to those that are already actively interested and implementing PBL. As such, the results cannot necessarily be expected to represent PBL that is carried out in an average classroom; the focus is clearly on teachers who are already actively engaged in PBL and science education programmes. As one would expect, PBL implementation can be greatly influenced by school context and whether it is supported by school leadership or not (Condliffe et al., 2017 ).

A further limitation to the results is the scope of the materials and the limitations they had for determining the extent of student-centredness in projects; only inferences can be ascertained about which decisions were made by the students and which by the teachers. However, the interpretations that were made during the coding process have been carefully described in ‘Methods’. As such, whilst the materials limited the deductions that could be made confidently, the analysis is reliable within said limitations.

The number of separate schools in this study is 12. However, three of them did interact with each other as they participated in the same development programme organised by a local university. Nevertheless, as Stake ( 2000 ) states, the main aim of a case study is not to generalise results but to understand the cases better. The aim of the study is not to claim that the results would be true to all teachers but to gain more understanding of how individual teachers might see PBL and find trends across individual cases.

This study supports the notion that teachers have varying conceptions of PBL and its characteristics (Hasni et al., 2016 ). The study provided new information of PBL that takes place at schools that are active participants in international education competitions, as they have not been researched from the perspective of the characteristics of PBL earlier. As such, it also shows how teachers who are actively engaged in PBL implement the characteristics, therefore giving an idea of what the ‘best-case scenario’ of the implementation of PBL units that are not guided by researchers might be. Additionally, due to the international sample of schools studied, the study is not limited to a specific educational context.

This study provides important information for teacher training, as it has paved the way into studying the quality of PBL units created by teachers as opposed to those created by researchers through the lens of key characteristics of PBL. Based on the results, the authors believe it is important to ensure that teacher training and curriculum development consider how teachers can use PBL to teach central content, and how schools can better support teachers to carry this out in terms of resources and time.

In line with Morrison et al. ( 2020 ) and Tsybulsky & Muchnik-Rozanov ( 2019 ), the authors believe it would be important for teachers themselves to learn through PBL during their pre-service training. Furthermore, for teachers to be able to fully grasp the pedagogical approach required in PBL, both teacher training and research should consider the key characteristics and their implementation, especially those that have been shown to cause more difficulties for teachers through this and earlier studies (for example, teaching central content, students’ questions and driving questions). Additionally, it may be useful to direct efforts into studying the key characteristics from the perspective of flexible implementation; which of the characteristics should be followed rigidly, and could some of them be interpreted more flexibly to suit local educational contexts better? For example, considering the importance placed on the driving question in PBL literature, and the difficulties in its implementation, it would be useful to understand how the characteristic could be contextualised into a format that is more easily accessible to teachers.

Finally, a viable framework was created for analysing how the key characteristics of PBL were implemented in teachers’ projects. It can be adapted for studying PBL units also in other settings. The used approach to analysing project units can also be used as a starting point for studying PBL artefacts, which has been advocated for by Guo et al. ( 2020 ) and Hasni et al. ( 2016 ). What is more, it allows studying PBL from the point of view of students, which has also been done clearly less in PBL research (Habók & Nagy, 2016 ).

The authors believe that research should continue to address PBL units from the perspective of the key characteristics of PBL. This allows research to be grounded in the practice of schools, and for researchers to pinpoint the most critical aspects of PBL that professional development initiatives should focus on. PBL remains a challenging instructional method and a lot more training and resources are still needed for it to live up to its potential. The results from this study and the constructed framework of key characteristics can be useful in promoting research-based implementation and design of PBL science education, and in teacher training related to it.

Abbreviations

• Project-based learning

Science, technology, engineering and mathematics.

Abd-El-Khalick, F., Boujaoude, S., Duschl, R., Lederman, N. G., Mamlok-Naaman, R., Hofstein, A., & Tuan, H. L. (2004). Inquiry in science education: International perspectives. Science education , 88 (3), 397–419. https://doi.org/10.1002/sce.10118Aksela M. K. & Haatainen, O. M. (2019). Project-Based Learning (PBL) in Practise: Active Teachers’ Views of Its’ Advantages And Challenges . Integrated Education for the Real World: 5th International STEM in Education Conference Post-Conference Proceedings, The Queensland University of Technology, 9–16.

Article   Google Scholar  

Aksela, M., & Haatainen, O. (2019). Project‐based learning (PBL) in practise: Active Teachers’ Views of Its Advantages and Challenges. In Integrated Education for the Real World: 5th International STEM in Education Conference Post-Conference Proceedings (pp. 9–16). Queensland University of Technology.

Annetta, L. A., Lamb, R., & Vallett D.& Shapiro M. (2019). In Project-based learning progressions: Identifying the nodes of learning in a project-based environment, O. Adesope, & A. Rud (Eds.), Contemporary Technologies in Education , (pp. 163–181). Cham: Palgrave Macmillan. https://doi.org/10.1007/978-3-319-89680-9_9 .

Chapter   Google Scholar  

Balemen, N., & Özer Keskin, M. (2018). The effectiveness of project-based learning on science education: A meta-analysis search. International Online Journal of Education and Teaching (IOJET) , 5 (4), 849–865 http://iojet.org/index.php/IOJET/article/view/452/297 .

Google Scholar  

Baxter, P., & Jack, S. (2008). Qualitative case study methodology: Study design and implementation for novice researchers. The Qualitative Report , 13 (4), 544–559.

Bell, S. (2010). Project-based learning for the 21st century: Skills for the future. The Clearing House , 83 (2), 39–43. https://doi.org/10.1080/00098650903505415 .

Bestelmeyer, S. V., Elser, M. M., Spellman, K. V., Sparrow, E. B., Haan-Amato, S. S., & Keener, A. (2015). Collaboration, interdisciplinary thinking, and communication: New approaches to K–12 ecology education. Frontiers in Ecology and the Environment , 13 (1), 37–43. https://doi.org/10.1890/140130 .

Blumenfeld, P. C., Soloway, E., Marx, R. W., Krajcik, J. S., Guzdial, M., & Palincsar, A. (1991). Motivating project-based learning: Sustaining the doing, supporting the learning. Educational Psychologist , 26 (3–4), 369–398. https://doi.org/10.1080/00461520.1991.9653139 .

Cabbar, B. G., & Senel, H. (2020). Content analysis of biology education research that used context-based approaches: The case of Turkey. Journal of Educational Issues , 6 (1), 203–218. https://doi.org/10.5296/jei.v6i1.16920 .

Capraro, R. M., Capraro, M. M., Scheurich, J. J., Jones, M., Morgan, J., Huggins, K. S., … Han, S. (2016). Impact of sustained professional development in STEM on outcome measures in a diverse urban district. The Journal of Educational Research , 109 (2), 181–196. https://doi.org/10.1080/00220671.2014.936997 .

Chen, C. H., & Yang, Y. C. (2019). Revisiting the effects of project-based learning on students’ academic achievement: A meta-analysis investigating moderators. Educational Research Review , 26 , 71–81. https://doi.org/10.1016/j.edurev.2018.11.001 .

Chin, C., & Osborne, J. (2008). Students' questions: A potential resource for teaching and learning science. Studies in Science Education , 44 (1), 1–39. https://doi.org/10.1080/03057260701828101 .

Cohen, L., Manion, L., & Morrison, K. (2007). Research methods in education , (6th ed., ). London: Taylor & Francis. https://doi.org/10.4324/9780203029053 .

Book   Google Scholar  

Colley, K. E. (2006). Understanding ecology content knowledge and acquiring science process skills through project-based science instruction. Science Activities: Classroom Projects and Curriculum Ideas , 43 (1), 26–33. https://doi.org/10.3200/SATS.43.1.26-33 .

Condliffe, B., Quint, J., Visher, M. G., Bangser, M. R., Drohojowska, S., Saco, L., & Nelson, E. (2017). Project-based learning: A literature review . MDRC: Working Paper https://www.mdrc.org/publication/project-based-learning . Accessed 31 May 2021.

Edelson, D. C. (2001). Learning-for-use: A framework for the design of technology-supported inquiry activities. Journal of Research in Science Teaching , 38 (3), 355–385. https://doi.org/10.1002/1098-2736(200103)38:33.0.CO;2-M .

Elo, S., Kääriäinen, M., Kanste, O., Pölkki, T., Utriainen, K., & Kyngäs, H. (2014). Qualitative content analysis: A focus on trustworthiness. SAGE Open , 4 (1). https://doi.org/10.1177/2158244014522633 .

Erdoğan, N., Navruz, B., Younes, R., & Capraro, R. M. (2016). Viewing how STEM project-based learning influences students' science achievement through the implementation lens: A latent growth modeling. Eurasia Journal of Mathematics, Science & Technology Education , 12 (8), 2139–2154. https://doi.org/10.12973/eurasia.2016.1294a .

European Commission. (2007). Science education NOW: A renewed pedagogy for the future of Europe . Available https://ec.europa.eu/research/swafs/index.cfm?pg=library&lib=science_edu . Accessed 1 May 2020.

Grossman, P., Dean, C. G. P., Kavanagh, S. S., & Herrmann, Z. (2019). Preparing teachers for project-based teaching. Phi Delta Kappan , 100 (7), 43–48. https://doi.org/10.1177/0031721719841338 .

Guo, P., Saab, N., Post, L. S., & Admiraal, W. (2020). A review of project-based learning in higher education: Student outcomes and measures. International Journal of Educational Research , 102 , 101586. https://doi.org/10.1016/j.ijer.2020.10158Gustafsson J. (2017). Single case studies vs. multiple case studies: A comparative study. Available: https://www.diva-portal.org/smash/get/diva2:1064378/FULLTEXT01.pdf Accessed 1 October 2020.

Haatainen, O., & Aksela, M. (2021). Project-based learning in integrated science education: Active teachers’ perceptions and practices. LUMAT: International Journal on Math, Science and Technology Education , 9 (1), 149–173. https://doi.org/10.31129/LUMAT.9.1.1392 .

Habók, A., & Nagy, J. (2016). In-service teachers' perceptions of project-based learning. SpringerPlus , 5 (1), 83. https://doi.org/10.1186/s40064-016-1725-4 .

Han, S., Yalvac, B., Capraro, M. M., & Capraro, R. M. (2015). In-service teachers' implementation and understanding of STEM project-based learning, EURASIA Journal of Mathematics, Science & Technology Education, 11 (1), 63–76. https://doi.org/10.12973/eurasia.2015.1306a .

Hasni, A., Bousadra, F., Belletête, V., Benabdallah, A., Nicole, M., & Dumais, N. (2016). Trends in research on project-based science and technology teaching and learning at K–12 levels: A systematic review. Studies in Science Education , 52 (2), 199–231. https://doi.org/10.1080/03057267.2016.1226573 .

Herranen, J., & Aksela, M. (2019). Student-question-based inquiry in science education. Studies in Science Education , 55 (1), 1–36. https://doi.org/10.1080/03057267.2019.1658059 .

Hmelo-Silver, C. (2004). Problem-based learning: What and how do students learn? Educational Psychology Review , 16 (3), 235–266. https://doi.org/10.1023/B:EDPR.0000034022.16470.f3 .

Hsieh, H. F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research , 15 (9), 1277–1288. https://doi.org/10.1177/1049732305276687 .

Hugerat, M. (2020). Incorporating sustainability into chemistry education by teaching through project-based learning. In chemistry education for a sustainable society volume 1: High school, outreach, & global perspectives. American Chemical Society , 79–96. https://doi.org/10.1021/bk-2020-1344.ch007 .

Jeronen, E., Palmberg, I., & Yli-Panula, E. (2017). Teaching methods in biology education and sustainability education including outdoor education for promoting sustainability – A literature review. Education Sciences , 7 (1), 1. https://doi.org/10.3390/educsci7010001 .

Kaldi, S., Filippatou, D., & Govaris, C. (2011). Project-based learning in primary schools: Effects on pupils' learning and attitudes. Education 3–13 , 39 (1), 35–47. https://doi.org/10.1080/03004270903179538 .

Konrad, T., Wiek, A., & Barth, M. (2020). Embracing conflicts for interpersonal competence development in project-based sustainability courses. International Journal of Sustainability in Higher Education , 21 (1), 76–96. https://doi.org/10.1108/IJSHE-06-2019-0190 .

Kortam, N., Basheer, A., Hofstein, A., & Hugerat, M. (2018). How project-based learning promotes 7th grade students' motivation and attitudes towards studying biology. Action Research and Innovation in Science Education , 1 (2), 9–17. https://doi.org/10.12973/arise/103043 .

Krajcik, J. (2015). Project-based science. The Science Teacher , 82 (1), 25–27. https://doi.org/10.2505/4/tst15_082_01_25 .

Krajcik, J. S., & Shin, N. (2014). In Project-based learning, & R. K. Sawyer (Eds.), The Cambridge handbook of the learning sciences , (2nd ed., pp. 275–297). Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9781139519526.018 .

Kricsfalusy, V., George, C., & Reed, M. G. (2018). Integrating problem-and project-based learning opportunities: Assessing outcomes of a field course in environment and sustainability. Environmental Education Research , 24 (4), 593–610. https://doi.org/10.1080/13504622.2016.1269874 .

Lähdemäki, J. (2019). In Case study: The Finnish National Curriculum 2016—A co-created National Education Policy, & J. Cook (Eds.), Sustainability, human well-being, and the future of education . Cham: Palgrave Macmillan. https://doi.org/10.1007/978-3-319-78580-6_13 .

Lattimer, H., & Riordan, R. (2011). Project-based learning engages students in meaningful work. Middle School Journal , 43 (2), 18–23. https://doi.org/10.1080/00940771.2011.11461797 .

Lavonen, J., Inkinen, J., Juuti, K., Salmela-Aro, K., Krajcik, J., & Schneider, B. (2017). In The influence of an international professional development project for the design of engaging secondary science teaching in Finland, M. K. Mhlolo, S. N. Matoti, & B. Fredericks (Eds.), Book of long papers: 25th annual meeting of the southern African Association of Researchers in Mathematics Science & Technology Education (SAARMSTE) , (pp. 206–220) SAARMSTE.

Malone, T. W., & Lepper, M. R. (1987). In ). Making learning fun: A taxonomy of intrinsic motivations for learning, R. Snow, & M. Farr (Eds.), Aptitude, learning, and instruction: Conative and affective process analyses , (pp. 223–253). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc..

Marshall, C., & Rossman, G. B. (2014). Designing qualitative research , (6th ed., ). Inc: SAGE Publications.

Mentzer, G. A., Czerniak, C. M., & Lisa, B. (2017). An examination of teacher understanding of project-based science as a result of participating in an extended professional development program: Implications for implementation. School Science and Mathematics , 117 (1–2), 76–86. https://doi.org/10.1111/ssm.12208 .

Miller, E. C., & Krajcik, J. S. (2019). Promoting deep learning through project-based learning: A design problem. Disciplinary and Interdisciplinary Science Education Research , 1 (1), 1–10. https://doi.org/10.1186/s43031-019-0009-6 .

Morrison, J., Frost, J., Gotch, C., McDuffie, A. R., Austin, B., & French, B. (2020). Teachers’ role in students’ learning at a project-based STEM high school: Implications for teacher education. International Journal of Science and Mathematics Education , 1–21. https://doi.org/10.1007/s10763-020-10108-3 .

National Academy of Engineering & National Research Council. (2014). STEM integration in K-12 education: Status, prospects, and an agenda for research. Washington, DC: The National Academies Press. Available https://www.nap.edu/catalog/18612/stem-integration-in-k-12-education-status-prospects-and-an . Accessed 1 of May 2020.

National Research Council. (2012 ). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas . Washington, DC: The National Academies Press. Available https://www.nap.edu/catalog/13165/a-framework-for-k-12-science-education-practices-crosscutting-concepts . Accessed 1 May 2020.

Novak, A. M., & Krajcik, J. S. (2020). In M. Moallem, W. Hung, & N. Dabbagh (Eds.), A case study of project-based learning of middle school students exploring water quality , (pp. 551–527). The Wiley handbook of problem-based learning. https://doi.org/10.1002/9781119173243.ch24 .

Pan, G., Seow, P. S., Shankararaman, V., & Koh, K. (2020). An exploration into key roles in making project-based learning happen: Insights from a case study of a university. Journal of International Education in Business , 14 (1), 109–129. https://doi.org/10.2139/ssrn.3603881 .

Parker, W. C., Lo, J., Yeo, A. J., Valencia, S. W., Nguyen, D., Abbott, R. D., … Vye, N. J. (2013). Beyond breadth-speed-test: Toward deeper knowing and engagement in an advanced placement course. American Educational Research Journal , 50 (6), 1424–1459. https://doi.org/10.3102/0002831213504237 .

Pedaste, M., Mäeots, M., Siiman, L. A., De Jong, T., Van Riesen, S. A., Kamp, E. T., … Tsourlidaki, E. (2015). Phases of inquiry-based learning: Definitions and the inquiry cycle. Educational Research Review , 14 , 47–61. https://doi.org/10.1016/j.edurev.2015.02.003 .

Sadler, T. D. (2009). Situated learning in science education: Socio-scientific issues as contexts for practice. Studies in Science Education , 45 (1), 1–42. https://doi.org/10.1080/03057260802681839 .

Sahin, A. (2013). In STEM project-based learning, R. M. Capraro, M. M. Capraro, & J. R. Morgan (Eds.), STEM project-based learning (2nd volume, p. 59-64) . Rotterdam: SensePublishers. https://doi.org/10.1007/978-94-6209-143-6_7 .

Sasson, I., Yehuda, I., & Malkinson, N. (2018). Fostering the skills of critical thinking and question-posing in a project-based learning environment. Thinking Skills and Creativity , 29 , 203–212. https://doi.org/10.1016/j.tsc.2018.08.001 .

Severance, S., & Krajcik, J. (2018). In Examining primary teacher expertise and Agency in the Collaborative Design of project-based learning innovations, J. Kay, & R. Luckin (Eds.), Rethinking learning in the digital age: Making the learning sciences count, 13th international conference of the learning sciences (ICLS) 2018, volume 2 . International Society of the Learning Sciences: London, UK.

Stake, R. E. (2000). Case studies. In N. K. Denzin, & Y. S. Lincoln (Eds.), Handbook of qualitative research , (2nd ed., pp. 236–247). Thousand Oaks: Sage Publications.

StarT programme. The International StarT Competition. https://start.luma.fi/en/start-competition/ . Accessed 15 Apr 2020.

Stearns, L. M., Morgan, J., Capraro, M. M., & Capraro, R. M. (2012). A teacher observation instrument for PBL classroom instruction. Journal of STEM Education: Innovations and Research , 13 (3), 7–16.

Tal, T., Krajcik, J. S., & Blumenfeld, P. C. (2006). Urban schools' teachers enacting project-based science. Journal of Research in Science Teaching , 43 (7), 722–745. https://doi.org/10.1002/tea.20102 .

Tamim, S. R., & Grant, M. M. (2013). Definitions and uses: Case study of teachers implementing project-based learning. Interdisciplinary Journal of Problem-Based Learning , 7 (2), 72–101. https://doi.org/10.7771/1541-5015.1323 .

Thomas, J. W. (2000). A review of research on project-based learning . San Rafael, CA: Autodesk Foundation Available http://www.bie.org/object/document/a_review_of_research_on_project_based_learning . Accessed 30 August 2019.

Thys, M., Verschaffel, L., Van Dooren, W., & Laevers, F. (2016). Investigating the quality of project-based science and technology learning environments in elementary school: A critical review of instruments. Studies in Science Education , 52 (1), 1–27. https://doi.org/10.1080/03057267.2015.1078575 .

Triana, D., Anggraito, Y. U., & Ridlo, S. (2020). Effectiveness of environmental change learning tools based on STEM-PjBL towards 4C skills of students. Journal of Innovative Science Education , 9 (2), 181–187. https://doi.org/10.15294/JISE.V8I3.34048 .

Tsybulsky, D., & Muchnik-Rozanov, Y. (2019). The development of student-teachers' professional identity while team-teaching science classes using a project-based learning approach: A multi-level analysis. Teaching and Teacher Education , 79 , 48–59. https://doi.org/10.1016/j.tate.2018.12.006 .

Turner, S. F., Cardinal, L. B., & Burton, R. M. (2017). Research Design for Mixed Methods: A triangulation-based framework and roadmap. Organizational Research Methods , 20 (2), 243–267. https://doi.org/10.1177/1094428115610808 .

Viro, E., Lehtonen, D., Joutsenlahti, J., & Tahvanainen, V. (2020). Teachers' perspectives on project-based learning in mathematics and science. European Journal of Science and Mathematics Education , 8 (1), 12–31. https://doi.org/10.30935/scimath/9544 .

Yin, R. K. (2014). Case study research: Design and methods , (5th ed., ). Inc: SAGE Publications.

Download references

LUMA resources Ms Anette Markula MSc

Author information

Authors and affiliations.

LUMA Science Helsinki, Faculty of Science, University of Helsinki, Helsinki, Finland

Anette Markula & Maija Aksela

You can also search for this author in PubMed   Google Scholar

Contributions

The study was designed by AM, with comments from MA. AM designed the framework used in the analysis and carried out the data collection and analysis. AM was responsible for writing the manuscript, with contributions and comments from MA. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Maija Aksela .

Ethics declarations

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Markula, A., Aksela, M. The key characteristics of project-based learning: how teachers implement projects in K-12 science education. Discip Interdscip Sci Educ Res 4 , 2 (2022). https://doi.org/10.1186/s43031-021-00042-x

Download citation

Received : 31 October 2020

Accepted : 02 December 2021

Published : 06 January 2022

DOI : https://doi.org/10.1186/s43031-021-00042-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Key characteristics
• Design principles
• Interdisciplinary education
• Implementation
• Science education
• Biology education

• Our Mission

A Simple, Effective Framework for PBL

This plan was designed to guide teachers who haven’t had formal training in project-based learning.

Teachers trying their hand at project-based learning (PBL) may be uncertain as to how to strengthen their project ideas and make them the best possible learning experiences for students. For teachers without access to training, a research-informed framework for PBL and a few strategies for defining and organizing the student experience can considerably improve outcomes.

The High Quality PBL (HQPBL) framework , when executed effectively, provides elements like authenticity, project management, and public products for educators to use for creating the conditions for learning to stick and continue after projects.

For example, content or elective teachers can increase authenticity in projects by bringing in industry experts (e.g., engineers, environmental scientists, computer programmers, activists) at the launch to introduce the type of work that students will be learning to do.

Teachers can also help students improve their work by having them develop public products with a call to action advocating for causes they care about and instructing audiences of community members on the next steps to take.

Before diving into the framework, let’s quickly dispel two of the biggest misconceptions and roadblocks to attempting PBL that I’ve heard from educators.

Common PBL Hurdles

1. I have to prepare my students for exams (or cover lots of content) and can’t dedicate an entire school year or semester to planning or teaching this way. I agree—do not abandon the teaching practice you have carefully honed. Instead, implement one project a semester, connect it to learning in your area as best as possible, and implement it for no more than two to three weeks at a time.

2. I’m a content teacher and am not exactly sure how to make real-world projects. I admit this can be tricky the first time around. Focus on important problems in the community (e.g.,  health, financial inclusion, environment ). Let the kids pick the issue(s) they want to tackle and develop a plan for knowing their topic inside and out, along with solutions.

See this video example where educator Jose Gonzalez of Compton Unified School District in California implemented a terrific interdisciplinary project: allowing students to choose their path to advocate for change in their communities.

Using the High Quality PBL Framework

Established in 2018, the HQPBL framework is a consensus of both the research and the accumulated practice of PBL leaders and experts worldwide. It can be used with learners of all ages, but it’s particularly well-suited to middle and high school students who are passionate about solving meaningful problems.

The framework is designed to provide educators who have no access to formal training with resources that enable them to enact PBL practices on their own by setting the criteria for the student experience using the following six elements.

1. Intellectual challenge and accomplishment. Students investigate challenging problems or issues over an extended period of time. I recommend two to three weeks for teachers new to the process. Throughout this period, they should develop the essential content knowledge and concepts central to academic disciplines. Therefore, I encourage teachers to have students use the thinking routines and problem-solving strategies they typically use (e.g., Blooms, design thinking, scientific Inquiry, computational thinking) to think critically in their content area.

2. Authenticity. Projects focus on real-world connections that are meaningful to students—including their cultures and backgrounds . Additionally, the tools and techniques they employ mimic those used by career professionals. By inviting experts into the classroom and having students assume authentic career roles (e.g., engineer, doctor, auto technician), they can learn valuable career pathway options and see how their work and the solutions they develop impact others.

3. Public product. The students’ final products are presented to the public as a culminating event. This means the work they produce is seen and discussed with the broader community—including parents, industry professionals, other classes, administrators, and community members.

When students know that others will see their work, this may motivate them to put their best foot forward. Public products are not limited to presentation nights—student work can be displayed as public art, as exhibits, or online via social media, YouTube, and safe school websites.

4. Collaboration. Working with others is a PBL hallmark where students collaborate with both adults and their peers in a number of different ways. Adults serve as mentors and guides and can include teachers, community members, or outside experts. In teamwork between students , each learner contributes their individual skills and talents. I find that learners of all ages need good collaboration tools— team contracts and task lists are an excellent place to start.

5. Project management. Students help manage the project process, using tools and strategies similar to those used by adults. I’ve seen teachers using several tools for assisting learners in keeping their work organized—good ones include scrum boards , using design thinking during the ideation process, and maintaining important documents in Google Classroom and Schoology.

I’ve also found that some learners benefit greatly from keeping a daily schedule before attempting to help manage projects. As students’ capacity for self-management increases, teachers take on the role of facilitator, helping guide students through the process rather than directing it.

6. Reflection. The learning process is enhanced by frequent reflections that help students think about their progress and how to improve their work. I like to have them complete products in drafts and jump-start reflection through critique protocols —this helps learners retain content and skills longer and gives them the awareness of how they learn best by using reflection for metacognition. Other methods for reflection may include journaling, the 3-2-1 strategy , and the one-minute paper .

“Framework first, mindset second” is a powerful principle I use for helping colleagues understand that having good general guidelines for doing something new is the prerequisite to developing second-nature expertise. The HQPBL framework can be a good place to start for beginning to use PBL as a research-informed instructional approach.

Project-Based Learning Research Archive

Between 2013 and 2023, Lucas Education Research collaborated with prominent university education researchers to validate the effectiveness of rigorous project-based learning for students from diverse backgrounds.

This site continues to offer access to the research findings and supplementary materials from that period.

Explore our collection , or read our press release detailing the initiative’s outcomes.

Project-Based Learning Approach Pioneered at the UW College of Education Significantly Improves Student Performance

Students in project-based learning (PBL) classrooms across the United States significantly outperform students in typical classrooms, according to findings in research released by Lucas Educational Research, a division of the George Lucas Education Foundation (GLEF) with researchers from five major universities. The findings confirm that using a pedagogic model like, “Knowledge in Action” (KIA) which was created by Walter Parker, professor emeritus at the UW College of Education, and his colleagues, Professors John Bransford, Sheila Valencia, and Susan Nolen is an effective way to improve student performance on Advancement Placement test results.

The KIA approach centers on a rigorous form of project-based learning where projects are weeks-long simulations. GLEF used the year-long curricula developed by Parker and team to compare and assess student performance. The first study ever reported on project-based learning and Advancement Placement results, found that students taught with a PBL approach outperformed peers on exams by eight percentage points in one year of a randomized controlled trial, and were more likely to earn a passing score of 3 or above with the chance to receive college credit. In year two, PBL students outperformed peers by ten percentage points .

For more information and additional resources:

• View the video about the “Knowledge in Action” model created by UW College of Education professors
• Read GLEF’s full press release and research findings
• Watch the recorded webinar that provides interviews with researchers, teachers, and students
• Learn more about the project-based learning research and student experience
• Research Briefs
• White papers
• Multiple Literacies in Project-based Learning (ML-PBL) Video:  https://www.edutopia.org/video/project-based-approach-teaching-elementary-science  +  direct YouTube link
• A news article on the KIA and ML-PBL findings  https://www.edutopia.org/article/new-research-makes-powerful-case-pbl
• A  new link to Edutopia's youtube channel  with videos about each of the 4 research projects. 

• Corpus ID: 9414703

A REVIEW OF RESEARCH ON PROJECT-BASED LEARNING

• John W. Thomas
• Published 2000

Tables from this paper

2,017 Citations

Literature review of project-based learning.

• Highly Influenced

THE IMPLEMENTATION OF PROJECT-BASED LEARNING IN K-12 EDUCATION: TEACHER QUALITIES AND STUDENTS ACHIEVEMENTS

Project-based learning in efl classroom: strategies for success, using project-based learning to develop teachers for leadership, teaching mathematics: the role of project-based learning, implementation of project based learning: research overview, interdisciplinary stem project-based learning, project-based learning: a review of the literature, identifying opportunities for collaborations in international engineering education research on problem- and project-based learning, project-based service-learning in an instructional technology graduate program, 81 references, motivating project-based learning: sustaining the doing, supporting the learning.

• Highly Influential
• 10 Excerpts

Enacting Project-Based Science

Doing with understanding: lessons from research on problem- and project-based learning, a collaborative model for helping middle grade science teachers learn project-based instruction, enacting project-based science: experiences of four middle grade teachers, a qualitative examination of problem-based learning at the k-8 level: preliminary findings., implementing problem‐based learning in science classrooms, impediments to a project-based and integrated curriculum: a qualitative study of curriculum reform., examining how middle school students use problem-based learning software., a middle grade science teacher's emerging understanding of project-based instruction, related papers.

Showing 1 through 3 of 0 Related Papers

Context Matters for Foundation Models in Biology

Just as words can have multiple meanings depending on the context of a sentence, proteins can play different roles in a cell based on their cellular environments. Advances in our understanding of protein and biomolecule functions have been propelled by recent breakthroughs in transformer-based models, such as large language models and generative pre-trained transformers, which automatically learn word semantics from diverse language contexts. Innovating a similar approach for protein functions—viewing them as distributions across various cellular contexts—could enhance the use of foundation models in biology. This would allow the models to dynamically adjust their outputs based on the biological contexts in which they operate. To this end, we have developed PINNACLE , a novel contextual AI model for single-cell biology that supports a broad array of biomedical AI tasks by tailoring its outputs to the cell type context in which the model is asked to make predictions. 

To glean the meaning of a word, we examine nearby words for context clues. For example, “buy an apple” and “grow an apple” yield different recommendations: the first phrase is used to refer to apple products, whereas the second is better associated with apple trees (Figure 1a). To resolve the role of a protein, we interrogate it in the context of the proteins with which it interacts and the cells in which it exists. For instance, H2AFX is a gene that can be involved in homologous recombination or end joining depending on its cellular context (Figure 1b).

Cellular context is critical to understanding protein function and developing molecular therapies. Still, modeling proteins across biological contexts, such as the cell types that they are activated in and the proteins they interact with, remains an algorithmic challenge. Current approaches are context-free: learning on a reference context-agnostic dataset, a single context at a time, or an integrated summary across multiple contexts. As a result, they cannot tailor outputs based on a given context, which can lead to poor predictive performance when applied to a context-specific setting or a never-before-seen context. We develop PINNACLE, a new geometric deep learning approach that generates context-aware protein representations to address these challenges.

Context-specific geometric deep learning PINNACLE model

PINNACLE is a novel geometric deep learning model that learns on contextualized protein interaction networks to produce 394,760 protein representations from 156 cell type contexts across 24 tissues. By leveraging a multi-organ single-cell atlas Tabula Sapiens from CZ CELLxGENE Discover , we construct 156 cell type specific protein interaction networks that are maximally similar to the global reference protein interaction network while maintaining cell type specificity (left and middle panels of Figure 2). We additionally create a metagraph to capture the tissue hierarchy and cell type communication among the cell type specific protein interaction networks (right panel of Figure 2). There are four distinct edge types in the metagraph: cell type to cell type (i.e., cell type interaction), cell type to tissue, tissue to cell type (i.e., tissue membership of the cell type), and tissue to tissue (i.e., parent-child tissue relationship in a tissue ontology). This results in multi-scale networks representing protein, cell type, and tissue information in a unified data representation.

PINNACLE’s algorithm specifies graph neural message passing transformations on multi-scale protein interaction networks . It performs neural message passing with attention for each cell type specific protein interaction network (component 1 in Figure 3) and metagraph (component 2 in Figure 3), and aligns the protein and cell type embeddings using an attention bridge (component 3 in Figure 3). First, PINNACLE learns a trainable weight matrix, node embeddings, and attention weights for each cell type specific protein interaction network. They are optimized based on two protein-level tasks: link prediction (i.e., whether an edge exists between a pair of proteins) and cell type identity (i.e., which cell type a protein is activated in). Secondly, on the metagraph, PINNACLE learns edge type specific trainable weight matrices, node embeddings, and attention weights and aggregates the edge type specific node embeddings via another attention mechanism. These trainable parameters are optimized using edge type specific link prediction (i.e., whether a specific type of edge exists between a pair of nodes). Thirdly, PINNACLE learns attention weights to bridge protein and cell type embeddings. This attention bridge facilitates the propagation of neural messages from cell types and tissues to the cell type specific protein embeddings. It enables PINNACLE to generate a unified embedding space of proteins, cell types, and tissues. Further, the attention bridge enforces cellular and tissue organization of the latent protein space based on tissue hierarchy and cell type communication, enabling contextualization of protein representations.

Context-specific predictions

PINNACLE’s contextual representations can be adapted for diverse downstream tasks in which context specificity may play a significant role. Designing safe and effective molecular therapies is one such task that requires understanding the mechanisms of proteins across cell type contexts. We hypothesize that, in contrast to context-free protein representations, contextualized protein representations can enhance 3D structure-based protein representations for resolving immuno-oncological protein interactions (Figure 4) and facilitate the investigation of drugs’ effects across cell types (Figure 5).

Contextualizing 3D molecular structures of proteins using existing structure-based models is limited by the scarcity of structures captured in context-specific conformations. We show via demonstrative case studies that PINNACLE’s contextualized protein representations can improve structure-based predictions of binding (and non-binding) proteins (Figure 4a). For two immuno-oncological protein interactors, PD-1/PD-L1 and CTLA-4/B7-1, we generate embeddings of each protein using a state-of-the-art structure-based model, MaSIF. We aggregate these structure-based embeddings with a corresponding genomic-based context-free or contextualized protein embedding (i.e., from PINNACLE). By calculating a binding score (i.e., cosine similarity between a pair of protein embeddings), we find that contextualized embeddings enable better differentiation between binding and non-binding proteins (Figure 4b). This zero-shot analysis of contextualizing 3D structure-based representations exemplifies the potential of contextual learning to improve the modeling of molecular structures across biological contexts.

Nominating therapeutic targets with cell type resolution holds the promise of maximizing the efficacy and safety of a candidate drug. However, it is not possible with current models to systematically predict cell type specific therapeutic potential across all proteins and cell type contexts. By finetuning PINNACLE’s contextualized protein representations, we demonstrate that PINNACLE outperforms state-of-the-art, yet context-free, models in nominating therapeutic targets for rheumatoid arthritis (RA) and inflammatory bowel diseases (IBD). We can pinpoint cell type contexts with higher predictive capability than context-free models (Figure 5a). In collaboration with RA and IBD clinical experts, we find that the most predictive cell types are indeed relevant to RA and IBD. Further, examining predictions of individual proteins across cell types allows us to interrogate each candidate target’s therapeutic potential in each cell type context.

PINNACLE is a contextual AI model for representing proteins with cell type resolution. While we demonstrate PINNACLE’s capabilities through cell type specific protein interaction networks, the model can easily be re-trained on any protein network. Our study focuses on single-cell transcriptomic data of healthy individuals, but we expect that training disease-specific PINNACLE models can enable even more accurate predictions of candidate therapeutic targets across cell type contexts. PINNACLE’s ability to adjust its outputs based on the context in which it operates paves the way for large-scale context-specific predictions in biology.  

PINNACLE exemplifies the potential of contextual AI to mimic distinctly human behavior, operating within specific contexts. As humans, we naturally consider and utilize context without conscious effort in our daily interactions and decision-making processes. For instance, we adjust our language, tone, and actions based on the environment and the people we interact with. This inherent ability to dynamically adapt to varying contexts is a cornerstone of human intelligence. In contrast, many current biomedical AI models often lack this contextual adaptability. They tend to operate in a static manner, applying the same logic and patterns regardless of differing biological environments. This limitation can hinder their effectiveness and accuracy in biological systems where context plays a crucial role.

By integrating contextual awareness, models such as PINNACLE can transform biomedical AI. We envision that context-aware models will dynamically adjust their outputs based on the specific cellular environments they encounter, leading to more accurate and relevant predictions and insights. This advancement enhances the functionality of AI models in biology and brings them a step closer to emulating the nuanced and adaptable nature of human thought processes. 

• Research Briefing
• Project website

• Skip to main content
• Accessibility help

Information

We use cookies to collect anonymous data to help us improve your site browsing experience.

Click 'Accept all cookies' to agree to all cookies that collect anonymous data. To only allow the cookies that make the site work, click 'Use essential cookies only.' Visit 'Set cookie preferences' to control specific cookies.

Your cookie preferences have been saved. You can change your cookie settings at any time.

Exploring the link between Learning for Sustainability and attainment

Report setting out the findings and recommendations from a small-scale qualitative research project which examined practitioners’ perceptions of the link between Learning for Sustainability (LfS) and attainment

Supporting documents

There is a problem.

Thanks for your feedback

Your feedback helps us to improve this website. Do not give any personal information because we cannot reply to you directly.

• Open access
• Published: 16 August 2024

A theory-informed deep learning approach to extracting and characterizing substance use-related stigma in social media

• David Roesler 1 ,
• Shana Johnny 2 ,
• Mike Conway 3 &
• Annie T. Chen 1  

BMC Digital Health volume  2 , Article number:  60 ( 2024 ) Cite this article

45 Accesses

Metrics details

Stigma surrounding substance use can result in severe consequences for physical and mental health. Identifying situations in which stigma occurs and characterizing its impact could be a critical step toward improving outcomes for individuals experiencing stigma. As part of a larger research project with the goal of informing the development of interventions for substance use disorder, this study leverages natural language processing methods and a theory-informed approach to identify and characterize manifestations of substance use stigma in social media data.

We harvested social media data, creating an annotated corpus of 2,214 Reddit posts from subreddits relating to substance use. We trained a set of binary classifiers; each classifier detected one of three stigma types: Internalized Stigma, Anticipated Stigma, and Enacted Stigma, from the Stigma Framework. We evaluated hybrid models that combine contextual embeddings with features derived from extant lexicons and handcrafted lexicons based on stigma theory, and assessed the performance of these models. Then, using the trained and evaluated classifiers, we performed a mixed-methods analysis to quantify the presence and type of stigma in a corpus of 161,448 unprocessed posts derived from subreddits relating to substance use.

For all stigma types, we identified hybrid models (RoBERTa combined with handcrafted stigma features) that significantly outperformed RoBERTa-only baselines. In the model’s predictions on our unseen data, we observed that Internalized Stigma was the most prevalent stigma type for alcohol and cannabis, but in the case of opioids, Anticipated Stigma was the most frequent. Feature analysis indicated that language conveying Internalized Stigma was predominantly characterized by emotional content, with a focus on shame, self-blame, and despair. In contrast, Enacted Stigma and Anticipated involved a complex interplay of emotional, social, and behavioral features.

Our main contributions are demonstrating a theory-based approach to extracting and comparing different types of stigma in a social media dataset, and employing patterns in word usage to explore and characterize its manifestations. The insights from this study highlight the need to consider the impacts of stigma differently by mechanism (internalized, anticipated, and enacted), and enhance our current understandings of how each stigma mechanism manifests within language in particular cognitive, emotional, social, and behavioral aspects.

Peer Review reports

Persons with substance use disorders (SUDs) can experience stigma in various forms, including stereotypes, prejudice, and discrimination, and this stigma can have far-ranging consequences for their health, employment, housing, and relationships [ 1 ]. Individuals experiencing stigma may internalize these negative beliefs and feelings, have diminished self-esteem and recovery capital [ 2 , 3 ], and be reluctant to seek treatment [ 4 ].

Interventions focused on stigma reduction in the context of substance use have been limited, and these have tended to focus on structural stigma (e.g., education of professionals that work with persons with SUDs) as opposed to social or self-stigma [ 5 ]. There is also awareness of the bias in words used to describe SUDs, and the need to consider word choice [ 6 , 7 ]. However, despite the potential harms of substance use stigma, our knowledge of how different types of stigma affect persons within the context of SUDs remains limited [ 5 , 8 , 9 , 10 , 11 , 12 ].

In this article, we demonstrate a stigma theory-informed deep learning approach to the task of identifying examples of substance use stigma in a large dataset. To ensure that we capture stigma in the diverse forms in which it occurs, we employ the Stigma Framework [ 13 ], which defines three stigma mechanisms for those who experience stigma: Internalized Stigma , Anticipated Stigma , and Enacted Stigma . The Stigma Framework has been used to characterize stigma processes in various health-related contexts, including problematic substance use [ 11 ] and HIV [ 13 ], and extant literature has sought to develop instruments to assess the experience of these three types of stigma [ 11 ]. To our knowledge, however, prior work has not explored how the three stigma mechanisms are conveyed by the language used in social media. We examine stigma as expressed in social media for two main reasons: 1) previous literature has shown that stigma relating to mental health is endemic in social media [ 14 , 15 ]; and 2) social media can serve an important role in understanding and promoting public health [ 16 , 17 ].

This current study aims to answer the research question: How do the three stigma mechanisms in the Stigma Framework manifest differently in terms of distribution and nature in social media? We take the following approach:

We develop classifiers to identify three stigma mechanisms in an annotated social media dataset and evaluate the performance of these classifiers.

To gain a deeper understanding of the prevalence of the three stigma mechanisms in social media at large, we analyze how each stigma mechanism is distributed in the predictions made by the classifiers on the unseen portion of our data.

To better understand the linguistic expression of the different stigma mechanisms in social media, we identify the highest-ranking features associated with each mechanism and offer illustrative examples.

Related work

Conceptualizations of stigma.

Goffman [ 18 ] influentially defined stigma as “an attribute that is deeply discrediting”, and which reduces the stigmatized “from a whole and usual person to a tainted, discounted one” (p. 3). Goffman described stigma as a product of interactions, and stated that “a language of relations, not attributes, is really needed to describe stigma” [ 18 ] (p.3). The relational nature of stigma was emphasized by subsequent stigma theory [ 19 , 20 ] that characterized stigma as a social process situated in a social context, with Link and Phelan [ 19 ] conceptualizing stigma as a convergence of labeling, stereotyping, separation, status loss, and discrimination, all within a power structure.

To complement existing societal-level conceptualizations of stigma with individual-level ones and create a more comprehensive theory of stigma and its impact, Earnshaw and Chaudoir [ 13 ] proposed the Stigma Framework. In this framework, which draws on stigma theory from a variety of domains [ 19 , 20 , 21 , 22 , 23 ], attention is given to both the mechanisms of stigma employed by those with power, and also the ways that stigma is experienced or adopted by stigmatized individuals. Earnshaw and Chaudoir distinguish three mechanisms employed by those who distance themselves from the “mark” of stigma: prejudice, stereotyping, and discrimination; and three mechanisms (hereafter primarily called “types”) for those who experience stigma: Internalized Stigma , Anticipated Stigma , and Enacted Stigma . Table 1 provides definitions and examples of each of the three types of experienced stigma, in the context of substance use, as defined in Smith et al. [ 11 ]. The stigma mechanisms identified by the Stigma Framework have been assessed in various health-related contexts and have been associated with physical, mental, and behavioral outcomes for those that experience stigma [ 11 , 24 , 25 ].

Despite the existence of different conceptualizations of stigma, there is much that we do not yet understand about stigma processes. In particular, there is a recognized need to more clearly define and characterize the nature of stigma [ 9 , 26 ]; to identify societal and individual-level factors affecting stereotyping, prejudice, and discrimination [ 12 ]; and to develop a more nuanced understanding of how different stigma mechanisms may affect substance use recovery [ 11 ]. In this study, we develop models to identify stigma in a large social media dataset for subsequent qualitative analysis intended to enhance our understanding of the complex interplay of the effects of stigma on the individual within their embedded contexts.

Computational models of stigma detection

Although a multitude of computational models for the detection of abusive language and hate speech in social media texts has been proposed [ 27 , 28 ], the computational detection of social stigma has been less extensively explored. Whereas hate speech is commonly defined as a communicative act of disparagement of a person or group [ 29 ], the arguably broader concept of stigma can include, in addition to direct antagonism, more subtle and systematic forms of discrimination and distancing, of both others and the self [ 1 , 18 , 19 , 30 ]. Research on stigma detection in a variety of specific domains has been conducted, with works on the detection of depression stigma [ 14 ], mental health stigma [ 31 , 32 ], stigmatizing language in healthcare discussions [ 33 ], Alzheimer’s Disease stigma [ 34 ], schizophrenia stigma [ 35 ], and obesity stigma [ 36 ].

Li et al. [ 14 ] produce models for the detection of depression stigma in Mandarin Chinese Weibo posts. In their data, they find only 6% of the posts contain stigmatizing content; however, when training their model, the authors create a balanced corpus of texts (stigmatizing vs. non-stigmatizing). The researchers test logistic regression, multi-layer perceptron (MLP), support vector machine, and random forest classifiers trained in conjunction with a simplified Chinese version of Linguistic Inquiry and Word Count (LIWC) features [ 37 ]. The trained models detect stigmatizing posts and also classify each stigma-positive instance as an instance of one of three depression stigma sub-narratives (‘unpredictability’, ‘weakness’, or ‘false illness’), with the researchers finding best results when using random forest models.

Straton et al. [ 33 ] build a model for the detection of stigmatizing language in Facebook healthcare discussions around the topic of vaccination. In their annotated corpus of postings from anti-vaccination message walls, they find language stigmatizing government organizations and institutions, and in pro-vaccination message walls, they find language stigmatizing the anti-vaccination movement. Using a balanced dataset, the researchers use term frequency-inverse document frequency (TF-IDF) weighted n-grams and LIWC psychological features to train a variety of classifiers, with a convolutional neural network model resulting in the best performance.

Gottipati et al. [ 32 ] perform mental disorder stigma detection on a corpus of mental health-related news articles published by Singapore’s largest media organizations. The authors create an (approximately) balanced dataset of stigmatizing and non-stigmatizing news article titles paired with a sentence from the same article. The researchers create features from TF-IDF weighted n-grams and compare a variety of machine learning classifiers, finding best performance with XGBoost [ 38 ].

To develop a model for detecting stigmatizing language related to mental health, Lee and Kyung [ 31 ] create a corpus of 240 sentence pairs (stigmatizing and non-stigmatizing), entitled the Mental Health Stigma Corpus. The authors fine-tune a BERT-base model [ 39 ] to classify sentences as stigma-positive or stigma-negative and achieve promising results, though the synthetic nature of their dataset may raise questions with regard its ability to generalize to real-world data. We summarize the results of the four stigma detection studies described here in Table  2 .

Although research on health-related stigma detection has been performed in a variety of domains, to our knowledge, all have treated stigma as a single monolithic concept. In this work, we incorporate the three stigma mechanisms (Internalized, Anticipated, and Enacted Stigma) of the Stigma Framework [ 13 ] to better differentiate between different types of stigma experiences, including identifying linguistic features which are most characteristic of each stigma type. For instance, the social media examples that we observed included stigmatizing language (“my sister is a hopeless alcoholic”), reports of stigmatization (“my husband took away the kids and said I’d never get clean”), and the experience of stigma (“I feel so much shame that I can’t tell anyone”).

Based on the effectiveness of BERT contextual embeddings, TF-IDF-weighted n-grams, and LIWC features for the purpose of stigmatizing language detection [ 14 , 31 , 33 ], we experiment with combinations of these resources. Given the prevalence of affect types such as sadness, anxiety, and fear in social media posts discussing experiences of substance use [ 40 ] and prior literature arguing that emotion regulation can be a factor in stigma coping [ 41 , 42 ], we also experiment with count-based features derived from extant affect lexicons and our own handcrafted stigma lexicons. These handcrafted lexicons incorporate affective, social, and behavioral concepts based on stigma theory, including anxiety, depression, and secretive behavior [ 5 , 9 ].

In this study, we employ classifiers to identify three different types of stigma in a social media dataset. We train and evaluate a set of models for each stigma type and then perform a mixed-methods analysis of the data identified by these models. A flowchart overview of our project is depicted in Fig.  1 .

Project overview flowchart

Dataset creation

Harvesting data.

To create our dataset, approximately 160 thousand English-language Reddit posts authored between January 1, 2013 and December 31, 2019 were collected using Pushshift.io [ 43 ]. To capture diverse manifestations of substance use stigma and stigma-related behaviors (including navigation of legality for users), we focused on three substances for this analysis: alcohol, cannabis, and opioids. We selected subreddits related to the three substances of interest (e.g., ‘r/stopdrinking’, ‘r/marijuana’, and ‘r/opiates’) and sampled only thread-initiating posts, as these posts often contain richer descriptions of Redditor’s experiences [ 44 ]. In our previous research [ 40 , 45 ], we found these subreddits contained detailed accounts of both substance use and SUD recovery. Table 3 provides a breakdown of post counts for each subreddit in the harvested Reddit data. Subreddits that allude to or mention recovery or support in subreddit titles, descriptions or rules are labeled with checkmarks.

Sampling for annotation

We observed that posts containing explicit references to stigma were relatively uncommon. To increase the volume of relevant data for annotation and to support subsequent natural language processing, we employed the keyword sampling method used in Chen et al. [ 40 ] to build our annotated corpus. Only the posts that matched a regular expression containing a keyword list were sampled to increase the probability of sampling stigma-related content. The theory-informed keyword list, derived from stigma literature [ 10 , 11 , 24 , 25 ], includes terms with stigma-related connotations (such as ‘shame’, ‘disappoint’, and ‘untrustworthy’) and terms referring to the actors who may be involved in stigma-related experiences (‘family’, ‘co-worker’, ‘husband’). Over the course of the annotation process, this list of keywords was iteratively refined to increase the prevalence of stigma in samples. The final set of sampling keywords is listed in Table  4 . Additionally, subreddits that produced low yields for stigma content (e.g., r/alcohol, r/Petioles, r/trees) were removed from the candidates for annotation sampling. Table 5 shows the breakdown of post counts for each of the subreddits and the distribution of the three stigma types in the annotated dataset.

Annotation process

Three annotators with expertise in informatics, natural language processing, nursing, and public health annotated a total of 2,214 Reddit posts at the span-level for three stigma types based on the Stigma Framework [ 13 ]: Internalized Stigma, Anticipated Stigma, and Enacted Stigma. We developed an annotation guide including definitions, synthetic examples, and instructions for identifying and distinguishing these three stigma types based on extant literature [ 11 , 46 ]. A detailed description of our annotation guidelines is provided as Additional file 1 .

Annotators independently identified passages containing stigma in the posts before discussing and reconciling the annotations. In addition to labeling stigma spans, annotators also labeled posts for substance type and the author’s recovery outlook (positive, neutral, or negative), and identified spans containing mentions of social isolation and labels (e.g., ‘addict’). Table 6 lists pairwise inter-annotator agreement for the three annotators at post level, prior to reconciliation, measured using Cohen’s Kappa [ 47 ]. Overall, pair-wise agreement on the stigma mechanisms reflected moderate agreement [ 48 ], with the highest agreement being for Internalized Stigma. Pair-wise agreement scores on all annotation types varied between 0.66 and 0.71, indicating substantial agreement.

Text segmentation

In the annotated corpus, we observed that Reddit posts ranged in length from 28 characters to 25,743 characters, with a mean length of 1,816 characters (Fig.  2 ). As many posts exceed the 512-token input length limit of the RoBERTa encoder [ 49 ] that we use in our detection model, we opt to chunk posts into text segments. We use the term ‘segment’ to refer to the chunks of text used as inputs to our classifiers, and we use ‘span’ to refer to passages of text within posts labeled by annotators. We map the annotated span labels onto the segments, and then use the labeled segments to train our models. When the trained models make predictions, they first make predictions on individual segments before we map these predictions back to the post level, where, if any segment within a post is predicted as stigma-positive, the entire post is then predicted to be stigma-positive.

Architecture of the hybrid model

Although segmenting posts solves the input limitation issue, this also increases class imbalance in our dataset. In our annotated corpus, we find that within individual posts, the stigma-positive spans can be infrequent, with multi-paragraph posts sometimes only containing a few stigma-positive words. As a result, when we split the Reddit posts into smaller units (such as sentences), we produce far more negative examples than positive ones, and the portion of stigma-positive texts in our corpus decreases (Table  7 ). When splitting posts down to the level of sentences, we see severe class imbalance, with only 1.69% of the data containing Enacted Stigma.

Class imbalance can result in classifiers which perform well for the majority class, but poorly for the minority class [ 50 , 51 ]. To mitigate class imbalance, we experimented with a variety of segmentation lengths, and found the best performing length to be approximately 600 characters. At this length, text segments seem to be short enough to mitigate the amount of irrelevant information (features unrelated to stigma), but they also remain lengthy enough to keep the imbalance of classes from becoming severe.

To build segments from our post data, we begin by splitting all posts into sentences using Natural Language Toolkit (NLTK) 3.5 [ 52 ]. We then join the resulting sentences in the order they appear in the post until the threshold value of 600 characters in length is reached, after which, a new segment is started. We do not split sentences, and thus segments vary in length. After segmenting texts, labels are assigned to segments by checking for overlap between segment spans and annotation spans. The texts are then pre-processed by removing URLs, hyperlinks, and other HTML-related text residue.

Substance use stigma detection model

To identify Reddit posts in the harvested data that have a high probability of containing reports and instances of substance use stigma, we create binary classifiers for each stigma type: Internalized Stigma, Anticipated Stigma, and Enacted Stigma. Because each segment of input text may be stigma-positive for multiple stigma types, we treat this classification task as a set of independent binary classification tasks rather than a single multi-class classification task.

We utilize a RoBERTa encoder [ 49 ] as the main component of the classifier, and also make use of n-gram features, features derived from affective and psychological lexicons, and handcrafted features to enrich the model with external knowledge relevant to the task. To integrate RoBERTa embeddings with the additional features, we use a hybrid model (Fig.  3 ) based on Prakash et al. [ 53 ], where the first stage is MLP pre-training. The MLP is pre-trained on a concatenated vector of TF-IDF weighted n-grams, features derived from the NRC Footnote 1 Emotional Intensity Lexicon [ 54 ], features derived from Wordnet-Affect [ 55 ], features generated from the LIWC 2015 lexicon [ 37 ], and handcrafted substance use stigma features.

Histogram of post character length

After pre-training is complete, the trained MLP weights are used along with a pre-trained RoBERTa encoder in the fine-tuning process. The < s > token output of the RoBERTa encoder and the MLP output are normalized and then concatenated before being passed to an MLP classifier head, which outputs the probability that a given sequence of text contains the current type of substance use stigma.

Feature vector construction

When building input to the MLP component of the classifier, we create the following feature sets:

TF-IDF weighted n-grams (TF-IDF)

To create TF-IDF features, we remove English stop words from the text using the NLTK 3.5 package, and then use Scikit-learn 1.8 [ 56 ] to create TF-IDF weighted n-grams in the range (2, 6) with a dimensionality of 10,000.

NRC affective intensity features (NRC)

We include NRC features [ 54 ] to take advantage of the scaled emotional intensity scores that the NRC lexicon provides. We use the NRC Emotional Intensity Lexicon to generate 10-dimensional intensity-scaled affect features (with each dimension corresponding to one of the concepts listed in Table  8 ). To produce feature vectors, we follow the method of Babanejad et al. [ 57 ], who create ‘EAISe’ representations (Emotion Affective Intensity with Sentiment Features) for their sarcasm detection model.

Wordnet Affect features (WNA)

Wordnet-Affect [ 1 ], developed based on Wordnet 1.6 [ 58 ], enabled us to incorporate finer-grained affect types. Based on literature relating to substance use, stigma, and emotion and an examination of our Reddit corpus, we identified 13 Wordnet-Affect concepts that were relevant to substance use stigma (Table  8 ) and constructed lexical sets around each of the 13 Wordnet-Affect concepts using Wordnet. Using these sets, we generate 13-dimensional feature vectors using the same method that we use to build our NRC vectors.

LIWC features

Linguistic, grammatical, and psychological features are generated using LIWC 2015 software [ 37 ]. We remove the ‘word count’ feature and retain all others, resulting in a 92-dimensional vector.

Substance use stigma features (INT / ANT / ENA)

We create handcrafted lexicons (identified as ‘INT’, ‘ANT’, and ‘ENA’) to capture affective, behavioral, and social concepts related to each stigma type. These lexicons were developed through examination of TF-IDF weighted n-gram chi-square rankings for the training data, identification of recurring concepts in the stigma-positive examples of the training data that corresponded to concepts from stigma literature and survey instruments [ 10 , 11 , 24 , 25 , 46 , 59 ], and iterative building and evaluation of lexical sets for each concept using a validation set. For Anticipated Stigma, an associated behavior such as concealment [ 25 ] is included in the ‘secrecy’ concept through keywords such as ‘sneak’, ‘hid’, or ‘throwaway’ (used in mentions of ‘throwaway’ Reddit accounts created to preserve anonymity). The six concepts included in each feature set is listed here in Table  8 , and the complete list of keywords included in each concept is listed in Additional file 2 . To create 6-dimensional feature vectors, we start with a vector of zeros. We then search text segments for each of the words in our lexical sets. If a lexicon word is present, we add ‘1’ to the concept dimension associated with the word.

After building all feature vectors, we separately normalize each set of features, then concatenate them to form a 10,121-dimensional input vector.

Data handling

Training sets are sampled from our segment-level data and contain a mixture of stigma-positive and stigma-negative texts. In development, the best results for MLP and hybrid models were found when using a training set with a negative to positive rate of 3:1, and we use this rate to train our final hybrid models. Our validation and test sets are randomly sampled from 10% of the post-level data. After a set of Reddit posts is sampled, the constituent segments are retrieved and used as the evaluation set.

Hyperparameters

We train all models on a single Tesla A100 GPU on the Google Colab platform. Training is implemented using Pytorch 1.12 [ 60 ] and the Huggingface library [ 61 ]. We pre-train our MLP for 30 epochs using the AdamW optimizer with a learning rate of 5.e-5 (controlled by a learning rate scheduler) and a batch size of 32. We determine the optimal threshold for positive class F1 after each training epoch using a precision-recall curve on the validation set. The best model is checkpointed based on positive class F1 performance.

During fine-tuning, we fine-tune cased RoBERTa-base (123 million parameters) for 10 epochs with a learning rate of 5.e-5 and batch size of 32. We also experiment with the cased RoBERTa-large encoder (354 million parameters), and when fine-tuning RoBERTa-large, we train for 10 epochs with a learning rate of 7.e-6 and a batch size of 32. Less than 15 min of GPU time were required to train a single hybrid model.

Model evaluation

As we sought to identify the stigma-positive Reddit posts within the unseen harvested Reddit data, we evaluate each model’s predictions at the post-level by mapping segment predictions to each post. We compare the performance of models by reporting the mean macro F1 score of five runs on the same data, using different random seeds. We list results from variations of hybrid models utilizing different sets of features. As a baseline for comparison to the hybrid models, we list results using RoBERTa-base and RoBERTa-large with a simple classifier head, trained on a balanced training set (via undersampling), and using the same threshold moving method as used in our hybrid model.

Improvements over the RoBERTa-only baselines are considered significant at a significance level (α) of 0.05 according to McNemar’s test [ 62 ] with false discovery rate (FDR) correction [ 63 ]. McNemar’s significance test has been considered appropriate for binary classification tasks [ 64 ]; thus, we employ it on the predictions of the paired models. Because we make multiple hypothesis tests in our comparisons, FDR correction is applied to p -values.

To explore each feature set’s potential for use in stigma detection, we also considered the results of MLP evaluation on single feature sets and set combinations. We use an MLP for this comparison rather than a hybrid model since in the hybrid models, redundancies in the information encoded by feature set combinations and the information encoded by RoBERTa can make the relative performance contribution of each feature set difficult to disentangle. We also perform exploratory feature ranking of all features using the chi-square measure to explore the strength of association between each feature and its relevant stigma type. The feature selection tools of the Scikit-learn package were used to implement this experiment [ 56 ].

Last, we perform an error analysis of the hybrid model’s predictions. This evaluation not only informs future improvements on our approach, but also provides insights into difficulties that arise in the perception and experience of stigma.

Mixed-methods analysis

Mixed-methods research can facilitate research that cannot be answered using a single method. Though there is controversy concerning what constitutes mixed-methods research, integrating quantitative and qualitative approaches is considered increasingly important, and extant literature has observed and demonstrated that the definition of mixed-methods research will continue to grow [ 65 , 66 ]. In this study, we leverage both quantitative and qualitative methods for various affordances identified by Doyle et al. [ 66 ] including: triangulation, completeness, and illustration of data.

We performed a mixed-methods analysis to: 1) estimate the amount of stigma in the larger social media data store; and 2) characterize the nature of the different stigma mechanisms. First, we characterized the presence of stigma in the unseen portion of the harvested Reddit data by examining patterns in the distribution of stigma predictions with respect to substance and subreddit, and the correlations between stigma type predictions. We employ chi-square tests to compare the presence of the stigma mechanisms in the three substances. As a chi-square test of independence on its own merely shows that there is an association between two nominal variables and does not show which cells are contributing to the lack of fit [ 67 , 68 ], we calculated standardized Pearson residuals. A standardized Pearson residual exceeding two in absolute value in a given cell indicates a lack of fit [ 67 , 68 ]. Second, we considered the feature rankings and the instances of predicted stigma in the test data in concert to illustrate how the three types of stigma concretely manifest in cognitive and emotional processes, social interactions, and behaviors in everyday life. To protect the identities of the posters, we employ synthetic quotes in our illustration [ 69 ].

Results and discussion

Model performance and evaluation, overall model performance.

Table 9 lists the results of post-level stigma detection for the three stigma types. For all three stigma types, we found hybrid models that significantly outperformed their respective RoBERTa-only baselines, with the largest gain observed for the Anticipated Stigma RoBERTa-large hybrid model using only the handcrafted stigma features (+ 7.08 F1). These results provide evidence that n-gram, affective, behavioral, and social features can be combined with contextual embeddings to improve substance use stigma detection.

In the results of MLP evaluation (Table  10 ), the handcrafted lexicons (STIG) appeared to be relatively effective resources for the task of stigma detection, and the other feature sets (NRC, WNA, and LIWC) also appear to be viable resources (to varying degrees). For individual feature sets, the handcrafted stigma lexicons appeared to provide the best results for Internalized Stigma and Anticipated Stigma, whereas LIWC provided best results for Enacted Stigma. For feature set combinations, adding additional feature sets usually led to improvement for MLP models (with some exceptions), although the combination of all features only outperformed the handcrafted stigma lexicons for the case of Enacted Stigma.

Comparing performance by stigma mechanisms and contributing features

The results in Tables  9 and 10 show that, overall, scores for Internalized Stigma are higher than for the other stigma types; Internalized Stigma was the most frequent of the three stigma types in the annotated corpus (making it the stigma type with the greatest number of examples). When performing exploratory feature ranking of all features (Table  11 ), count-based features had stronger associations (higher chi-square scores) with Internalized Stigma than they did with the other stigma types. Affective concepts such as ‘shame’ and ‘guilt’ had strong relationships with Internalized Stigma, which likely benefitted performance.

Overall performance for Anticipated and Enacted Stigma was weaker than for Internalized Stigma. There may be a number of reasons for this. First, Anticipated and Enacted Stigma had fewer examples and relatively weaker associations with count-based features in comparison with Internalized Stigma. For Enacted Stigma, the highest-ranking features were labels such as ‘alcoholic’ and ‘junkie’, which were fairly common in the entire corpus. Labeling terms such as ‘alcoholic’ may be used to enact stigma, but they may also be used to express membership in recovery groups and are a part of ‘recovery dialects’ used within such groups [ 2 ]. Moreover, labeling terms may also be appropriated by members of stigmatized groups to increase perceptions of power for the stigmatized individual or group [ 70 ]. The variety of motivations behind the uses of such labeling terms such as ‘junkie’ may be a limiting factor to their viability as features for stigma detection.

Another potential factor for the weaker performance for Anticipated and Enacted Stigma may be their social nature. Whereas Internalized Stigma frequently focus on a single entity (the post author), with feature rankings showing strong relationships with inward features (n-grams such as ‘i ashamed’), both Anticipated and Enacted Stigma involved other actors. With Anticipated Stigma, the highest ranking features involved concealment of use (ANT_secrecy) and other actors (ANT_social), as post authors were concerned about concealing their use from others. With Enacted Stigma, there was a wide variety of actors involved in the relationships between the stigmatizer and the person(s) being stigmatized (e.g. ‘family to partner’, ‘partner to society’, ‘co-workers to society’). Further, while Internalized Stigma frequently focused on the act of shaming oneself, Enacted Stigma involved a more diverse set of verbs/actions through which stigma was performed (e.g., disapproving looks, expressions of distrust, arrests, searches, evictions, insults, generalizations, coerced drug tests, denial of healthcare services, termination of employment, termination of personal relationships). Many of the verbs related to these stigmatizing actions were included in the ENA_stigmatizing_actions and ENA_trust features, which ranked second and third, respectively, in the feature ranking.

Model performance by stigma type followed a similar pattern to that of inter-annotator agreement across stigma types (Table  6 ), in which annotators found highest agreement on Internalized Stigma and less agreement on Anticipated and Enacted Stigma. The complexities involved in identifying these two stigma types seemed to be a challenge for both human annotators as well as the models.

Error analysis

We provide an error analysis of the Anticipated and Enacted Stigma models to gain insights into the challenges involved in detecting these stigma types. We give synthetic quotes based on our data to demonstrate error types, with features typical of Anticipate or Enacted Stigma texts bolded.

Temporal errors

We observed that both the Anticipated and Enacted Stigma hybrid models produced false positives for texts which do not match the temporal requirements of their respective stigma type (future for Anticipated Stigma, present or past for Enacted Stigma). The following example (a false positive for Enacted Stigma due to temporal mismatch), is representative of this error type:

If I come clean, my family will disown me – that isn’t even an option.

For the RoBERTa-only baseline models, this error type was noticeably less frequent. This may be a limitation of the use of count-based features in the hybrid models, as the model may weighting keywords such as disown more heavily than the tense-related syntactic information that has been shown to be encoded by BERT [ 71 ].

Stigmatizing quitters

During annotation, we observed that individuals abstaining from substance use were pressured by persons who engaged in substance use, often in the context of alcohol use when it is normalized in home or work-related settings. Though this behavior was not annotated as stigma, when it appeared in texts, it led to false positive predictions by both the baseline and hybrid models, and is exemplified by the following excerpt:

I told my mother I quit drinking and she laughed at me. I quit in May and have avoided telling my family because they drink a lot and I didn't want to put up with the questions or judgement .

In examples like this, the model seems to leverage features relevant to stigma ( she laughed at me , judgement ) while failing to learn cues that indicate the mother is an alcohol user critical of another user’s abstinence.

Motivations

Both the baseline and hybrid models for Anticipated and Enacted Stigma were prone to produce false positives for texts where typical features of stigma are present, but the motivation behind an action potentially construed as stigmatizing is unrelated to stereotyping, prejudice, or discrimination. In the following example, a partner appears to terminate a relationship due to apathetic behavior rather than stigma, and thus should be labeled as stigma-negative:

I struggled for a long time with the sadness that comes with addiction, so the feelings of apathy that followed it seemed like a relief. Eventually, they resulted in my partner breaking up with me.

Although BERT models have been demonstrated to encode information that can be leveraged to make predictions about causality [ 72 ], interpreting the motivations behind the actions described in texts can be a difficult task even for human judgement. We further discuss this issue in our limitations section.

Characterizing the presence of stigma in the unexplored data

To better understand how the three stigma mechanisms outlined in the Stigma Framework manifest within our social media dataset, we employed the classifiers to identify instances of the stigma types in the previously unexplored portion of our collected Reddit data ( n  = 161,448). The distribution of stigma predictions across subreddits is presented in Table  12 . Overall, the portion of stigma-positive predictions for each type were noticeably lower than the portions seen in the annotated data (Table  5 ). This outcome aligns with expectations, given that: 1) keyword sampling was used to increase the proportion of stigma in the annotated data; and 2) in the unexplored data, a larger portion of posts originated from subreddits focused on general substance use, rather than on support or recovery. In both the predictions and annotations, we observed that, for all three substance types, the estimated stigma proportion was highest for support-focused subreddits, where posters often described challenging experiences relating to their attempts at recovery.

With respect to alcohol and cannabis, Internalized Stigma appeared to be the most common of the three stigma types. The focus on the self makes intuitive sense given the first-person viewpoint of social media narratives, and the prominent features of Internalized Stigma (Table  11 ) suggest that these data could serve as a rich source for future research on how individuals may seek to internally reconcile the cognitive and emotional aspects of shame and guilt that accompany Internalized Stigma.

However, in the case of opioids, we observed a higher frequency of Anticipated Stigma compared to Internalized Stigma. Chi-square tests examining the presence of the three stigma mechanisms in the three substances, with the standardized Pearson residual for Anticipated Stigma x Opioids, also confirm that the observed presence of Anticipated Stigma exceeds the expected in that case (see Additional file 3 ).

Co-occurrence of the three stigma mechanisms

We also explored the extent to which the stigma mechanisms co-occurred in the data. Figure  4 shows a Pearson correlation matrix between stigma labels for text segments in the annotated data (left) and also for the predictions on the unseen data (right). The largest correlation score is a value of 0.11 between Internalized and Anticipated Stigma (in the annotated data), indicating that text segments with multiple stigma labels are relatively infrequent in the annotated data. Although we observed some concepts were shared across stigma types in the feature rankings, such as labeling terms (e.g., ‘addict’), the relatively low correlation between paired stigma types illustrates the utility of developing separate models for each stigma type. Furthermore, this underscores the potential utility of the three stigma types distinguished by the Stigma Framework [ 13 ] for future research in clarifying the mechanisms by which stigma can affect persons with SUDs.

Pearson correlation between stigma types for text segments in the annotated dataset (left) and the predictions on the unseen data (right)

Exploring the relationship between language and stigma experience

To characterize the nature of stigma as manifested in social media, we consider the feature rankings associated with each stigma type, along with the instances of stigma in the test data. Figure  5 depicts the concepts from the handcrafted stigma lexicons that were among the highest-ranking features for each stigma type, along with synthetic examples. Among the posts associated with Internalized Stigma, we observed an abundance of affective content (shame, self-blame, and despair). Our examination of the test data further uncovers that posts containing shame and self-blame also often involved the poster using self-deprecating language (in the form of pejoratives) and labels to describe themselves, and express feelings of weakness and perceptions of failure.

Conceptual differentiation of stigma types. All examples are synthetic quotes that resemble the phenomena and sentiment observed in the data

For Anticipated and Enacted Stigma, emotion was still important, but social and behavioral features were also prominent (i.e., ANT_social, ENA_stigmatizing_actions). The ‘ANT_social’ lexicon includes possible members of a user’s social circle (e.g., ‘parents’, ‘partner’, ‘friend’). Since, by definition, Internalized Stigma is focused on the self, Anticipated Stigma is focused on one’s expectation of how they are perceived by others, and Enacted stigma, by stigmatizing behavior, these associations make intuitive sense. The social media data highlights additional features tied to Anticipated Stigma, such as secretive behavior, concern over how one is perceived, and a fear of disappointing others. Notably, the theme of concealment, especially from close relations like family members, partners, or employers, is prominent in the Anticipated Stigma texts (as exemplified in the examples 3–5 in Fig.  5 ).

Enacted Stigma often involved the use of labels to describe another person, and as seen in the final two examples of Fig.  5 , the usage of these terms can be descriptive (‘He is always drunk’) or may have judgmental motivations in their usage (‘down-and-out junkies’). Stigmatizing actions related to judging, disparaging, or confronting others figured prominently in terms of this type of stigma, and could involve many different pairs of stigmatizer and stigmatized persons (e.g., parent–child, child-parent, friends, partners, co-workers, and the poster feeling stigmatized by the public, people, or society at large). Features related to trust also ranked highly for Enacted Stigma, corresponding to previous stigma research which identified ‘untrustworthiness’ as a common stereotype espoused by user’s family members [ 24 ].

Other phenomena to consider were instances in which multiple stigma types were present. The third text in Fig.  5 exemplifies a common scenario for the pairing of Internalized Stigma and Anticipated Stigma, with posters expressing reticence to interact with others due to their own shame. Text segments containing all three stigma types were relatively rare in the annotated corpus (0.78% of all stigma-positive segments), though the fifth example in Fig.  5 illustrates an instance where an author appears to negatively judge persons experiencing SUDs, describe concealment of their own use, and express internal guilt for their use, all within a relatively brief sequence of text.

Similar to Straton et al. [ 33 ], we observed that the LIWC categories for emotional tone and clout showed fairly strong relationships with stigma; however, we observed a limited relation to stigma for the remaining 90 LIWC categories. The clout feature, derived from ratios of personal pronoun frequencies, is based on Kacewicz et al. [ 73 ], who found that high-status authors consistently used more 1st person plural (e.g., ‘we’, ‘our’) and 2nd person singular (‘you’) pronouns, whereas low-status authors were more frequently self-focused and used more 1st person singular pronouns (‘I’, ‘me’). This may explain the effectiveness of the clout feature for predicting Internalized Stigma (low clout scores appeared to be indicative of Internalized Stigma), which is heavily focused on inner experiences, with heavy use of 1st person pronouns. The LIWC emotional tone feature [ 74 ] calculates the difference between positive emotion word count and negative emotion word count, with higher scores indicating greater overall positivity. The generally negative emotional content of stigma-positive texts is a likely factor for the high ranking of the tone feature for all three stigma types.

Discussion and limitations

In this study, our objective was to investigate how the three different stigma mechanisms in the Stigma Framework manifest differently in terms of distribution and nature in a social media dataset. Through an analysis of feature rankings, the distribution of predictions, and specific instances of stigma in our data, we discerned distinct patterns across Internalized, Anticipated, and Enacted Stigma. Furthermore, we characterized the language used to convey and describe each of these three mechanisms.

In terms of the distributions of the three stigma mechanisms, we observed that Internalized Stigma was the most prevalent stigma type with respect to alcohol and cannabis. However, in the case of opioids, Anticipated Stigma was more frequent than Internalized Stigma. Though these patterns were only observed in a single dataset and further exploration of the presence of different stigma mechanisms in other data is needed, it is worthwhile to consider these findings in the context of the larger societal concern about opioid use. Extant literature emphasizes that great care must be used in crafting public health messaging concerning opioid addiction due to the potential for increased stigmatization of those who use opioids [ 75 ]. The social environment surrounding opioid use appears to lead to greater anticipation of stigma and a tendency to conceal behavior, compared to the environments surrounding cannabis and alcohol. Thus, it may be important to focus on the portrayal of opioid use, anonymous forms of support, and an emphasis on support for interpersonal interactions in the context of opioid use.

Additionally, our study considered the nature of language used to express stigma as it manifests in social media. This exploration not only confirms that language is a powerful vehicle for expressing stigma, as established in prior literature [ 2 ], but also illuminates the nuanced relationship between word usage and specific stigma types, and the pivotal roles of affect, social perceptions, personal interactions, and behavior in the expression of stigma, in social media. In the social media data, we found that Internalized Stigma is predominantly characterized by emotional content, with a focus on shame, self-blame, and despair. In contrast, Enacted Stigma and Anticipated involve a complex interplay of emotional, social, and behavioral features. The former encompasses stigmatizing behaviors and issues of trust, while the latter centers on expectations of external perceptions and the fear of disappointing others. For Anticipated Stigma, the feature analysis demonstrated that issues of concealment were prominent, along with the presence of close interpersonal relationships.

Insights from this study can serve as priorities in the design of stigma reduction interventions. For example, the high-ranking features from the Enacted Stigma lexicon include both stigmatizing actions such as confronting and blaming, as well as indicators of trust (e.g., expressed as disappointment, suspicion, or a lack of respect for privacy). In future intervention development, the integration of components addressing these core issues is critical.

Overall, our findings improve our understanding of stigma mechanisms in social media discourse and could also inform the development of targeted interventions that address the challenges of those affected by stigma. Furthermore, the adaptability of our lexicons to stigma research in other contexts, such as HIV/AIDS or disordered eating, where similar emotions, behaviors (e.g., hiding, concealment), and attitudinal constructs such as trust [ 24 , 76 ] are at play, hold promise for broader applications beyond substance use.

Limitations

Although the purposive sampling used in this study allowed us to develop a sufficient corpus of stigma-positive texts within a reasonable amount of time, our sampling method may also be viewed as one of its limitations. By sampling from a limited set of subreddits focused on substance use, we realize that our detection model may not generalize to other types of texts. Additionally, since keyword matching enrichment was used during the sampling process, the distribution of texts in our corpus differs from that of the substance recovery subreddits which they were sampled from. When making predictions on random samples, our models may have faced performance issues due to the increased imbalance between stigma-positive and stigma-negative texts.

To facilitate the aims of this research, we sought to identify stigma and accounts of stigma within social media narratives. In many of the possible instances of stigma that appear, the motivations behind the potentially stigmatizing actions are unclear or unstated. For posts containing sequences such as ‘my parents kicked me out of the house’, it may be difficult to determine whether the parents’ actions are motivated by stigma or by other factors. Causal ambiguity can lead our models to produce errors, and also lead to disagreement among our annotators. Collection and triangulation of data collected through other means, such as interview, survey, or diary data, could perhaps complement insights from social media.

In this study, we performed an examination of stigma surrounding substance use within the realm of social media. Our approach encompassed data collection, corpus annotation, and the development of binary classifiers tailored to detect three different stigma mechanisms. By synergizing contextual embeddings with count-based features, we achieved models that exhibited superior performance across all three stigma categories compared to RoBERTa-only baselines. Through a mixed-methods analysis of the model's predictions, we unraveled critical insights into the relations of word usage to the expression of different types of stigma. Affective, social, and behavioral features emerged as pivotal components in the expression of substance use stigma.

Our main contributions include: demonstrating a theory-based approach to extracting and comparing different types of stigma in a large social media dataset, and employing patterns in word usage to explore and characterize its manifestations. The insights from this study highlight the need to consider the impacts of stigma differently by mechanism (internalized, anticipated, and enacted), and enhance our current understandings of how the stigma mechanisms manifest within language in particular cognitive, emotional, social, and behavioral aspects. Moving forward, we envisage further analysis of stigma instances in our dataset to glean insights into how individuals navigate the challenges they encounter, informing the development of more effective stigma reduction strategies. Furthermore, the concepts encapsulated in our handcrafted lexicons hold promise for future stigma research in diverse contexts, extending the applicability of our findings beyond substance use disorders.

Availability of data and materials

Stigma datasets and models trained to detect stigma could potentially be used by bad actors to target vulnerable individuals. In order to reduce the risk of any potential harms to the authors of the sensitive posts examined in our research, we do not share our models or annotated dataset publicly.

National Research Council Canada.

Abbreviations

Substance use disorder

Multi-layer perceptron

Linguistic inquiry and word count

Term frequency-inverse document frequency

Bidirectional encoder representations from transformers

Robustly optimized bidirectional encoder representations from transformers

National research council Canada

Natural language toolkit

Wordnet-affect

Internalized stigma feature lexicon

Anticipated stigma feature lexicon

Enacted stigma feature lexicon

Kulesza M, Ramsey S, Brown R, Larimer M. Stigma among Individuals with Substance Use Disorders: Does it Predict Substance Use, and Does it Diminish with Treatment? J Addict Behav Ther Rehabil. 2014;3(1):1000115. https://doi.org/10.4172/2324-9005.1000115 .

Article   CAS   PubMed   PubMed Central   Google Scholar  

Ashford RD, Brown AM, Ashford A, Curtis B. Recovery dialects: A pilot study of stigmatizing and nonstigmatizing label use by individuals in recovery from substance use disorders. Exp Clin Psychopharmacol. 2019;27(6):530–5. https://doi.org/10.1037/pha0000286 .

Article   PubMed   PubMed Central   Google Scholar  

Bozdağ N, Çuhadar D. Internalized stigma, self-efficacy and treatment motivation in patients with substance use disorders. J Subst Use. 2022;27(2):174–80. https://doi.org/10.1080/14659891.2021.1916846 .

Article   Google Scholar  

Hammarlund R, Crapanzano KA, Luce L, Mulligan L, Ward KM. Review of the effects of self-stigma and perceived social stigma on the treatment-seeking decisions of individuals with drug- and alcohol-use disorders. Subst Abuse Rehabil. 2018;9:115–36. https://doi.org/10.2147/SAR.S183256 .

Livingston JD, Milne T, Fang ML, Amari E. The effectiveness of interventions for reducing stigma related to substance use disorders: a systematic review. Addiction. 2012;107(1):39–50. https://doi.org/10.1111/j.1360-0443.2011.03601.x .

Ashford RD, Brown AM, Curtis B. Substance use, recovery, and linguistics: The impact of word choice on explicit and implicit bias. Drug Alcohol Depend. 2018;189:131–8. https://doi.org/10.1080/07347324.2019.1585216 .

Volkow ND, Gordon JA, Koob GF. Choosing appropriate language to reduce the stigma around mental illness and substance use disorders. Neuropsychopharmacol. 2021;46(13):2230–2. https://doi.org/10.1038/s41386-021-01069-4 .

Brown SA. Standardized measures for substance use stigma. Drug Alcohol Depend. 2011;116(1):137–41. https://doi.org/10.1016/j.drugalcdep.2010.12.005 .

Article   PubMed   Google Scholar  

Kulesza M, Larimer ME, Rao D. Substance Use Related Stigma: What we Know and the Way Forward. J Addict Behav Ther Rehabil. 2013;2(2). https://doi.org/10.4172/2324-9005.1000106 .

Kulesza M, Watkins KE, Ober AJ, Osilla KC, Ewing B. Internalized stigma as an independent risk factor for substance use problems among primary care patients: Rationale and preliminary support. Drug Alcohol Depend. 2017;180:52–5. https://doi.org/10.1016/j.drugalcdep.2017.08.002 .

Smith LR, Earnshaw VA, Copenhaver MM, Cunningham CO. Substance use stigma: Reliability and validity of a theory-based scale for substance-using populations. Drug Alcohol Depend. 2016;162:34–43. https://doi.org/10.1016/j.drugalcdep.2016.02.019 .

Corrigan P, Schomerus G, Shuman V, Kraus D, Perlick D, Harnish A, et al. Developing a research agenda for understanding the stigma of addictions Part I: Lessons from the Mental Health Stigma Literature. Am J Addict. 2017;26(1):59–66. https://doi.org/10.1111/ajad.12458 .

Earnshaw VA, Chaudoir SR. From Conceptualizing to Measuring HIV Stigma: A Review of HIV Stigma Mechanism Measures. AIDS Behav. 2009;13(6):1160–77. https://doi.org/10.1007/s10461-009-9593-3 .

Li A, Jiao D, Zhu T. Detecting depression stigma on social media: A linguistic analysis. J Affect Disord. 2018;232:358–62. https://doi.org/10.1016/j.jad.2018.02.087 .

Article   CAS   PubMed   Google Scholar  

Li A, Jiao D, Liu X, Zhu T. A Comparison of the Psycholinguistic Styles of Schizophrenia-Related Stigma and Depression-Related Stigma on Social Media: Content Analysis. J Med Internet Res. 2020;22(4): e16470. https://doi.org/10.2196/16470 .

Clark O, Lee MM, Jingree ML, O’Dwyer E, Yue Y, Marrero A, et al. Weight Stigma and Social Media: Evidence and Public Health Solutions. Front Nutr. 2021;8:739056.

Dredze M. How Social Media Will Change Public Health. IEEE Intell Syst. 2012;27(4):81–4. https://doi.org/10.1109/MIS.2012.76 .

Goffman E. Stigma: Notes on the management of spoiled identity. New York: Simon and Schuster; 2009. p. 52, 65.

Link BG, Phelan JC. Conceptualizing Stigma. Annu Rev Sociol. 2001;27(1):363–85. https://doi.org/10.1146/annurev.soc.27.1.363 .

Parker R, Aggleton P. HIV and AIDS-related stigma and discrimination: a conceptual framework and implications for action. Soc Sci Med. 2003;57(1):13–24. https://doi.org/10.1016/S0277-9536(02)00304-0 .

Brewer MB, Brown RJ. Intergroup relations. In: Gilbert DT, Fiske ST, Lindzey G, editors. The handbook of social psychology. 4. New York: Oxford University Press; 1998.

Google Scholar  

Meyer IH. Prejudice, Social Stress, and Mental Health in Lesbian, Gay, and Bisexual Populations: Conceptual Issues and Research Evidence. Psychol Bull. 2003;129(5):674–97. https://doi.org/10.1037/0033-2909.129.5.674 .

Phelan J, Link BG, Dovidio JF. Stigma and Prejudice: One Animal or Two? Soc Sci Med. 2008;67(3):358–67. https://doi.org/10.1016/j.socscimed.2008.03.022 .

Earnshaw V, Smith L, Copenhaver M. Drug Addiction Stigma in the Context of Methadone Maintenance Therapy: An Investigation into Understudied Sources of Stigma. Int J Ment Health Addict. 2013;11(1):110–22. https://doi.org/10.1007/s11469-012-9402-5 .

Quinn DM, Earnshaw VA. Concealable Stigmatized Identities and Psychological Well-Being. Soc Personal Psychol Compass. 2013;7(1):40–51 ( https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3664915/ ).

Crapanzano KA, Hammarlund R, Ahmad B, Hunsinger N, Kullar R. The association between perceived stigma and substance use disorder treatment outcomes: a review. Subst Abuse Rehabil. 2018;10:1–12. https://doi.org/10.2147/SAR.S183252 .

Schmidt A, Wiegand M. A Survey on Hate Speech Detection using Natural Language Processing. In: Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media. Valencia, Spain: Association for Computational Linguistics; 2017. p. 1–10. https://doi.org/10.18653/v1/W17-1101 .

Chapter   Google Scholar  

Yin W, Zubiaga A. Towards generalisable hate speech detection: a review on obstacles and solutions. PeerJ Comput Sci. 2021;7:e598. https://doi.org/10.7717/peerj-cs.598 .

Nockleby JT, Levy LW, Karst KL, Mahoney DJ. Encyclopedia of the American constitution. Detroit, MI: Macmillan Reference; 2000. p. 1277–9.

Allport GW, Clark K, Pettigrew T. The Nature of Prejudice. 1954.

Lee MH, Kyung R. Mental Health Stigma and Natural Language Processing: Two Enigmas Through the Lens of a Limited Corpus. In: 2022 IEEE World AI IoT Congress (AIIoT). 2022. p. 688–91. https://doi.org/10.1109/AIIoT54504.2022.9817362 .

Gottipati S, Chong M, Kiat A, Kawidiredjo B. Exploring Media Portrayals of People with Mental Disorders using NLP: In: Proceedings of the 14th International Joint Conference on Biomedical Engineering Systems and Technologies. p. 708–15. https://doi.org/10.5220/0010380007080715 .

Straton N, Jang H, Ng R. Stigma Annotation Scheme and Stigmatized Language Detection in Health-Care Discussions on Social Media. In: Proceedings of the 12th Language Resources and Evaluation Conference. Marseille, France: European Language Resources Association; 2020. p. 1178–90 https://aclanthology.org/2020.lrec-1.148 .

Oscar N, Fox PA, Croucher R, Wernick R, Keune J, Hooker K. Machine Learning, Sentiment Analysis, and Tweets: An Examination of Alzheimer’s Disease Stigma on Twitter. J Gerontol B. 2017;72(5):742–51. https://doi.org/10.1093/geronb/gbx014 .

Jilka S, Odoi CM, van Bilsen J, Morris D, Erturk S, Cummins N, et al. Identifying schizophrenia stigma on Twitter: a proof of principle model using service user supervised machine learning. Schizophr. 2022;8(1):1–8. https://doi.org/10.1038/s41537-021-00197-6 .

Pollack CC, Emond JA, O’Malley AJ, Byrd A, Green P, Miller KE, et al. Characterizing the Prevalence of Obesity Misinformation, Factual Content, Stigma, and Positivity on the Social Media Platform Reddit Between 2011 and 2019: Infodemiology Study. Journal of Medical Internet Research. 2022;24(12):e36729. Available from: https://www.jmir.org/2022/12/e36729 .

Pennebaker JW, Boyd RL, Jordan K, Blackburn K. The Development and Psychometric Properties of LIWC2015. 2015. https://repositories.lib.utexas.edu/handle/2152/31333 .

Chen T, Guestrin C. XGBoost: A Scalable Tree Boosting System. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2016. p. 785–94. Available from: http://arxiv.org/abs/1603.02754 . Accessed 14 Jan 2022.

Devlin J, Chang MW, Lee K, Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:181004805. 2019. http://arxiv.org/abs/1810.04805 . Accessed 11 May 2020.

Chen AT, Johnny S, Conway M. Examining stigma relating to substance use and contextual factors in social media discussions. Drug and Alcohol Dependence Reports. 2022;3:100061. https://doi.org/10.1016/j.dadr.2022.100061 .

Hatzenbuehler ML, Nolen-Hoeksema S, Dovidio J. How Does Stigma “Get Under the Skin”?: The Mediating Role of Emotion Regulation. Psychol Sci. 2009;20(10):1282–9. https://doi.org/10.1111/j.1467-9280.2009.02441.x .

Wang K, Burton CL, Pachankis JE. Depression and Substance Use: Towards the Development of an Emotion Regulation Model of Stigma Coping. Subst Use Misuse. 2018;53(5):859–66. https://doi.org/10.1080/10826084.2017.1391011 .

Baumgartner J, Zannettou S, Keegan B, Squire M, Blackburn J. The Pushshift Reddit Dataset. Proc Int AAAI Conf Web Soc Media. 2020;14:830–9.

MacLean D, Gupta S, Lembke A, Manning C, Heer J. Forum77: An Analysis of an Online Health Forum Dedicated to Addiction Recovery. In: Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing. New York, NY, USA: Association for Computing Machinery; 2015. p. 1511–26. (CSCW ’15). https://doi.org/10.1007/BF02295996 .

Benson R, Hu M, Chen AT, Zhu SH, Conway M. Examining Cannabis, Tobacco, and Vaping Discourse on Reddit: An Exploratory Approach Using Natural Language Processing. Front Public Health. 2022. Available from: https://www.frontiersin.org/article/10.3389/fpubh.2021.738513 .

Palamar JJ, Kiang MV, Halkitis PN. Development and Psychometric Evaluation of Scales that Assess Stigma Associated With Illicit Drug Users. Subst Use Misuse. 2011;46(12):1457–67. https://doi.org/10.3109/10826084.2011.596606 .

Cohen J. A Coefficient of Agreement for Nominal Scales. Educ Psychol Measur. 1960;20(1):37–46. https://doi.org/10.1177/001316446002000104 .

Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med. 2005;37(5):360–3.

PubMed   Google Scholar  

Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, et al. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv. 2019. https://doi.org/10.48550/arXiv.1907.11692 .

He H, Garcia EA. Learning from Imbalanced Data. IEEE Trans Knowl Data Eng. 2009;21(9):1263–84. https://doi.org/10.1109/TKDE.2008.239 .

Haixiang G, Yijing L, Shang J, Mingyun G, Yuanyue H, Bing G. Learning from class-imbalanced data: Review of methods and applications. Expert Syst Appl. 2017;73:220–39. https://doi.org/10.1016/j.eswa.2016.12.035 .

Bird S, Klein E, Loper E. Natural language processing with Python: analyzing text with the natural language toolkit. O’Reilly Media, Inc. 2009.

Prakash A, Tayyar Madabushi H. Incorporating Count-Based Features into Pre-Trained Models for Improved Stance Detection. In: Proceedings of the 3rd NLP4IF Workshop on NLP for Internet Freedom: Censorship, Disinformation, and Propaganda. Barcelona, Spain: International Committee on Computational Linguistics (ICCL); 2020. p. 22–32 https://aclanthology.org/2020.nlp4if-1.3 .

Mohammad S. Word Affect Intensities. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). Miyazaki, Japan: European Language Resources Association (ELRA); 2018. https://aclanthology.org/L18-1027 .

Strapparava C, Valitutti A. WordNet-Affect: An Affective Extension of WordNet. In: Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC 2004). Lisbon: European Language Resources Association (ELRA); 2004. p. 1083–6.

Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. Scikit-learn: Machine Learning in Python. J Mach Learn Res. 2011;12:2825–30.

Babanejad N, Davoudi H, An A, Papagelis M. Affective and Contextual Embedding for Sarcasm Detection. In: Proceedings of the 28th International Conference on Computational Linguistics. Barcelona, Spain: International Committee on Computational Linguistics; 2020. p. 225–43. https://doi.org/10.18653/v1/2020.coling-main.20 .

Miller GA. WordNet: a lexical database for English. Commun ACM. 1995;38(11):39–41. https://doi.org/10.1145/219717.219748 .

Brown-Johnson CG, Cataldo PhD JK, Orozco N, Lisha NE, Hickman N, Prochaska JJ. Validity and Reliability of the Internalized Stigma of Smoking Inventory: An Exploration of Shame, Isolation, and Discrimination in Smokers with Mental Health Diagnoses. Am J Addict. 2015;24(5):410–8. https://doi.org/10.1111/ajad.12215 .

Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In: Wallach H, Larochelle H, Beygelzimer A, Alché-Buc F d’, Fox E, Garnett R, editors. Advances in Neural Information Processing Systems 32. Curran Associates, Inc.; 2019. p. 8024–35. http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf .

Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, et al. HuggingFace’s Transformers: State-of-the-art Natural Language Processing. 2019. https://doi.org/10.48550/arXiv.1910.03771 .

Book   Google Scholar  

McNemar Q. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika. 1947;12(2):153–7. https://doi.org/10.1007/BF02295996 .

Benjamini Y, Hochberg Y. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. J Roy Stat Soc: Ser B (Methodol). 1995;57(1):289–300.

Dror R, Baumer G, Shlomov S, Reichart R. The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Melbourne, Australia: Association for Computational Linguistics; 2018. p. 1383–92. https://doi.org/10.18653/v1/P18-1128 .

Creswell JW, Clark VLP. Designing and Conducting Mixed Methods Research. SAGE Publications; 2017.

Doyle L, Brady AM, Byrne G. An overview of mixed methods research. J Res Nurs. 2009;14(2):175–85. https://doi.org/10.1177/1744987108093962 .

Agresti A. Categorical data analysis, Second Edition. New York: Wiley; 2002.

Sharpe D. Chi-square test is statistically significant: Now what? Pract Assess Res Eval. 2015;20(1):8.

Moreno MA, Goniu N, Moreno PS, Diekema D. Ethics of Social Media Research: Common Concerns and Practical Considerations. Cyberpsychol Behav Soc Netw. 2013;16(9):708–13. https://doi.org/10.1089/cyber.2012.0334 .

Galinsky AD, Wang CS, Whitson JA, Anicich EM, Hugenberg K, Bodenhausen GV. The Reappropriation of Stigmatizing Labels: The Reciprocal Relationship Between Power and Self-Labeling. Psychol Sci. 2013;24(10):2020–9. https://doi.org/10.1177/0956797613482943 .

Jawahar G, Sagot B, Seddah D. What Does BERT Learn about the Structure of Language? In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy: Association for Computational Linguistics; 2019. p. 3651–7. https://doi.org/10.18653/v1/P19-1356 .

Khetan V, Ramnani R, Anand M, Sengupta S, Fano AE. Causal BERT: Language Models for Causality Detection Between Events Expressed in Text. In: Arai K, editor. Intelligent Computing. Cham: Springer International Publishing; 2022. p. 965–80. (Lecture Notes in Networks and Systems). https://doi.org/10.1007/978-3-030-80119-9_64 .

Kacewicz E, Pennebaker JW, Davis M, Jeon M, Graesser AC. Pronoun Use Reflects Standings in Social Hierarchies. J Lang Soc Psychol. 2014;33(2):125–43. https://doi.org/10.1177/0261927X13502654 .

Cohn MA, Mehl MR, Pennebaker JW. Linguistic Markers of Psychological Change Surrounding September 11, 2001. Psychol Sci. 2004;15(10):687–93. https://doi.org/10.1111/j.0956-7976.2004.00741.x .

Moore MD, Ali S, Burnich-Line D, Gonzales W, Stanton MV. Stigma, Opioids, and Public Health Messaging: The Need to Disentangle Behavior From Identity. Am J Public Health. 2020;110(6):807–10. https://doi.org/10.2105/AJPH.2020.305628 .

O’Connor C, McNamara N, O’Hara L, McNicholas M, McNicholas F. How do people with eating disorders experience the stigma associated with their condition? A mixed-methods systematic review. J Ment Health. 2021;30(4):454–69. https://doi.org/10.1080/09638237.2019.1685081 .

Download references

Acknowledgements

Not applicable.

Research reported in this publication was supported by the National Institute On Drug Abuse of the National Institutes of Health under Award Number R21DA056684. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Author information

Authors and affiliations.

University of Washington School of Medicine, Biomedical Informatics and Medical Education, 850 Republican St., Box 358047, Seattle, WA, 98109, USA

David Roesler & Annie T. Chen

School of Nursing, University of Washington, Box 357260, Seattle, WA, 98195, USA

Shana Johnny

School of Computing and Information Systems, University of Melbourne, Parkville, VIC, 3052, Australia

Mike Conway

You can also search for this author in PubMed   Google Scholar

Contributions

ATC and DR conceptualized the study. All authors performed data curation, and DR and ATC performed data analysis. DR drafted the initial manuscript and iteratively revised with ATC. All authors reviewed and approved the final manuscript.

Corresponding authors

Correspondence to Mike Conway or Annie T. Chen .

Ethics declarations

Ethics approval and consent to participate.

The work performed in this study was determined as non-human subjects research by the Human Subjects Division at the University of Washington (STUDY00015737), and approved by the Office of Research Ethics and Integrity of the University of Melbourne (reference no. 2022–25512-34338–4). All methods were carried out in accordance with relevant guidelines and regulations.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1..

A detailed description of our annotation guidelines.

Additional file 2.

A complete list of keywords included in each of the handcrafted stigma lexicons.

Additional file 3.

Results of chi-square tests examining the distribution of stigma labels for each substance.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Roesler, D., Johnny, S., Conway, M. et al. A theory-informed deep learning approach to extracting and characterizing substance use-related stigma in social media. BMC Digit Health 2 , 60 (2024). https://doi.org/10.1186/s44247-024-00065-0

Download citation

Received : 19 May 2023

Accepted : 26 January 2024

Published : 16 August 2024

DOI : https://doi.org/10.1186/s44247-024-00065-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Substance use
• Social media
• Machine learning
• Information extraction

BMC Digital Health

ISSN: 2731-684X

• Submission enquiries: [email protected]
• General enquiries: [email protected]