research paper on teaching and learning

• Tutorial Review
• Open access
• Published: 24 January 2018

Teaching the science of learning

• Yana Weinstein   ORCID: orcid.org/0000-0002-5144-968X 1 ,
• Christopher R. Madan 2 , 3 &
• Megan A. Sumeracki 4  

Cognitive Research: Principles and Implications volume  3 , Article number:  2 ( 2018 ) Cite this article

244k Accesses

91 Citations

765 Altmetric

Metrics details

The science of learning has made a considerable contribution to our understanding of effective teaching and learning strategies. However, few instructors outside of the field are privy to this research. In this tutorial review, we focus on six specific cognitive strategies that have received robust support from decades of research: spaced practice, interleaving, retrieval practice, elaboration, concrete examples, and dual coding. We describe the basic research behind each strategy and relevant applied research, present examples of existing and suggested implementation, and make recommendations for further research that would broaden the reach of these strategies.

Significance

Education does not currently adhere to the medical model of evidence-based practice (Roediger, 2013 ). However, over the past few decades, our field has made significant advances in applying cognitive processes to education. From this work, specific recommendations can be made for students to maximize their learning efficiency (Dunlosky, Rawson, Marsh, Nathan, & Willingham, 2013 ; Roediger, Finn, & Weinstein, 2012 ). In particular, a review published 10 years ago identified a limited number of study techniques that have received solid evidence from multiple replications testing their effectiveness in and out of the classroom (Pashler et al., 2007 ). A recent textbook analysis (Pomerance, Greenberg, & Walsh, 2016 ) took the six key learning strategies from this report by Pashler and colleagues, and found that very few teacher-training textbooks cover any of these six principles – and none cover them all, suggesting that these strategies are not systematically making their way into the classroom. This is the case in spite of multiple recent academic (e.g., Dunlosky et al., 2013 ) and general audience (e.g., Dunlosky, 2013 ) publications about these strategies. In this tutorial review, we present the basic science behind each of these six key principles, along with more recent research on their effectiveness in live classrooms, and suggest ideas for pedagogical implementation. The target audience of this review is (a) educators who might be interested in integrating the strategies into their teaching practice, (b) science of learning researchers who are looking for open questions to help determine future research priorities, and (c) researchers in other subfields who are interested in the ways that principles from cognitive psychology have been applied to education.

While the typical teacher may not be exposed to this research during teacher training, a small cohort of teachers intensely interested in cognitive psychology has recently emerged. These teachers are mainly based in the UK, and, anecdotally (e.g., Dennis (2016), personal communication), appear to have taken an interest in the science of learning after reading Make it Stick (Brown, Roediger, & McDaniel, 2014 ; see Clark ( 2016 ) for an enthusiastic review of this book on a teacher’s blog, and “Learning Scientists” ( 2016c ) for a collection). In addition, a grassroots teacher movement has led to the creation of “researchED” – a series of conferences on evidence-based education (researchED, 2013 ). The teachers who form part of this network frequently discuss cognitive psychology techniques and their applications to education on social media (mainly Twitter; e.g., Fordham, 2016 ; Penfound, 2016 ) and on their blogs, such as Evidence Into Practice ( https://evidenceintopractice.wordpress.com/ ), My Learning Journey ( http://reflectionsofmyteaching.blogspot.com/ ), and The Effortful Educator ( https://theeffortfuleducator.com/ ). In general, the teachers who write about these issues pay careful attention to the relevant literature, often citing some of the work described in this review.

These informal writings, while allowing teachers to explore their approach to teaching practice (Luehmann, 2008 ), give us a unique window into the application of the science of learning to the classroom. By examining these blogs, we can not only observe how basic cognitive research is being applied in the classroom by teachers who are reading it, but also how it is being misapplied, and what questions teachers may be posing that have gone unaddressed in the scientific literature. Throughout this review, we illustrate each strategy with examples of how it can be implemented (see Table  1 and Figs.  1 , 2 , 3 , 4 , 5 , 6 and 7 ), as well as with relevant teacher blog posts that reflect on its application, and draw upon this work to pin-point fruitful avenues for further basic and applied research.

Spaced practice schedule for one week. This schedule is designed to represent a typical timetable of a high-school student. The schedule includes four one-hour study sessions, one longer study session on the weekend, and one rest day. Notice that each subject is studied one day after it is covered in school, to create spacing between classes and study sessions. Copyright note: this image was produced by the authors

a Blocked practice and interleaved practice with fraction problems. In the blocked version, students answer four multiplication problems consecutively. In the interleaved version, students answer a multiplication problem followed by a division problem and then an addition problem, before returning to multiplication. For an experiment with a similar setup, see Patel et al. ( 2016 ). Copyright note: this image was produced by the authors. b Illustration of interleaving and spacing. Each color represents a different homework topic. Interleaving involves alternating between topics, rather than blocking. Spacing involves distributing practice over time, rather than massing. Interleaving inherently involves spacing as other tasks naturally “fill” the spaces between interleaved sessions. Copyright note: this image was produced by the authors, adapted from Rohrer ( 2012 )

Concept map illustrating the process and resulting benefits of retrieval practice. Retrieval practice involves the process of withdrawing learned information from long-term memory into working memory, which requires effort. This produces direct benefits via the consolidation of learned information, making it easier to remember later and causing improvements in memory, transfer, and inferences. Retrieval practice also produces indirect benefits of feedback to students and teachers, which in turn can lead to more effective study and teaching practices, with a focus on information that was not accurately retrieved. Copyright note: this figure originally appeared in a blog post by the first and third authors ( http://www.learningscientists.org/blog/2016/4/1-1 )

Illustration of “how” and “why” questions (i.e., elaborative interrogation questions) students might ask while studying the physics of flight. To help figure out how physics explains flight, students might ask themselves the following questions: “How does a plane take off?”; “Why does a plane need an engine?”; “How does the upward force (lift) work?”; “Why do the wings have a curved upper surface and a flat lower surface?”; and “Why is there a downwash behind the wings?”. Copyright note: the image of the plane was downloaded from Pixabay.com and is free to use, modify, and share

Three examples of physics problems that would be categorized differently by novices and experts. The problems in ( a ) and ( c ) look similar on the surface, so novices would group them together into one category. Experts, however, will recognize that the problems in ( b ) and ( c ) both relate to the principle of energy conservation, and so will group those two problems into one category instead. Copyright note: the figure was produced by the authors, based on figures in Chi et al. ( 1981 )

Example of how to enhance learning through use of a visual example. Students might view this visual representation of neural communications with the words provided, or they could draw a similar visual representation themselves. Copyright note: this figure was produced by the authors

Example of word properties associated with visual, verbal, and motor coding for the word “SPOON”. A word can evoke multiple types of representation (“codes” in dual coding theory). Viewing a word will automatically evoke verbal representations related to its component letters and phonemes. Words representing objects (i.e., concrete nouns) will also evoke visual representations, including information about similar objects, component parts of the object, and information about where the object is typically found. In some cases, additional codes can also be evoked, such as motor-related properties of the represented object, where contextual information related to the object’s functional intention and manipulation action may also be processed automatically when reading the word. Copyright note: this figure was produced by the authors and is based on Aylwin ( 1990 ; Fig.  2 ) and Madan and Singhal ( 2012a , Fig.  3 )

Spaced practice

The benefits of spaced (or distributed) practice to learning are arguably one of the strongest contributions that cognitive psychology has made to education (Kang, 2016 ). The effect is simple: the same amount of repeated studying of the same information spaced out over time will lead to greater retention of that information in the long run, compared with repeated studying of the same information for the same amount of time in one study session. The benefits of distributed practice were first empirically demonstrated in the 19 th century. As part of his extensive investigation into his own memory, Ebbinghaus ( 1885/1913 ) found that when he spaced out repetitions across 3 days, he could almost halve the number of repetitions necessary to relearn a series of 12 syllables in one day (Chapter 8). He thus concluded that “a suitable distribution of [repetitions] over a space of time is decidedly more advantageous than the massing of them at a single time” (Section 34). For those who want to read more about Ebbinghaus’s contribution to memory research, Roediger ( 1985 ) provides an excellent summary.

Since then, hundreds of studies have examined spacing effects both in the laboratory and in the classroom (Kang, 2016 ). Spaced practice appears to be particularly useful at large retention intervals: in the meta-analysis by Cepeda, Pashler, Vul, Wixted, and Rohrer ( 2006 ), all studies with a retention interval longer than a month showed a clear benefit of distributed practice. The “new theory of disuse” (Bjork & Bjork, 1992 ) provides a helpful mechanistic explanation for the benefits of spacing to learning. This theory posits that memories have both retrieval strength and storage strength. Whereas retrieval strength is thought to measure the ease with which a memory can be recalled at a given moment, storage strength (which cannot be measured directly) represents the extent to which a memory is truly embedded in the mind. When studying is taking place, both retrieval strength and storage strength receive a boost. However, the extent to which storage strength is boosted depends upon retrieval strength, and the relationship is negative: the greater the current retrieval strength, the smaller the gains in storage strength. Thus, the information learned through “cramming” will be rapidly forgotten due to high retrieval strength and low storage strength (Bjork & Bjork, 2011 ), whereas spacing out learning increases storage strength by allowing retrieval strength to wane before restudy.

Teachers can introduce spacing to their students in two broad ways. One involves creating opportunities to revisit information throughout the semester, or even in future semesters. This does involve some up-front planning, and can be difficult to achieve, given time constraints and the need to cover a set curriculum. However, spacing can be achieved with no great costs if teachers set aside a few minutes per class to review information from previous lessons. The second method involves putting the onus to space on the students themselves. Of course, this would work best with older students – high school and above. Because spacing requires advance planning, it is crucial that the teacher helps students plan their studying. For example, teachers could suggest that students schedule study sessions on days that alternate with the days on which a particular class meets (e.g., schedule review sessions for Tuesday and Thursday when the class meets Monday and Wednesday; see Fig.  1 for a more complete weekly spaced practice schedule). It important to note that the spacing effect refers to information that is repeated multiple times, rather than the idea of studying different material in one long session versus spaced out in small study sessions over time. However, for teachers and particularly for students planning a study schedule, the subtle difference between the two situations (spacing out restudy opportunities, versus spacing out studying of different information over time) may be lost. Future research should address the effects of spacing out studying of different information over time, whether the same considerations apply in this situation as compared to spacing out restudy opportunities, and how important it is for teachers and students to understand the difference between these two types of spaced practice.

It is important to note that students may feel less confident when they space their learning (Bjork, 1999 ) than when they cram. This is because spaced learning is harder – but it is this “desirable difficulty” that helps learning in the long term (Bjork, 1994 ). Students tend to cram for exams rather than space out their learning. One explanation for this is that cramming does “work”, if the goal is only to pass an exam. In order to change students’ minds about how they schedule their studying, it might be important to emphasize the value of retaining information beyond a final exam in one course.

Ideas for how to apply spaced practice in teaching have appeared in numerous teacher blogs (e.g., Fawcett, 2013 ; Kraft, 2015 ; Picciotto, 2009 ). In England in particular, as of 2013, high-school students need to be able to remember content from up to 3 years back on cumulative exams (General Certificate of Secondary Education (GCSE) and A-level exams; see CIFE, 2012 ). A-levels in particular determine what subject students study in university and which programs they are accepted into, and thus shape the path of their academic career. A common approach for dealing with these exams has been to include a “revision” (i.e., studying or cramming) period of a few weeks leading up to the high-stakes cumulative exams. Now, teachers who follow cognitive psychology are advocating a shift of priorities to spacing learning over time across the 3 years, rather than teaching a topic once and then intensely reviewing it weeks before the exam (Cox, 2016a ; Wood, 2017 ). For example, some teachers have suggested using homework assignments as an opportunity for spaced practice by giving students homework on previous topics (Rose, 2014 ). However, questions remain, such as whether spaced practice can ever be effective enough to completely alleviate the need or utility of a cramming period (Cox, 2016b ), and how one can possibly figure out the optimal lag for spacing (Benney, 2016 ; Firth, 2016 ).

There has been considerable research on the question of optimal lag, and much of it is quite complex; two sessions neither too close together (i.e., cramming) nor too far apart are ideal for retention. In a large-scale study, Cepeda, Vul, Rohrer, Wixted, and Pashler ( 2008 ) examined the effects of the gap between study sessions and the interval between study and test across long periods, and found that the optimal gap between study sessions was contingent on the retention interval. Thus, it is not clear how teachers can apply the complex findings on lag to their own classrooms.

A useful avenue of research would be to simplify the research paradigms that are used to study optimal lag, with the goal of creating a flexible, spaced-practice framework that teachers could apply and tailor to their own teaching needs. For example, an Excel macro spreadsheet was recently produced to help teachers plan for lagged lessons (Weinstein-Jones & Weinstein, 2017 ; see Weinstein & Weinstein-Jones ( 2017 ) for a description of the algorithm used in the spreadsheet), and has been used by teachers to plan their lessons (Penfound, 2017 ). However, one teacher who found this tool helpful also wondered whether the more sophisticated plan was any better than his own method of manually selecting poorly understood material from previous classes for later review (Lovell, 2017 ). This direction is being actively explored within personalized online learning environments (Kornell & Finn, 2016 ; Lindsey, Shroyer, Pashler, & Mozer, 2014 ), but teachers in physical classrooms might need less technologically-driven solutions to teach cohorts of students.

It seems teachers would greatly appreciate a set of guidelines for how to implement spacing in the curriculum in the most effective, but also the most efficient manner. While the cognitive field has made great advances in terms of understanding the mechanisms behind spacing, what teachers need more of are concrete evidence-based tools and guidelines for direct implementation in the classroom. These could include more sophisticated and experimentally tested versions of the software described above (Weinstein-Jones & Weinstein, 2017 ), or adaptable templates of spaced curricula. Moreover, researchers need to evaluate the effectiveness of these tools in a real classroom environment, over a semester or academic year, in order to give pedagogically relevant evidence-based recommendations to teachers.

Interleaving

Another scheduling technique that has been shown to increase learning is interleaving. Interleaving occurs when different ideas or problem types are tackled in a sequence, as opposed to the more common method of attempting multiple versions of the same problem in a given study session (known as blocking). Interleaving as a principle can be applied in many different ways. One such way involves interleaving different types of problems during learning, which is particularly applicable to subjects such as math and physics (see Fig.  2 a for an example with fractions, based on a study by Patel, Liu, & Koedinger, 2016 ). For example, in a study with college students, Rohrer and Taylor ( 2007 ) found that shuffling math problems that involved calculating the volume of different shapes resulted in better test performance 1 week later than when students answered multiple problems about the same type of shape in a row. This pattern of results has also been replicated with younger students, for example 7 th grade students learning to solve graph and slope problems (Rohrer, Dedrick, & Stershic, 2015 ). The proposed explanation for the benefit of interleaving is that switching between different problem types allows students to acquire the ability to choose the right method for solving different types of problems rather than learning only the method itself, and not when to apply it.

Do the benefits of interleaving extend beyond problem solving? The answer appears to be yes. Interleaving can be helpful in other situations that require discrimination, such as inductive learning. Kornell and Bjork ( 2008 ) examined the effects of interleaving in a task that might be pertinent to a student of the history of art: the ability to match paintings to their respective painters. Students who studied different painters’ paintings interleaved at study were more successful on a later identification test than were participants who studied the paintings blocked by painter. Birnbaum, Kornell, Bjork, and Bjork ( 2013 ) proposed the discriminative-contrast hypothesis to explain that interleaving enhances learning by allowing the comparison between exemplars of different categories. They found support for this hypothesis in a set of experiments with bird categorization: participants benefited from interleaving and also from spacing, but not when the spacing interrupted side-by-side comparisons of birds from different categories.

Another type of interleaving involves the interleaving of study and test opportunities. This type of interleaving has been applied, once again, to problem solving, whereby students alternate between attempting a problem and viewing a worked example (Trafton & Reiser, 1993 ); this pattern appears to be superior to answering a string of problems in a row, at least with respect to the amount of time it takes to achieve mastery of a procedure (Corbett, Reed, Hoffmann, MacLaren, & Wagner, 2010 ). The benefits of interleaving study and test opportunities – rather than blocking study followed by attempting to answer problems or questions – might arise due to a process known as “test-potentiated learning”. That is, a study opportunity that immediately follows a retrieval attempt may be more fruitful than when that same studying was not preceded by retrieval (Arnold & McDermott, 2013 ).

For problem-based subjects, the interleaving technique is straightforward: simply mix questions on homework and quizzes with previous materials (which takes care of spacing as well); for languages, mix vocabulary themes rather than blocking by theme (Thomson & Mehring, 2016 ). But interleaving as an educational strategy ought to be presented to teachers with some caveats. Research has focused on interleaving material that is somewhat related (e.g., solving different mathematical equations, Rohrer et al., 2015 ), whereas students sometimes ask whether they should interleave material from different subjects – a practice that has not received empirical support (Hausman & Kornell, 2014 ). When advising students how to study independently, teachers should thus proceed with caution. Since it is easy for younger students to confuse this type of unhelpful interleaving with the more helpful interleaving of related information, it may be best for teachers of younger grades to create opportunities for interleaving in homework and quiz assignments rather than putting the onus on the students themselves to make use of the technique. Technology can be very helpful here, with apps such as Quizlet, Memrise, Anki, Synap, Quiz Champ, and many others (see also “Learning Scientists”, 2017 ) that not only allow instructor-created quizzes to be taken by students, but also provide built-in interleaving algorithms so that the burden does not fall on the teacher or the student to carefully plan which items are interleaved when.

An important point to consider is that in educational practice, the distinction between spacing and interleaving can be difficult to delineate. The gap between the scientific and classroom definitions of interleaving is demonstrated by teachers’ own writings about this technique. When they write about interleaving, teachers often extend the term to connote a curriculum that involves returning to topics multiple times throughout the year (e.g., Kirby, 2014 ; see “Learning Scientists” ( 2016a ) for a collection of similar blog posts by several other teachers). The “interleaving” of topics throughout the curriculum produces an effect that is more akin to what cognitive psychologists call “spacing” (see Fig.  2 b for a visual representation of the difference between interleaving and spacing). However, cognitive psychologists have not examined the effects of structuring the curriculum in this way, and open questions remain: does repeatedly circling back to previous topics throughout the semester interrupt the learning of new information? What are some effective techniques for interleaving old and new information within one class? And how does one determine the balance between old and new information?

Retrieval practice

While tests are most often used in educational settings for assessment, a lesser-known benefit of tests is that they actually improve memory of the tested information. If we think of our memories as libraries of information, then it may seem surprising that retrieval (which happens when we take a test) improves memory; however, we know from a century of research that retrieving knowledge actually strengthens it (see Karpicke, Lehman, & Aue, 2014 ). Testing was shown to strengthen memory as early as 100 years ago (Gates, 1917 ), and there has been a surge of research in the last decade on the mnemonic benefits of testing, or retrieval practice . Most of the research on the effectiveness of retrieval practice has been done with college students (see Roediger & Karpicke, 2006 ; Roediger, Putnam, & Smith, 2011 ), but retrieval-based learning has been shown to be effective at producing learning for a wide range of ages, including preschoolers (Fritz, Morris, Nolan, & Singleton, 2007 ), elementary-aged children (e.g., Karpicke, Blunt, & Smith, 2016 ; Karpicke, Blunt, Smith, & Karpicke, 2014 ; Lipko-Speed, Dunlosky, & Rawson, 2014 ; Marsh, Fazio, & Goswick, 2012 ; Ritchie, Della Sala, & McIntosh, 2013 ), middle-school students (e.g., McDaniel, Thomas, Agarwal, McDermott, & Roediger, 2013 ; McDermott, Agarwal, D’Antonio, Roediger, & McDaniel, 2014 ), and high-school students (e.g., McDermott et al., 2014 ). In addition, the effectiveness of retrieval-based learning has been extended beyond simple testing to other activities in which retrieval practice can be integrated, such as concept mapping (Blunt & Karpicke, 2014 ; Karpicke, Blunt, et al., 2014 ; Ritchie et al., 2013 ).

A debate is currently ongoing as to the effectiveness of retrieval practice for more complex materials (Karpicke & Aue, 2015 ; Roelle & Berthold, 2017 ; Van Gog & Sweller, 2015 ). Practicing retrieval has been shown to improve the application of knowledge to new situations (e.g., Butler, 2010 ; Dirkx, Kester, & Kirschner, 2014 ); McDaniel et al., 2013 ; Smith, Blunt, Whiffen, & Karpicke, 2016 ); but see Tran, Rohrer, and Pashler ( 2015 ) and Wooldridge, Bugg, McDaniel, and Liu ( 2014 ), for retrieval practice studies that showed limited or no increased transfer compared to restudy. Retrieval practice effects on higher-order learning may be more sensitive than fact learning to encoding factors, such as the way material is presented during study (Eglington & Kang, 2016 ). In addition, retrieval practice may be more beneficial for higher-order learning if it includes more scaffolding (Fiechter & Benjamin, 2017 ; but see Smith, Blunt, et al., 2016 ) and targeted practice with application questions (Son & Rivas, 2016 ).

How does retrieval practice help memory? Figure  3 illustrates both the direct and indirect benefits of retrieval practice identified by the literature. The act of retrieval itself is thought to strengthen memory (Karpicke, Blunt, et al., 2014 ; Roediger & Karpicke, 2006 ; Smith, Roediger, & Karpicke, 2013 ). For example, Smith et al. ( 2013 ) showed that if students brought information to mind without actually producing it (covert retrieval), they remembered the information just as well as if they overtly produced the retrieved information (overt retrieval). Importantly, both overt and covert retrieval practice improved memory over control groups without retrieval practice, even when feedback was not provided. The fact that bringing information to mind in the absence of feedback or restudy opportunities improves memory leads researchers to conclude that it is the act of retrieval – thinking back to bring information to mind – that improves memory of that information.

The benefit of retrieval practice depends to a certain extent on successful retrieval (see Karpicke, Lehman, et al., 2014 ). For example, in Experiment 4 of Smith et al. ( 2013 ), students successfully retrieved 72% of the information during retrieval practice. Of course, retrieving 72% of the information was compared to a restudy control group, during which students were re-exposed to 100% of the information, creating a bias in favor of the restudy condition. Yet retrieval led to superior memory later compared to the restudy control. However, if retrieval success is extremely low, then it is unlikely to improve memory (e.g., Karpicke, Blunt, et al., 2014 ), particularly in the absence of feedback. On the other hand, if retrieval-based learning situations are constructed in such a way that ensures high levels of success, the act of bringing the information to mind may be undermined, thus making it less beneficial. For example, if a student reads a sentence and then immediately covers the sentence and recites it out loud, they are likely not retrieving the information but rather just keeping the information in their working memory long enough to recite it again (see Smith, Blunt, et al., 2016 for a discussion of this point). Thus, it is important to balance success of retrieval with overall difficulty in retrieving the information (Smith & Karpicke, 2014 ; Weinstein, Nunes, & Karpicke, 2016 ). If initial retrieval success is low, then feedback can help improve the overall benefit of practicing retrieval (Kang, McDermott, & Roediger, 2007 ; Smith & Karpicke, 2014 ). Kornell, Klein, and Rawson ( 2015 ), however, found that it was the retrieval attempt and not the correct production of information that produced the retrieval practice benefit – as long as the correct answer was provided after an unsuccessful attempt, the benefit was the same as for a successful retrieval attempt in this set of studies. From a practical perspective, it would be helpful for teachers to know when retrieval attempts in the absence of success are helpful, and when they are not. There may also be additional reasons beyond retrieval benefits that would push teachers towards retrieval practice activities that produce some success amongst students; for example, teachers may hesitate to give students retrieval practice exercises that are too difficult, as this may negatively affect self-efficacy and confidence.

In addition to the fact that bringing information to mind directly improves memory for that information, engaging in retrieval practice can produce indirect benefits as well (see Roediger et al., 2011 ). For example, research by Weinstein, Gilmore, Szpunar, and McDermott ( 2014 ) demonstrated that when students expected to be tested, the increased test expectancy led to better-quality encoding of new information. Frequent testing can also serve to decrease mind-wandering – that is, thoughts that are unrelated to the material that students are supposed to be studying (Szpunar, Khan, & Schacter, 2013 ).

Practicing retrieval is a powerful way to improve meaningful learning of information, and it is relatively easy to implement in the classroom. For example, requiring students to practice retrieval can be as simple as asking students to put their class materials away and try to write out everything they know about a topic. Retrieval-based learning strategies are also flexible. Instructors can give students practice tests (e.g., short-answer or multiple-choice, see Smith & Karpicke, 2014 ), provide open-ended prompts for the students to recall information (e.g., Smith, Blunt, et al., 2016 ) or ask their students to create concept maps from memory (e.g., Blunt & Karpicke, 2014 ). In one study, Weinstein et al. ( 2016 ) looked at the effectiveness of inserting simple short-answer questions into online learning modules to see whether they improved student performance. Weinstein and colleagues also manipulated the placement of the questions. For some students, the questions were interspersed throughout the module, and for other students the questions were all presented at the end of the module. Initial success on the short-answer questions was higher when the questions were interspersed throughout the module. However, on a later test of learning from that module, the original placement of the questions in the module did not matter for performance. As with spaced practice, where the optimal gap between study sessions is contingent on the retention interval, the optimum difficulty and level of success during retrieval practice may also depend on the retention interval. Both groups of students who answered questions performed better on the delayed test compared to a control group without question opportunities during the module. Thus, the important thing is for instructors to provide opportunities for retrieval practice during learning. Based on previous research, any activity that promotes the successful retrieval of information should improve learning.

Retrieval practice has received a lot of attention in teacher blogs (see “Learning Scientists” ( 2016b ) for a collection). A common theme seems to be an emphasis on low-stakes (Young, 2016 ) and even no-stakes (Cox, 2015 ) testing, the goal of which is to increase learning rather than assess performance. In fact, one well-known charter school in the UK has an official homework policy grounded in retrieval practice: students are to test themselves on subject knowledge for 30 minutes every day in lieu of standard homework (Michaela Community School, 2014 ). The utility of homework, particularly for younger children, is often a hotly debated topic outside of academia (e.g., Shumaker, 2016 ; but see Jones ( 2016 ) for an opposing viewpoint and Cooper ( 1989 ) for the original research the blog posts were based on). Whereas some research shows clear links between homework and academic achievement (Valle et al., 2016 ), other researchers have questioned the effectiveness of homework (Dettmers, Trautwein, & Lüdtke, 2009 ). Perhaps amending homework to involve retrieval practice might make it more effective; this remains an open empirical question.

One final consideration is that of test anxiety. While retrieval practice can be very powerful at improving memory, some research shows that pressure during retrieval can undermine some of the learning benefit. For example, Hinze and Rapp ( 2014 ) manipulated pressure during quizzing to create high-pressure and low-pressure conditions. On the quizzes themselves, students performed equally well. However, those in the high-pressure condition did not perform as well on a criterion test later compared to the low-pressure group. Thus, test anxiety may reduce the learning benefit of retrieval practice. Eliminating all high-pressure tests is probably not possible, but instructors can provide a number of low-stakes retrieval opportunities for students to help increase learning. The use of low-stakes testing can serve to decrease test anxiety (Khanna, 2015 ), and has recently been shown to negate the detrimental impact of stress on learning (Smith, Floerke, & Thomas, 2016 ). This is a particularly important line of inquiry to pursue for future research, because many teachers who are not familiar with the effectiveness of retrieval practice may be put off by the implied pressure of “testing”, which evokes the much maligned high-stakes standardized tests (e.g., McHugh, 2013 ).

Elaboration

Elaboration involves connecting new information to pre-existing knowledge. Anderson ( 1983 , p.285) made the following claim about elaboration: “One of the most potent manipulations that can be performed in terms of increasing a subject’s memory for material is to have the subject elaborate on the to-be-remembered material.” Postman ( 1976 , p. 28) defined elaboration most parsimoniously as “additions to nominal input”, and Hirshman ( 2001 , p. 4369) provided an elaboration on this definition (pun intended!), defining elaboration as “A conscious, intentional process that associates to-be-remembered information with other information in memory.” However, in practice, elaboration could mean many different things. The common thread in all the definitions is that elaboration involves adding features to an existing memory.

One possible instantiation of elaboration is thinking about information on a deeper level. The levels (or “depth”) of processing framework, proposed by Craik and Lockhart ( 1972 ), predicts that information will be remembered better if it is processed more deeply in terms of meaning, rather than shallowly in terms of form. The leves of processing framework has, however, received a number of criticisms (Craik, 2002 ). One major problem with this framework is that it is difficult to measure “depth”. And if we are not able to actually measure depth, then the argument can become circular: is it that something was remembered better because it was studied more deeply, or do we conclude that it must have been studied more deeply because it is remembered better? (See Lockhart & Craik, 1990 , for further discussion of this issue).

Another mechanism by which elaboration can confer a benefit to learning is via improvement in organization (Bellezza, Cheesman, & Reddy, 1977 ; Mandler, 1979 ). By this view, elaboration involves making information more integrated and organized with existing knowledge structures. By connecting and integrating the to-be-learned information with other concepts in memory, students can increase the extent to which the ideas are organized in their minds, and this increased organization presumably facilitates the reconstruction of the past at the time of retrieval.

Elaboration is such a broad term and can include so many different techniques that it is hard to claim that elaboration will always help learning. There is, however, a specific technique under the umbrella of elaboration for which there is relatively strong evidence in terms of effectiveness (Dunlosky et al., 2013 ; Pashler et al., 2007 ). This technique is called elaborative interrogation, and involves students questioning the materials that they are studying (Pressley, McDaniel, Turnure, Wood, & Ahmad, 1987 ). More specifically, students using this technique would ask “how” and “why” questions about the concepts they are studying (see Fig.  4 for an example on the physics of flight). Then, crucially, students would try to answer these questions – either from their materials or, eventually, from memory (McDaniel & Donnelly, 1996 ). The process of figuring out the answer to the questions – with some amount of uncertainty (Overoye & Storm, 2015 ) – can help learning. When using this technique, however, it is important that students check their answers with their materials or with the teacher; when the content generated through elaborative interrogation is poor, it can actually hurt learning (Clinton, Alibali, & Nathan, 2016 ).

Students can also be encouraged to self-explain concepts to themselves while learning (Chi, De Leeuw, Chiu, & LaVancher, 1994 ). This might involve students simply saying out loud what steps they need to perform to solve an equation. Aleven and Koedinger ( 2002 ) conducted two classroom studies in which students were either prompted by a “cognitive tutor” to provide self-explanations during a problem-solving task or not, and found that the self-explanations led to improved performance. According to the authors, this approach could scale well to real classrooms. If possible and relevant, students could even perform actions alongside their self-explanations (Cohen, 1981 ; see also the enactment effect, Hainselin, Picard, Manolli, Vankerkore-Candas, & Bourdin, 2017 ). Instructors can scaffold students in these types of activities by providing self-explanation prompts throughout to-be-learned material (O’Neil et al., 2014 ). Ultimately, the greatest potential benefit of accurate self-explanation or elaboration is that the student will be able to transfer their knowledge to a new situation (Rittle-Johnson, 2006 ).

The technical term “elaborative interrogation” has not made it into the vernacular of educational bloggers (a search on https://educationechochamberuncut.wordpress.com , which consolidates over 3,000 UK-based teacher blogs, yielded zero results for that term). However, a few teachers have blogged about elaboration more generally (e.g., Hobbiss, 2016 ) and deep questioning specifically (e.g., Class Teaching, 2013 ), just without using the specific terminology. This strategy in particular may benefit from a more open dialog between researchers and teachers to facilitate the use of elaborative interrogation in the classroom and to address possible barriers to implementation. In terms of advancing the scientific understanding of elaborative interrogation in a classroom setting, it would be informative to conduct a larger-scale intervention to see whether having students elaborate during reading actually helps their understanding. It would also be useful to know whether the students really need to generate their own elaborative interrogation (“how” and “why”) questions, versus answering questions provided by others. How long should students persist to find the answers? When is the right time to have students engage in this task, given the levels of expertise required to do it well (Clinton et al., 2016 )? Without knowing the answers to these questions, it may be too early for us to instruct teachers to use this technique in their classes. Finally, elaborative interrogation takes a long time. Is this time efficiently spent? Or, would it be better to have the students try to answer a few questions, pool their information as a class, and then move to practicing retrieval of the information?

Concrete examples

Providing supporting information can improve the learning of key ideas and concepts. Specifically, using concrete examples to supplement content that is more conceptual in nature can make the ideas easier to understand and remember. Concrete examples can provide several advantages to the learning process: (a) they can concisely convey information, (b) they can provide students with more concrete information that is easier to remember, and (c) they can take advantage of the superior memorability of pictures relative to words (see “Dual Coding”).

Words that are more concrete are both recognized and recalled better than abstract words (Gorman, 1961 ; e.g., “button” and “bound,” respectively). Furthermore, it has been demonstrated that information that is more concrete and imageable enhances the learning of associations, even with abstract content (Caplan & Madan, 2016 ; Madan, Glaholt, & Caplan, 2010 ; Paivio, 1971 ). Following from this, providing concrete examples during instruction should improve retention of related abstract concepts, rather than the concrete examples alone being remembered better. Concrete examples can be useful both during instruction and during practice problems. Having students actively explain how two examples are similar and encouraging them to extract the underlying structure on their own can also help with transfer. In a laboratory study, Berry ( 1983 ) demonstrated that students performed well when given concrete practice problems, regardless of the use of verbalization (akin to elaborative interrogation), but that verbalization helped students transfer understanding from concrete to abstract problems. One particularly important area of future research is determining how students can best make the link between concrete examples and abstract ideas.

Since abstract concepts are harder to grasp than concrete information (Paivio, Walsh, & Bons, 1994 ), it follows that teachers ought to illustrate abstract ideas with concrete examples. However, care must be taken when selecting the examples. LeFevre and Dixon ( 1986 ) provided students with both concrete examples and abstract instructions and found that when these were inconsistent, students followed the concrete examples rather than the abstract instructions, potentially constraining the application of the abstract concept being taught. Lew, Fukawa-Connelly, Mejí-Ramos, and Weber ( 2016 ) used an interview approach to examine why students may have difficulty understanding a lecture. Responses indicated that some issues were related to understanding the overarching topic rather than the component parts, and to the use of informal colloquialisms that did not clearly follow from the material being taught. Both of these issues could have potentially been addressed through the inclusion of a greater number of relevant concrete examples.

One concern with using concrete examples is that students might only remember the examples – especially if they are particularly memorable, such as fun or gimmicky examples – and will not be able to transfer their understanding from one example to another, or more broadly to the abstract concept. However, there does not seem to be any evidence that fun relevant examples actually hurt learning by harming memory for important information. Instead, fun examples and jokes tend to be more memorable, but this boost in memory for the joke does not seem to come at a cost to memory for the underlying concept (Baldassari & Kelley, 2012 ). However, two important caveats need to be highlighted. First, to the extent that the more memorable content is not relevant to the concepts of interest, learning of the target information can be compromised (Harp & Mayer, 1998 ). Thus, care must be taken to ensure that all examples and gimmicks are, in fact, related to the core concepts that the students need to acquire, and do not contain irrelevant perceptual features (Kaminski & Sloutsky, 2013 ).

The second issue is that novices often notice and remember the surface details of an example rather than the underlying structure. Experts, on the other hand, can extract the underlying structure from examples that have divergent surface features (Chi, Feltovich, & Glaser, 1981 ; see Fig.  5 for an example from physics). Gick and Holyoak ( 1983 ) tried to get students to apply a rule from one problem to another problem that appeared different on the surface, but was structurally similar. They found that providing multiple examples helped with this transfer process compared to only using one example – especially when the examples provided had different surface details. More work is also needed to determine how many examples are sufficient for generalization to occur (and this, of course, will vary with contextual factors and individual differences). Further research on the continuum between concrete/specific examples and more abstract concepts would also be informative. That is, if an example is not concrete enough, it may be too difficult to understand. On the other hand, if the example is too concrete, that could be detrimental to generalization to the more abstract concept (although a diverse set of very concrete examples may be able to help with this). In fact, in a controversial article, Kaminski, Sloutsky, and Heckler ( 2008 ) claimed that abstract examples were more effective than concrete examples. Later rebuttals of this paper contested whether the abstract versus concrete distinction was clearly defined in the original study (see Reed, 2008 , for a collection of letters on the subject). This ideal point along the concrete-abstract continuum might also interact with development.

Finding teacher blog posts on concrete examples proved to be more difficult than for the other strategies in this review. One optimistic possibility is that teachers frequently use concrete examples in their teaching, and thus do not think of this as a specific contribution from cognitive psychology; the one blog post we were able to find that discussed concrete examples suggests that this might be the case (Boulton, 2016 ). The idea of “linking abstract concepts with concrete examples” is also covered in 25% of teacher-training textbooks used in the US, according to the report by Pomerance et al. ( 2016 ); this is the second most frequently covered of the six strategies, after “posing probing questions” (i.e., elaborative interrogation). A useful direction for future research would be to establish how teachers are using concrete examples in their practice, and whether we can make any suggestions for improvement based on research into the science of learning. For example, if two examples are better than one (Bauernschmidt, 2017 ), are additional examples also needed, or are there diminishing returns from providing more examples? And, how can teachers best ensure that concrete examples are consistent with prior knowledge (Reed, 2008 )?

Dual coding

Both the memory literature and folk psychology support the notion of visual examples being beneficial—the adage of “a picture is worth a thousand words” (traced back to an advertising slogan from the 1920s; Meider, 1990 ). Indeed, it is well-understood that more information can be conveyed through a simple illustration than through several paragraphs of text (e.g., Barker & Manji, 1989 ; Mayer & Gallini, 1990 ). Illustrations can be particularly helpful when the described concept involves several parts or steps and is intended for individuals with low prior knowledge (Eitel & Scheiter, 2015 ; Mayer & Gallini, 1990 ). Figure  6 provides a concrete example of this, illustrating how information can flow through neurons and synapses.

In addition to being able to convey information more succinctly, pictures are also more memorable than words (Paivio & Csapo, 1969 , 1973 ). In the memory literature, this is referred to as the picture superiority effect , and dual coding theory was developed in part to explain this effect. Dual coding follows from the notion of text being accompanied by complementary visual information to enhance learning. Paivio ( 1971 , 1986 ) proposed dual coding theory as a mechanistic account for the integration of multiple information “codes” to process information. In this theory, a code corresponds to a modal or otherwise distinct representation of a concept—e.g., “mental images for ‘book’ have visual, tactual, and other perceptual qualities similar to those evoked by the referent objects on which the images are based” (Clark & Paivio, 1991 , p. 152). Aylwin ( 1990 ) provides a clear example of how the word “dog” can evoke verbal, visual, and enactive representations (see Fig.  7 for a similar example for the word “SPOON”, based on Aylwin, 1990 (Fig.  2 ) and Madan & Singhal, 2012a (Fig.  3 )). Codes can also correspond to emotional properties (Clark & Paivio, 1991 ; Paivio, 2013 ). Clark and Paivio ( 1991 ) provide a thorough review of dual coding theory and its relation to education, while Paivio ( 2007 ) provides a comprehensive treatise on dual coding theory. Broadly, dual coding theory suggests that providing multiple representations of the same information enhances learning and memory, and that information that more readily evokes additional representations (through automatic imagery processes) receives a similar benefit.

Paivio and Csapo ( 1973 ) suggest that verbal and imaginal codes have independent and additive effects on memory recall. Using visuals to improve learning and memory has been particularly applied to vocabulary learning (Danan, 1992 ; Sadoski, 2005 ), but has also shown success in other domains such as in health care (Hartland, Biddle, & Fallacaro, 2008 ). To take advantage of dual coding, verbal information should be accompanied by a visual representation when possible. However, while the studies discussed all indicate that the use of multiple representations of information is favorable, it is important to acknowledge that each representation also increases cognitive load and can lead to over-saturation (Mayer & Moreno, 2003 ).

Given that pictures are generally remembered better than words, it is important to ensure that the pictures students are provided with are helpful and relevant to the content they are expected to learn. McNeill, Uttal, Jarvin, and Sternberg ( 2009 ) found that providing visual examples decreased conceptual errors. However, McNeill et al. also found that when students were given visually rich examples, they performed more poorly than students who were not given any visual example, suggesting that the visual details can at times become a distraction and hinder performance. Thus, it is important to consider that images used in teaching are clear and not ambiguous in their meaning (Schwartz, 2007 ).

Further broadening the scope of dual coding theory, Engelkamp and Zimmer ( 1984 ) suggest that motor movements, such as “turning the handle,” can provide an additional motor code that can improve memory, linking studies of motor actions (enactment) with dual coding theory (Clark & Paivio, 1991 ; Engelkamp & Cohen, 1991 ; Madan & Singhal, 2012c ). Indeed, enactment effects appear to primarily occur during learning, rather than during retrieval (Peterson & Mulligan, 2010 ). Along similar lines, Wammes, Meade, and Fernandes ( 2016 ) demonstrated that generating drawings can provide memory benefits beyond what could otherwise be explained by visual imagery, picture superiority, and other memory enhancing effects. Providing convergent evidence, even when overt motor actions are not critical in themselves, words representing functional objects have been shown to enhance later memory (Madan & Singhal, 2012b ; Montefinese, Ambrosini, Fairfield, & Mammarella, 2013 ). This indicates that motoric processes can improve memory similarly to visual imagery, similar to memory differences for concrete vs. abstract words. Further research suggests that automatic motor simulation for functional objects is likely responsible for this memory benefit (Madan, Chen, & Singhal, 2016 ).

When teachers combine visuals and words in their educational practice, however, they may not always be taking advantage of dual coding – at least, not in the optimal manner. For example, a recent discussion on Twitter centered around one teacher’s decision to have 7 th Grade students replace certain words in their science laboratory report with a picture of that word (e.g., the instructions read “using a syringe …” and a picture of a syringe replaced the word; Turner, 2016a ). Other teachers argued that this was not dual coding (Beaven, 2016 ; Williams, 2016 ), because there were no longer two different representations of the information. The first teacher maintained that dual coding was preserved, because this laboratory report with pictures was to be used alongside the original, fully verbal report (Turner, 2016b ). This particular implementation – having students replace individual words with pictures – has not been examined in the cognitive literature, presumably because no benefit would be expected. In any case, we need to be clearer about implementations for dual coding, and more research is needed to clarify how teachers can make use of the benefits conferred by multiple representations and picture superiority.

Critically, dual coding theory is distinct from the notion of “learning styles,” which describe the idea that individuals benefit from instruction that matches their modality preference. While this idea is pervasive and individuals often subjectively feel that they have a preference, evidence indicates that the learning styles theory is not supported by empirical findings (e.g., Kavale, Hirshoren, & Forness, 1998 ; Pashler, McDaniel, Rohrer, & Bjork, 2008 ; Rohrer & Pashler, 2012 ). That is, there is no evidence that instructing students in their preferred learning style leads to an overall improvement in learning (the “meshing” hypothesis). Moreover, learning styles have come to be described as a myth or urban legend within psychology (Coffield, Moseley, Hall, & Ecclestone, 2004 ; Hattie & Yates, 2014 ; Kirschner & van Merriënboer, 2013 ; Kirschner, 2017 ); skepticism about learning styles is a common stance amongst evidence-informed teachers (e.g., Saunders, 2016 ). Providing evidence against the notion of learning styles, Kraemer, Rosenberg, and Thompson-Schill ( 2009 ) found that individuals who scored as “verbalizers” and “visualizers” did not perform any better on experimental trials matching their preference. Instead, it has recently been shown that learning through one’s preferred learning style is associated with elevated subjective judgements of learning, but not objective performance (Knoll, Otani, Skeel, & Van Horn, 2017 ). In contrast to learning styles, dual coding is based on providing additional, complementary forms of information to enhance learning, rather than tailoring instruction to individuals’ preferences.

Genuine educational environments present many opportunities for combining the strategies outlined above. Spacing can be particularly potent for learning if it is combined with retrieval practice. The additive benefits of retrieval practice and spacing can be gained by engaging in retrieval practice multiple times (also known as distributed practice; see Cepeda et al., 2006 ). Interleaving naturally entails spacing if students interleave old and new material. Concrete examples can be both verbal and visual, making use of dual coding. In addition, the strategies of elaboration, concrete examples, and dual coding all work best when used as part of retrieval practice. For example, in the concept-mapping studies mentioned above (Blunt & Karpicke, 2014 ; Karpicke, Blunt, et al., 2014 ), creating concept maps while looking at course materials (e.g., a textbook) was not as effective for later memory as creating concept maps from memory. When practicing elaborative interrogation, students can start off answering the “how” and “why” questions they pose for themselves using class materials, and work their way up to answering them from memory. And when interleaving different problem types, students should be practicing answering them rather than just looking over worked examples.

But while these ideas for strategy combinations have empirical bases, it has not yet been established whether the benefits of the strategies to learning are additive, super-additive, or, in some cases, incompatible. Thus, future research needs to (a) better formalize the definition of each strategy (particularly critical for elaboration and dual coding), (b) identify best practices for implementation in the classroom, (c) delineate the boundary conditions of each strategy, and (d) strategically investigate interactions between the six strategies we outlined in this manuscript.

Aleven, V. A., & Koedinger, K. R. (2002). An effective metacognitive strategy: learning by doing and explaining with a computer-based cognitive tutor. Cognitive Science, 26 , 147–179.

Article   Google Scholar  

Anderson, J. R. (1983). A spreading activation theory of memory. Journal of Verbal Learning and Verbal Behavior, 22 , 261–295.

Arnold, K. M., & McDermott, K. B. (2013). Test-potentiated learning: distinguishing between direct and indirect effects of tests. Journal of Experimental Psychology: Learning, Memory, and Cognition, 39 , 940–945.

PubMed   Google Scholar  

Aylwin, S. (1990). Imagery and affect: big questions, little answers. In P. J. Thompson, D. E. Marks, & J. T. E. Richardson (Eds.), Imagery: Current developments . New York: International Library of Psychology.

Google Scholar  

Baldassari, M. J., & Kelley, M. (2012). Make’em laugh? The mnemonic effect of humor in a speech. Psi Chi Journal of Psychological Research, 17 , 2–9.

Barker, P. G., & Manji, K. A. (1989). Pictorial dialogue methods. International Journal of Man-Machine Studies, 31 , 323–347.

Bauernschmidt, A. (2017). GUEST POST: two examples are better than one. [Blog post]. The Learning Scientists Blog . Retrieved from http://www.learningscientists.org/blog/2017/5/30-1 . Accessed 25 Dec 2017.

Beaven, T. (2016). @doctorwhy @FurtherEdagogy @doc_kristy Right, I thought the whole point of dual coding was to use TWO codes: pics + words of the SAME info? [Tweet]. Retrieved from https://twitter.com/TitaBeaven/status/807504041341308929 . Accessed 25 Dec 2017.

Bellezza, F. S., Cheesman, F. L., & Reddy, B. G. (1977). Organization and semantic elaboration in free recall. Journal of Experimental Psychology: Human Learning and Memory, 3 , 539–550.

Benney, D. (2016). (Trying to apply) spacing in a content heavy subject [Blog post]. Retrieved from https://mrbenney.wordpress.com/2016/10/16/trying-to-apply-spacing-in-science/ . Accessed 25 Dec 2017.

Berry, D. C. (1983). Metacognitive experience and transfer of logical reasoning. Quarterly Journal of Experimental Psychology, 35A , 39–49.

Birnbaum, M. S., Kornell, N., Bjork, E. L., & Bjork, R. A. (2013). Why interleaving enhances inductive learning: the roles of discrimination and retrieval. Memory & Cognition, 41 , 392–402.

Bjork, R. A. (1999). Assessing our own competence: heuristics and illusions. In D. Gopher & A. Koriat (Eds.), Attention and peformance XVII. Cognitive regulation of performance: Interaction of theory and application (pp. 435–459). Cambridge, MA: MIT Press.

Bjork, R. A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe & A. Shimamura (Eds.), Metacognition: Knowing about knowing (pp. 185–205). Cambridge, MA: MIT Press.

Bjork, R. A., & Bjork, E. L. (1992). A new theory of disuse and an old theory of stimulus fluctuation. From learning processes to cognitive processes: Essays in honor of William K. Estes, 2 , 35–67.

Bjork, E. L., & Bjork, R. A. (2011). Making things hard on yourself, but in a good way: creating desirable difficulties to enhance learning. Psychology and the real world: Essays illustrating fundamental contributions to society , 56–64.

Blunt, J. R., & Karpicke, J. D. (2014). Learning with retrieval-based concept mapping. Journal of Educational Psychology, 106 , 849–858.

Boulton, K. (2016). What does cognitive overload look like in the humanities? [Blog post]. Retrieved from https://educationechochamberuncut.wordpress.com/2016/03/05/what-does-cognitive-overload-look-like-in-the-humanities-kris-boulton-2/ . Accessed 25 Dec 2017.

Brown, P. C., Roediger, H. L., & McDaniel, M. A. (2014). Make it stick . Cambridge, MA: Harvard University Press.

Book   Google Scholar  

Butler, A. C. (2010). Repeated testing produces superior transfer of learning relative to repeated studying. Journal of Experimental Psychology: Learning, Memory, and Cognition, 36 , 1118–1133.

Caplan, J. B., & Madan, C. R. (2016). Word-imageability enhances association-memory by recruiting hippocampal activity. Journal of Cognitive Neuroscience, 28 , 1522–1538.

Article   PubMed   Google Scholar  

Cepeda, N. J., Pashler, H., Vul, E., Wixted, J. T., & Rohrer, D. (2006). Distributed practice in verbal recall tasks: a review and quantitative synthesis. Psychological Bulletin, 132 , 354–380.

Cepeda, N. J., Vul, E., Rohrer, D., Wixted, J. T., & Pashler, H. (2008). Spacing effects in learning a temporal ridgeline of optimal retention. Psychological Science, 19 , 1095–1102.

Chi, M. T., De Leeuw, N., Chiu, M. H., & LaVancher, C. (1994). Eliciting self-explanations improves understanding. Cognitive Science, 18 , 439–477.

Chi, M. T., Feltovich, P. J., & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5 , 121–152.

CIFE. (2012). No January A level and other changes. Retrieved from http://www.cife.org.uk/cife-general-news/no-january-a-level-and-other-changes/ . Accessed 25 Dec 2017.

Clark, D. (2016). One book on learning that every teacher, lecturer & trainer should read (7 reasons) [Blog post]. Retrieved from http://donaldclarkplanb.blogspot.com/2016/03/one-book-on-learning-that-every-teacher.html . Accessed 25 Dec 2017.

Clark, J. M., & Paivio, A. (1991). Dual coding theory and education. Educational Psychology Review, 3 , 149–210.

Class Teaching. (2013). Deep questioning [Blog post]. Retrieved from https://classteaching.wordpress.com/2013/07/12/deep-questioning/ . Accessed 25 Dec 2017.

Clinton, V., Alibali, M. W., & Nathan, M. J. (2016). Learning about posterior probability: do diagrams and elaborative interrogation help? The Journal of Experimental Education, 84 , 579–599.

Coffield, F., Moseley, D., Hall, E., & Ecclestone, K. (2004). Learning styles and pedagogy in post-16 learning: a systematic and critical review . London: Learning & Skills Research Centre.

Cohen, R. L. (1981). On the generality of some memory laws. Scandinavian Journal of Psychology, 22 , 267–281.

Cooper, H. (1989). Synthesis of research on homework. Educational Leadership, 47 , 85–91.

Corbett, A. T., Reed, S. K., Hoffmann, R., MacLaren, B., & Wagner, A. (2010). Interleaving worked examples and cognitive tutor support for algebraic modeling of problem situations. In Proceedings of the Thirty-Second Annual Meeting of the Cognitive Science Society (pp. 2882–2887).

Cox, D. (2015). No stakes testing – not telling students their results [Blog post]. Retrieved from https://missdcoxblog.wordpress.com/2015/06/06/no-stakes-testing-not-telling-students-their-results/ . Accessed 25 Dec 2017.

Cox, D. (2016a). Ditch revision. Teach it well [Blog post]. Retrieved from https://missdcoxblog.wordpress.com/2016/01/09/ditch-revision-teach-it-well/ . Accessed 25 Dec 2017.

Cox, D. (2016b). ‘They need to remember this in three years time’: spacing & interleaving for the new GCSEs [Blog post]. Retrieved from https://missdcoxblog.wordpress.com/2016/03/25/they-need-to-remember-this-in-three-years-time-spacing-interleaving-for-the-new-gcses/ . Accessed 25 Dec 2017.

Craik, F. I. (2002). Levels of processing: past, present… future? Memory, 10 , 305–318.

Craik, F. I., & Lockhart, R. S. (1972). Levels of processing: a framework for memory research. Journal of Verbal Learning and Verbal Behavior, 11 , 671–684.

Danan, M. (1992). Reversed subtitling and dual coding theory: new directions for foreign language instruction. Language Learning, 42 , 497–527.

Dettmers, S., Trautwein, U., & Lüdtke, O. (2009). The relationship between homework time and achievement is not universal: evidence from multilevel analyses in 40 countries. School Effectiveness and School Improvement, 20 , 375–405.

Dirkx, K. J., Kester, L., & Kirschner, P. A. (2014). The testing effect for learning principles and procedures from texts. The Journal of Educational Research, 107 , 357–364.

Dunlosky, J. (2013). Strengthening the student toolbox: study strategies to boost learning. American Educator, 37 (3), 12–21.

Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., & Willingham, D. T. (2013). Improving students’ learning with effective learning techniques: promising directions from cognitive and educational psychology. Psychological Science in the Public Interest, 14 , 4–58.

Ebbinghaus, H. (1913). Memory (HA Ruger & CE Bussenius, Trans.). New York: Columbia University, Teachers College. (Original work published 1885) . Retrieved from http://psychclassics.yorku.ca/Ebbinghaus/memory8.htm . Accessed 25 Dec 2017.

Eglington, L. G., & Kang, S. H. (2016). Retrieval practice benefits deductive inference. Educational Psychology Review , 1–14.

Eitel, A., & Scheiter, K. (2015). Picture or text first? Explaining sequential effects when learning with pictures and text. Educational Psychology Review, 27 , 153–180.

Engelkamp, J., & Cohen, R. L. (1991). Current issues in memory of action events. Psychological Research, 53 , 175–182.

Engelkamp, J., & Zimmer, H. D. (1984). Motor programme information as a separable memory unit. Psychological Research, 46 , 283–299.

Fawcett, D. (2013). Can I be that little better at……using cognitive science/psychology/neurology to plan learning? [Blog post]. Retrieved from http://reflectionsofmyteaching.blogspot.com/2013/09/can-i-be-that-little-better-atusing.html . Accessed 25 Dec 2017.

Fiechter, J. L., & Benjamin, A. S. (2017). Diminishing-cues retrieval practice: a memory-enhancing technique that works when regular testing doesn’t. Psychonomic Bulletin & Review , 1–9.

Firth, J. (2016). Spacing in teaching practice [Blog post]. Retrieved from http://www.learningscientists.org/blog/2016/4/12-1 . Accessed 25 Dec 2017.

Fordham, M. [mfordhamhistory]. (2016). Is there a meaningful distinction in psychology between ‘thinking’ & ‘critical thinking’? [Tweet]. Retrieved from https://twitter.com/mfordhamhistory/status/809525713623781377 . Accessed 25 Dec 2017.

Fritz, C. O., Morris, P. E., Nolan, D., & Singleton, J. (2007). Expanding retrieval practice: an effective aid to preschool children’s learning. The Quarterly Journal of Experimental Psychology, 60 , 991–1004.

Gates, A. I. (1917). Recitation as a factory in memorizing. Archives of Psychology, 6.

Gick, M. L., & Holyoak, K. J. (1983). Schema induction and analogical transfer. Cognitive Psychology, 15 , 1–38.

Gorman, A. M. (1961). Recognition memory for nouns as a function of abstractedness and frequency. Journal of Experimental Psychology, 61 , 23–39.

Hainselin, M., Picard, L., Manolli, P., Vankerkore-Candas, S., & Bourdin, B. (2017). Hey teacher, don’t leave them kids alone: action is better for memory than reading. Frontiers in Psychology , 8 .

Harp, S. F., & Mayer, R. E. (1998). How seductive details do their damage. Journal of Educational Psychology, 90 , 414–434.

Hartland, W., Biddle, C., & Fallacaro, M. (2008). Audiovisual facilitation of clinical knowledge: A paradigm for dispersed student education based on Paivio’s dual coding theory. AANA Journal, 76 , 194–198.

Hattie, J., & Yates, G. (2014). Visible learning and the science of how we learn . New York: Routledge.

Hausman, H., & Kornell, N. (2014). Mixing topics while studying does not enhance learning. Journal of Applied Research in Memory and Cognition, 3 , 153–160.

Hinze, S. R., & Rapp, D. N. (2014). Retrieval (sometimes) enhances learning: performance pressure reduces the benefits of retrieval practice. Applied Cognitive Psychology, 28 , 597–606.

Hirshman, E. (2001). Elaboration in memory. In N. J. Smelser & P. B. Baltes (Eds.), International encyclopedia of the social & behavioral sciences (pp. 4369–4374). Oxford: Pergamon.

Chapter   Google Scholar  

Hobbiss, M. (2016). Make it meaningful! Elaboration [Blog post]. Retrieved from https://hobbolog.wordpress.com/2016/06/09/make-it-meaningful-elaboration/ . Accessed 25 Dec 2017.

Jones, F. (2016). Homework – is it really that useless? [Blog post]. Retrieved from http://www.learningscientists.org/blog/2016/4/5-1 . Accessed 25 Dec 2017.

Kaminski, J. A., & Sloutsky, V. M. (2013). Extraneous perceptual information interferes with children’s acquisition of mathematical knowledge. Journal of Educational Psychology, 105 (2), 351–363.

Kaminski, J. A., Sloutsky, V. M., & Heckler, A. F. (2008). The advantage of abstract examples in learning math. Science, 320 , 454–455.

Kang, S. H. (2016). Spaced repetition promotes efficient and effective learning policy implications for instruction. Policy Insights from the Behavioral and Brain Sciences, 3 , 12–19.

Kang, S. H. K., McDermott, K. B., & Roediger, H. L. (2007). Test format and corrective feedback modify the effects of testing on long-term retention. European Journal of Cognitive Psychology, 19 , 528–558.

Karpicke, J. D., & Aue, W. R. (2015). The testing effect is alive and well with complex materials. Educational Psychology Review, 27 , 317–326.

Karpicke, J. D., Blunt, J. R., Smith, M. A., & Karpicke, S. S. (2014). Retrieval-based learning: The need for guided retrieval in elementary school children. Journal of Applied Research in Memory and Cognition, 3 , 198–206.

Karpicke, J. D., Lehman, M., & Aue, W. R. (2014). Retrieval-based learning: an episodic context account. In B. H. Ross (Ed.), Psychology of Learning and Motivation (Vol. 61, pp. 237–284). San Diego, CA: Elsevier Academic Press.

Karpicke, J. D., Blunt, J. R., & Smith, M. A. (2016). Retrieval-based learning: positive effects of retrieval practice in elementary school children. Frontiers in Psychology, 7 .

Kavale, K. A., Hirshoren, A., & Forness, S. R. (1998). Meta-analytic validation of the Dunn and Dunn model of learning-style preferences: a critique of what was Dunn. Learning Disabilities Research & Practice, 13 , 75–80.

Khanna, M. M. (2015). Ungraded pop quizzes: test-enhanced learning without all the anxiety. Teaching of Psychology, 42 , 174–178.

Kirby, J. (2014). One scientific insight for curriculum design [Blog post]. Retrieved from https://pragmaticreform.wordpress.com/2014/05/05/scientificcurriculumdesign/ . Accessed 25 Dec 2017.

Kirschner, P. A. (2017). Stop propagating the learning styles myth. Computers & Education, 106 , 166–171.

Kirschner, P. A., & van Merriënboer, J. J. G. (2013). Do learners really know best? Urban legends in education. Educational Psychologist, 48 , 169–183.

Knoll, A. R., Otani, H., Skeel, R. L., & Van Horn, K. R. (2017). Learning style, judgments of learning, and learning of verbal and visual information. British Journal of Psychology, 108 , 544-563.

Kornell, N., & Bjork, R. A. (2008). Learning concepts and categories is spacing the “enemy of induction”? Psychological Science, 19 , 585–592.

Kornell, N., & Finn, B. (2016). Self-regulated learning: an overview of theory and data. In J. Dunlosky & S. Tauber (Eds.), The Oxford Handbook of Metamemory (pp. 325–340). New York: Oxford University Press.

Kornell, N., Klein, P. J., & Rawson, K. A. (2015). Retrieval attempts enhance learning, but retrieval success (versus failure) does not matter. Journal of Experimental Psychology: Learning, Memory, and Cognition, 41 , 283–294.

Kraemer, D. J. M., Rosenberg, L. M., & Thompson-Schill, S. L. (2009). The neural correlates of visual and verbal cognitive styles. Journal of Neuroscience, 29 , 3792–3798.

Article   PubMed   PubMed Central   Google Scholar  

Kraft, N. (2015). Spaced practice and repercussions for teaching. Retrieved from http://nathankraft.blogspot.com/2015/08/spaced-practice-and-repercussions-for.html . Accessed 25 Dec 2017.

Learning Scientists. (2016a). Weekly Digest #3: How teachers implement interleaving in their curriculum [Blog post]. Retrieved from http://www.learningscientists.org/blog/2016/3/28/weekly-digest-3 . Accessed 25 Dec 2017.

Learning Scientists. (2016b). Weekly Digest #13: how teachers implement retrieval in their classrooms [Blog post]. Retrieved from http://www.learningscientists.org/blog/2016/6/5/weekly-digest-13 . Accessed 25 Dec 2017.

Learning Scientists. (2016c). Weekly Digest #40: teachers’ implementation of principles from “Make It Stick” [Blog post]. Retrieved from http://www.learningscientists.org/blog/2016/12/18-1 . Accessed 25 Dec 2017.

Learning Scientists. (2017). Weekly Digest #54: is there an app for that? Studying 2.0 [Blog post]. Retrieved from http://www.learningscientists.org/blog/2017/4/9/weekly-digest-54 . Accessed 25 Dec 2017.

LeFevre, J.-A., & Dixon, P. (1986). Do written instructions need examples? Cognition and Instruction, 3 , 1–30.

Lew, K., Fukawa-Connelly, T., Mejí-Ramos, J. P., & Weber, K. (2016). Lectures in advanced mathematics: Why students might not understand what the mathematics professor is trying to convey. Journal of Research in Mathematics Education, 47 , 162–198.

Lindsey, R. V., Shroyer, J. D., Pashler, H., & Mozer, M. C. (2014). Improving students’ long-term knowledge retention through personalized review. Psychological Science, 25 , 639–647.

Lipko-Speed, A., Dunlosky, J., & Rawson, K. A. (2014). Does testing with feedback help grade-school children learn key concepts in science? Journal of Applied Research in Memory and Cognition, 3 , 171–176.

Lockhart, R. S., & Craik, F. I. (1990). Levels of processing: a retrospective commentary on a framework for memory research. Canadian Journal of Psychology, 44 , 87–112.

Lovell, O. (2017). How do we know what to put on the quiz? [Blog Post]. Retrieved from http://www.ollielovell.com/olliesclassroom/know-put-quiz/ . Accessed 25 Dec 2017.

Luehmann, A. L. (2008). Using blogging in support of teacher professional identity development: a case study. The Journal of the Learning Sciences, 17 , 287–337.

Madan, C. R., Glaholt, M. G., & Caplan, J. B. (2010). The influence of item properties on association-memory. Journal of Memory and Language, 63 , 46–63.

Madan, C. R., & Singhal, A. (2012a). Motor imagery and higher-level cognition: four hurdles before research can sprint forward. Cognitive Processing, 13 , 211–229.

Madan, C. R., & Singhal, A. (2012b). Encoding the world around us: motor-related processing influences verbal memory. Consciousness and Cognition, 21 , 1563–1570.

Madan, C. R., & Singhal, A. (2012c). Using actions to enhance memory: effects of enactment, gestures, and exercise on human memory. Frontiers in Psychology, 3 .

Madan, C. R., Chen, Y. Y., & Singhal, A. (2016). ERPs differentially reflect automatic and deliberate processing of the functional manipulability of objects. Frontiers in Human Neuroscience, 10 .

Mandler, G. (1979). Organization and repetition: organizational principles with special reference to rote learning. In L. G. Nilsson (Ed.), Perspectives on Memory Research (pp. 293–327). New York: Academic Press.

Marsh, E. J., Fazio, L. K., & Goswick, A. E. (2012). Memorial consequences of testing school-aged children. Memory, 20 , 899–906.

Mayer, R. E., & Gallini, J. K. (1990). When is an illustration worth ten thousand words? Journal of Educational Psychology, 82 , 715–726.

Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist, 38 , 43–52.

McDaniel, M. A., & Donnelly, C. M. (1996). Learning with analogy and elaborative interrogation. Journal of Educational Psychology, 88 , 508–519.

McDaniel, M. A., Thomas, R. C., Agarwal, P. K., McDermott, K. B., & Roediger, H. L. (2013). Quizzing in middle-school science: successful transfer performance on classroom exams. Applied Cognitive Psychology, 27 , 360–372.

McDermott, K. B., Agarwal, P. K., D’Antonio, L., Roediger, H. L., & McDaniel, M. A. (2014). Both multiple-choice and short-answer quizzes enhance later exam performance in middle and high school classes. Journal of Experimental Psychology: Applied, 20 , 3–21.

McHugh, A. (2013). High-stakes tests: bad for students, teachers, and education in general [Blog post]. Retrieved from https://teacherbiz.wordpress.com/2013/07/01/high-stakes-tests-bad-for-students-teachers-and-education-in-general/ . Accessed 25 Dec 2017.

McNeill, N. M., Uttal, D. H., Jarvin, L., & Sternberg, R. J. (2009). Should you show me the money? Concrete objects both hurt and help performance on mathematics problems. Learning and Instruction, 19 , 171–184.

Meider, W. (1990). “A picture is worth a thousand words”: from advertising slogan to American proverb. Southern Folklore, 47 , 207–225.

Michaela Community School. (2014). Homework. Retrieved from http://mcsbrent.co.uk/homework-2/ . Accessed 25 Dec 2017.

Montefinese, M., Ambrosini, E., Fairfield, B., & Mammarella, N. (2013). The “subjective” pupil old/new effect: is the truth plain to see? International Journal of Psychophysiology, 89 , 48–56.

O’Neil, H. F., Chung, G. K., Kerr, D., Vendlinski, T. P., Buschang, R. E., & Mayer, R. E. (2014). Adding self-explanation prompts to an educational computer game. Computers In Human Behavior, 30 , 23–28.

Overoye, A. L., & Storm, B. C. (2015). Harnessing the power of uncertainty to enhance learning. Translational Issues in Psychological Science, 1 , 140–148.

Paivio, A. (1971). Imagery and verbal processes . New York: Holt, Rinehart and Winston.

Paivio, A. (1986). Mental representations: a dual coding approach . New York: Oxford University Press.

Paivio, A. (2007). Mind and its evolution: a dual coding theoretical approach . Mahwah: Erlbaum.

Paivio, A. (2013). Dual coding theory, word abstractness, and emotion: a critical review of Kousta et al. (2011). Journal of Experimental Psychology: General, 142 , 282–287.

Paivio, A., & Csapo, K. (1969). Concrete image and verbal memory codes. Journal of Experimental Psychology, 80 , 279–285.

Paivio, A., & Csapo, K. (1973). Picture superiority in free recall: imagery or dual coding? Cognitive Psychology, 5 , 176–206.

Paivio, A., Walsh, M., & Bons, T. (1994). Concreteness effects on memory: when and why? Journal of Experimental Psychology: Learning, Memory, and Cognition, 20 , 1196–1204.

Pashler, H., McDaniel, M., Rohrer, D., & Bjork, R. (2008). Learning styles: concepts and evidence. Psychological Science in the Public Interest, 9 , 105–119.

Pashler, H., Bain, P. M., Bottge, B. A., Graesser, A., Koedinger, K., McDaniel, M., & Metcalfe, J. (2007). Organizing instruction and study to improve student learning. IES practice guide. NCER 2007–2004. National Center for Education Research .

Patel, R., Liu, R., & Koedinger, K. (2016). When to block versus interleave practice? Evidence against teaching fraction addition before fraction multiplication. In Proceedings of the 38th Annual Meeting of the Cognitive Science Society, Philadelphia, PA .

Penfound, B. (2017). Journey to interleaved practice #2 [Blog Post]. Retrieved from https://fullstackcalculus.com/2017/02/03/journey-to-interleaved-practice-2/ . Accessed 25 Dec 2017.

Penfound, B. [BryanPenfound]. (2016). Does blocked practice/learning lessen cognitive load? Does interleaved practice/learning provide productive struggle? [Tweet]. Retrieved from https://twitter.com/BryanPenfound/status/808759362244087808 . Accessed 25 Dec 2017.

Peterson, D. J., & Mulligan, N. W. (2010). Enactment and retrieval. Memory & Cognition, 38 , 233–243.

Picciotto, H. (2009). Lagging homework [Blog post]. Retrieved from http://blog.mathedpage.org/2013/06/lagging-homework.html . Accessed 25 Dec 2017.

Pomerance, L., Greenberg, J., & Walsh, K. (2016). Learning about learning: what every teacher needs to know. Retrieved from http://www.nctq.org/dmsView/Learning_About_Learning_Report . Accessed 25 Dec 2017.

Postman, L. (1976). Methodology of human learning. In W. K. Estes (Ed.), Handbook of learning and cognitive processes (Vol. 3). Hillsdale: Erlbaum.

Pressley, M., McDaniel, M. A., Turnure, J. E., Wood, E., & Ahmad, M. (1987). Generation and precision of elaboration: effects on intentional and incidental learning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 13 , 291–300.

Reed, S. K. (2008). Concrete examples must jibe with experience. Science, 322 , 1632–1633.

researchED. (2013). How it all began. Retrieved from http://www.researched.org.uk/about/our-story/ . Accessed 25 Dec 2017.

Ritchie, S. J., Della Sala, S., & McIntosh, R. D. (2013). Retrieval practice, with or without mind mapping, boosts fact learning in primary school children. PLoS One, 8 (11), e78976.

Rittle-Johnson, B. (2006). Promoting transfer: effects of self-explanation and direct instruction. Child Development, 77 , 1–15.

Roediger, H. L. (1985). Remembering Ebbinghaus. [Retrospective review of the book On Memory , by H. Ebbinghaus]. Contemporary Psychology, 30 , 519–523.

Roediger, H. L. (2013). Applying cognitive psychology to education translational educational science. Psychological Science in the Public Interest, 14 , 1–3.

Roediger, H. L., & Karpicke, J. D. (2006). The power of testing memory: basic research and implications for educational practice. Perspectives on Psychological Science, 1 , 181–210.

Roediger, H. L., Putnam, A. L., & Smith, M. A. (2011). Ten benefits of testing and their applications to educational practice. In J. Mester & B. Ross (Eds.), The psychology of learning and motivation: cognition in education (pp. 1–36). Oxford: Elsevier.

Roediger, H. L., Finn, B., & Weinstein, Y. (2012). Applications of cognitive science to education. In Della Sala, S., & Anderson, M. (Eds.), Neuroscience in education: the good, the bad, and the ugly . Oxford, UK: Oxford University Press.

Roelle, J., & Berthold, K. (2017). Effects of incorporating retrieval into learning tasks: the complexity of the tasks matters. Learning and Instruction, 49 , 142–156.

Rohrer, D. (2012). Interleaving helps students distinguish among similar concepts. Educational Psychology Review, 24(3), 355–367.

Rohrer, D., Dedrick, R. F., & Stershic, S. (2015). Interleaved practice improves mathematics learning. Journal of Educational Psychology, 107 , 900–908.

Rohrer, D., & Pashler, H. (2012). Learning styles: Where’s the evidence? Medical Education, 46 , 34–35.

Rohrer, D., & Taylor, K. (2007). The shuffling of mathematics problems improves learning. Instructional Science, 35 , 481–498.

Rose, N. (2014). Improving the effectiveness of homework [Blog post]. Retrieved from https://evidenceintopractice.wordpress.com/2014/03/20/improving-the-effectiveness-of-homework/ . Accessed 25 Dec 2017.

Sadoski, M. (2005). A dual coding view of vocabulary learning. Reading & Writing Quarterly, 21 , 221–238.

Saunders, K. (2016). It really is time we stopped talking about learning styles [Blog post]. Retrieved from http://martingsaunders.com/2016/10/it-really-is-time-we-stopped-talking-about-learning-styles/ . Accessed 25 Dec 2017.

Schwartz, D. (2007). If a picture is worth a thousand words, why are you reading this essay? Social Psychology Quarterly, 70 , 319–321.

Shumaker, H. (2016). Homework is wrecking our kids: the research is clear, let’s ban elementary homework. Salon. Retrieved from http://www.salon.com/2016/03/05/homework_is_wrecking_our_kids_the_research_is_clear_lets_ban_elementary_homework . Accessed 25 Dec 2017.

Smith, A. M., Floerke, V. A., & Thomas, A. K. (2016). Retrieval practice protects memory against acute stress. Science, 354 , 1046–1048.

Smith, M. A., Blunt, J. R., Whiffen, J. W., & Karpicke, J. D. (2016). Does providing prompts during retrieval practice improve learning? Applied Cognitive Psychology, 30 , 784–802.

Smith, M. A., & Karpicke, J. D. (2014). Retrieval practice with short-answer, multiple-choice, and hybrid formats. Memory, 22 , 784–802.

Smith, M. A., Roediger, H. L., & Karpicke, J. D. (2013). Covert retrieval practice benefits retention as much as overt retrieval practice. Journal of Experimental Psychology: Learning, Memory, and Cognition, 39 , 1712–1725.

Son, J. Y., & Rivas, M. J. (2016). Designing clicker questions to stimulate transfer. Scholarship of Teaching and Learning in Psychology, 2 , 193–207.

Szpunar, K. K., Khan, N. Y., & Schacter, D. L. (2013). Interpolated memory tests reduce mind wandering and improve learning of online lectures. Proceedings of the National Academy of Sciences, 110 , 6313–6317.

Thomson, R., & Mehring, J. (2016). Better vocabulary study strategies for long-term learning. Kwansei Gakuin University Humanities Review, 20 , 133–141.

Trafton, J. G., & Reiser, B. J. (1993). Studying examples and solving problems: contributions to skill acquisition . Technical report, Naval HCI Research Lab, Washington, DC, USA.

Tran, R., Rohrer, D., & Pashler, H. (2015). Retrieval practice: the lack of transfer to deductive inferences. Psychonomic Bulletin & Review, 22 , 135–140.

Turner, K. [doc_kristy]. (2016a). My dual coding (in red) and some y8 work @AceThatTest they really enjoyed practising the technique [Tweet]. Retrieved from https://twitter.com/doc_kristy/status/807220355395977216 . Accessed 25 Dec 2017.

Turner, K. [doc_kristy]. (2016b). @FurtherEdagogy @doctorwhy their work is revision work, they already have the words on a different page, to compliment not replace [Tweet]. Retrieved from https://twitter.com/doc_kristy/status/807360265100599301 . Accessed 25 Dec 2017.

Valle, A., Regueiro, B., Núñez, J. C., Rodríguez, S., Piñeiro, I., & Rosário, P. (2016). Academic goals, student homework engagement, and academic achievement in elementary school. Frontiers in Psychology, 7 .

Van Gog, T., & Sweller, J. (2015). Not new, but nearly forgotten: the testing effect decreases or even disappears as the complexity of learning materials increases. Educational Psychology Review, 27 , 247–264.

Wammes, J. D., Meade, M. E., & Fernandes, M. A. (2016). The drawing effect: evidence for reliable and robust memory benefits in free recall. Quarterly Journal of Experimental Psychology, 69 , 1752–1776.

Weinstein, Y., Gilmore, A. W., Szpunar, K. K., & McDermott, K. B. (2014). The role of test expectancy in the build-up of proactive interference in long-term memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 40 , 1039–1048.

Weinstein, Y., Nunes, L. D., & Karpicke, J. D. (2016). On the placement of practice questions during study. Journal of Experimental Psychology: Applied, 22 , 72–84.

Weinstein, Y., & Weinstein-Jones, F. (2017). Topic and quiz spacing spreadsheet: a planning tool for teachers [Blog Post]. Retrieved from http://www.learningscientists.org/blog/2017/5/11-1 . Accessed 25 Dec 2017.

Weinstein-Jones, F., & Weinstein, Y. (2017). Topic spacing spreadsheet for teachers [Excel macro]. Zenodo. http://doi.org/10.5281/zenodo.573764 . Accessed 25 Dec 2017.

Williams, D. [FurtherEdagogy]. (2016). @doctorwhy @doc_kristy word accompanying the visual? I’m unclear how removing words benefit? Would a flow chart better suit a scientific exp? [Tweet]. Retrieved from https://twitter.com/FurtherEdagogy/status/807356800509104128 . Accessed 25 Dec 2017.

Wood, B. (2017). And now for something a little bit different….[Blog post]. Retrieved from https://justateacherstandinginfrontofaclass.wordpress.com/2017/04/20/and-now-for-something-a-little-bit-different/ . Accessed 25 Dec 2017.

Wooldridge, C. L., Bugg, J. M., McDaniel, M. A., & Liu, Y. (2014). The testing effect with authentic educational materials: a cautionary note. Journal of Applied Research in Memory and Cognition, 3 , 214–221.

Young, C. (2016). Mini-tests. Retrieved from https://colleenyoung.wordpress.com/revision-activities/mini-tests/ . Accessed 25 Dec 2017.

Download references

Acknowledgements

Not applicable.

YW and MAS were partially supported by a grant from The IDEA Center.

Availability of data and materials

Author information, authors and affiliations.

Department of Psychology, University of Massachusetts Lowell, Lowell, MA, USA

Yana Weinstein

Department of Psychology, Boston College, Chestnut Hill, MA, USA

Christopher R. Madan

School of Psychology, University of Nottingham, Nottingham, UK

Department of Psychology, Rhode Island College, Providence, RI, USA

Megan A. Sumeracki

You can also search for this author in PubMed   Google Scholar

Contributions

YW took the lead on writing the “Spaced practice”, “Interleaving”, and “Elaboration” sections. CRM took the lead on writing the “Concrete examples” and “Dual coding” sections. MAS took the lead on writing the “Retrieval practice” section. All authors edited each others’ sections. All authors were involved in the conception and writing of the manuscript. All authors gave approval of the final version.

Corresponding author

Correspondence to Yana Weinstein .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

YW and MAS run a blog, “The Learning Scientists Blog”, which is cited in the tutorial review. The blog does not make money. Free resources on the strategies described in this tutorial review are provided on the blog. Occasionally, YW and MAS are invited by schools/school districts to present research findings from cognitive psychology applied to education.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Cite this article.

Weinstein, Y., Madan, C.R. & Sumeracki, M.A. Teaching the science of learning. Cogn. Research 3 , 2 (2018). https://doi.org/10.1186/s41235-017-0087-y

Download citation

Received : 20 December 2016

Accepted : 02 December 2017

Published : 24 January 2018

DOI : https://doi.org/10.1186/s41235-017-0087-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Research article
• Open access
• Published: 06 February 2017

Blended learning effectiveness: the relationship between student characteristics, design features and outcomes

• Mugenyi Justice Kintu   ORCID: orcid.org/0000-0002-4500-1168 1 , 2 ,
• Chang Zhu 2 &
• Edmond Kagambe 1  

International Journal of Educational Technology in Higher Education volume  14 , Article number:  7 ( 2017 ) Cite this article

765k Accesses

224 Citations

37 Altmetric

Metrics details

This paper investigates the effectiveness of a blended learning environment through analyzing the relationship between student characteristics/background, design features and learning outcomes. It is aimed at determining the significant predictors of blended learning effectiveness taking student characteristics/background and design features as independent variables and learning outcomes as dependent variables. A survey was administered to 238 respondents to gather data on student characteristics/background, design features and learning outcomes. The final semester evaluation results were used as a measure for performance as an outcome. We applied the online self regulatory learning questionnaire for data on learner self regulation, the intrinsic motivation inventory for data on intrinsic motivation and other self-developed instruments for measuring the other constructs. Multiple regression analysis results showed that blended learning design features (technology quality, online tools and face-to-face support) and student characteristics (attitudes and self-regulation) predicted student satisfaction as an outcome. The results indicate that some of the student characteristics/backgrounds and design features are significant predictors for student learning outcomes in blended learning.

Introduction

The teaching and learning environment is embracing a number of innovations and some of these involve the use of technology through blended learning. This innovative pedagogical approach has been embraced rapidly though it goes through a process. The introduction of blended learning (combination of face-to-face and online teaching and learning) initiatives is part of these innovations but its uptake, especially in the developing world faces challenges for it to be an effective innovation in teaching and learning. Blended learning effectiveness has quite a number of underlying factors that pose challenges. One big challenge is about how users can successfully use the technology and ensuring participants’ commitment given the individual learner characteristics and encounters with technology (Hofmann, 2014 ). Hofmann adds that users getting into difficulties with technology may result into abandoning the learning and eventual failure of technological applications. In a report by Oxford Group ( 2013 ), some learners (16%) had negative attitudes to blended learning while 26% were concerned that learners would not complete study in blended learning. Learners are important partners in any learning process and therefore, their backgrounds and characteristics affect their ability to effectively carry on with learning and being in blended learning, the design tools to be used may impinge on the effectiveness in their learning.

This study tackles blended learning effectiveness which has been investigated in previous studies considering grades, course completion, retention and graduation rates but no studies regarding effectiveness in view of learner characteristics/background, design features and outcomes have been done in the Ugandan university context. No studies have also been done on how the characteristics of learners and design features are predictors of outcomes in the context of a planning evaluation research (Guskey, 2000 ) to establish the effectiveness of blended learning. Guskey ( 2000 ) noted that planning evaluation fits in well since it occurs before the implementation of any innovation as well as allowing planners to determine the needs, considering participant characteristics, analyzing contextual matters and gathering baseline information. This study is done in the context of a plan to undertake innovative pedagogy involving use of a learning management system (moodle) for the first time in teaching and learning in a Ugandan university. The learner characteristics/backgrounds being investigated for blended learning effectiveness include self-regulation, computer competence, workload management, social and family support, attitude to blended learning, gender and age. We investigate the blended learning design features of learner interactions, face-to-face support, learning management system tools and technology quality while the outcomes considered include satisfaction, performance, intrinsic motivation and knowledge construction. Establishing the significant predictors of outcomes in blended learning will help to inform planners of such learning environments in order to put in place necessary groundwork preparations for designing blended learning as an innovative pedagogical approach.

Kenney and Newcombe ( 2011 ) did their comparison to establish effectiveness in view of grades and found that blended learning had higher average score than the non-blended learning environment. Garrison and Kanuka ( 2004 ) examined the transformative potential of blended learning and reported an increase in course completion rates, improved retention and increased student satisfaction. Comparisons between blended learning environments have been done to establish the disparity between academic achievement, grade dispersions and gender performance differences and no significant differences were found between the groups (Demirkol & Kazu, 2014 ).

However, blended learning effectiveness may be dependent on many other factors and among them student characteristics, design features and learning outcomes. Research shows that the failure of learners to continue their online education in some cases has been due to family support or increased workload leading to learner dropout (Park & Choi, 2009 ) as well as little time for study. Additionally, it is dependent on learner interactions with instructors since failure to continue with online learning is attributed to this. In Greer, Hudson & Paugh’s study as cited in Park and Choi ( 2009 ), family and peer support for learners is important for success in online and face-to-face learning. Support is needed for learners from all areas in web-based courses and this may be from family, friends, co-workers as well as peers in class. Greer, Hudson and Paugh further noted that peer encouragement assisted new learners in computer use and applications. The authors also show that learners need time budgeting, appropriate technology tools and support from friends and family in web-based courses. Peer support is required by learners who have no or little knowledge of technology, especially computers, to help them overcome fears. Park and Choi, ( 2009 ) showed that organizational support significantly predicts learners’ stay and success in online courses because employers at times are willing to reduce learners’ workload during study as well as supervisors showing that they are interested in job-related learning for employees to advance and improve their skills.

The study by Kintu and Zhu ( 2016 ) investigated the possibility of blended learning in a Ugandan University and examined whether student characteristics (such as self-regulation, attitudes towards blended learning, computer competence) and student background (such as family support, social support and management of workload) were significant factors in learner outcomes (such as motivation, satisfaction, knowledge construction and performance). The characteristics and background factors were studied along with blended learning design features such as technology quality, learner interactions, and Moodle with its tools and resources. The findings from that study indicated that learner attitudes towards blended learning were significant factors to learner satisfaction and motivation while workload management was a significant factor to learner satisfaction and knowledge construction. Among the blended learning design features, only learner interaction was a significant factor to learner satisfaction and knowledge construction.

The focus of the present study is on examining the effectiveness of blended learning taking into consideration learner characteristics/background, blended learning design elements and learning outcomes and how the former are significant predictors of blended learning effectiveness.

Studies like that of Morris and Lim ( 2009 ) have investigated learner and instructional factors influencing learning outcomes in blended learning. They however do not deal with such variables in the contexts of blended learning design as an aspect of innovative pedagogy involving the use of technology in education. Apart from the learner variables such as gender, age, experience, study time as tackled before, this study considers social and background aspects of the learners such as family and social support, self-regulation, attitudes towards blended learning and management of workload to find out their relationship to blended learning effectiveness. Identifying the various types of learner variables with regard to their relationship to blended learning effectiveness is important in this study as we embark on innovative pedagogy with technology in teaching and learning.

Literature review

This review presents research about blended learning effectiveness from the perspective of learner characteristics/background, design features and learning outcomes. It also gives the factors that are considered to be significant for blended learning effectiveness. The selected elements are as a result of the researcher’s experiences at a Ugandan university where student learning faces challenges with regard to learner characteristics and blended learning features in adopting the use of technology in teaching and learning. We have made use of Loukis, Georgiou, and Pazalo ( 2007 ) value flow model for evaluating an e-learning and blended learning service specifically considering the effectiveness evaluation layer. This evaluates the extent of an e-learning system usage and the educational effectiveness. In addition, studies by Leidner, Jarvenpaa, Dillon and Gunawardena as cited in Selim ( 2007 ) have noted three main factors that affect e-learning and blended learning effectiveness as instructor characteristics, technology and student characteristics. Heinich, Molenda, Russell, and Smaldino ( 2001 ) showed the need for examining learner characteristics for effective instructional technology use and showed that user characteristics do impact on behavioral intention to use technology. Research has dealt with learner characteristics that contribute to learner performance outcomes. They have dealt with emotional intelligence, resilience, personality type and success in an online learning context (Berenson, Boyles, & Weaver, 2008 ). Dealing with the characteristics identified in this study will give another dimension, especially for blended learning in learning environment designs and add to specific debate on learning using technology. Lin and Vassar, ( 2009 ) indicated that learner success is dependent on ability to cope with technical difficulty as well as technical skills in computer operations and internet navigation. This justifies our approach in dealing with the design features of blended learning in this study.

Learner characteristics/background and blended learning effectiveness

Studies indicate that student characteristics such as gender play significant roles in academic achievement (Oxford Group, 2013 ), but no study examines performance of male and female as an important factor in blended learning effectiveness. It has again been noted that the success of e- and blended learning is highly dependent on experience in internet and computer applications (Picciano & Seaman, 2007 ). Rigorous discovery of such competences can finally lead to a confirmation of high possibilities of establishing blended learning. Research agrees that the success of e-learning and blended learning can largely depend on students as well as teachers gaining confidence and capability to participate in blended learning (Hadad, 2007 ). Shraim and Khlaif ( 2010 ) note in their research that 75% of students and 72% of teachers were lacking in skills to utilize ICT based learning components due to insufficient skills and experience in computer and internet applications and this may lead to failure in e-learning and blended learning. It is therefore pertinent that since the use of blended learning applies high usage of computers, computer competence is necessary (Abubakar & Adetimirin, 2015 ) to avoid failure in applying technology in education for learning effectiveness. Rovai, ( 2003 ) noted that learners’ computer literacy and time management are crucial in distance learning contexts and concluded that such factors are meaningful in online classes. This is supported by Selim ( 2007 ) that learners need to posses time management skills and computer skills necessary for effectiveness in e- learning and blended learning. Self-regulatory skills of time management lead to better performance and learners’ ability to structure the physical learning environment leads to efficiency in e-learning and blended learning environments. Learners need to seek helpful assistance from peers and teachers through chats, email and face-to-face meetings for effectiveness (Lynch & Dembo, 2004 ). Factors such as learners’ hours of employment and family responsibilities are known to impede learners’ process of learning, blended learning inclusive (Cohen, Stage, Hammack, & Marcus, 2012 ). It was also noted that a common factor in failure and learner drop-out is the time conflict which is compounded by issues of family , employment status as well as management support (Packham, Jones, Miller, & Thomas, 2004 ). A study by Thompson ( 2004 ) shows that work, family, insufficient time and study load made learners withdraw from online courses.

Learner attitudes to blended learning can result in its effectiveness and these shape behavioral intentions which usually lead to persistence in a learning environment, blended inclusive. Selim, ( 2007 ) noted that the learners’ attitude towards e-learning and blended learning are success factors for these learning environments. Learner performance by age and gender in e-learning and blended learning has been found to indicate no significant differences between male and female learners and different age groups (i.e. young, middle-aged and old above 45 years) (Coldwell, Craig, Paterson, & Mustard, 2008 ). This implies that the potential for blended learning to be effective exists and is unhampered by gender or age differences.

Blended learning design features

The design features under study here include interactions, technology with its quality, face-to-face support and learning management system tools and resources.

Research shows that absence of learner interaction causes failure and eventual drop-out in online courses (Willging & Johnson, 2009 ) and the lack of learner connectedness was noted as an internal factor leading to learner drop-out in online courses (Zielinski, 2000 ). It was also noted that learners may not continue in e- and blended learning if they are unable to make friends thereby being disconnected and developing feelings of isolation during their blended learning experiences (Willging & Johnson, 2009). Learners’ Interactions with teachers and peers can make blended learning effective as its absence makes learners withdraw (Astleitner, 2000 ). Loukis, Georgious and Pazalo (2007) noted that learners’ measuring of a system’s quality, reliability and ease of use leads to learning efficiency and can be so in blended learning. Learner success in blended learning may substantially be affected by system functionality (Pituch & Lee, 2006 ) and may lead to failure of such learning initiatives (Shrain, 2012 ). It is therefore important to examine technology quality for ensuring learning effectiveness in blended learning. Tselios, Daskalakis, and Papadopoulou ( 2011 ) investigated learner perceptions after a learning management system use and found out that the actual system use determines the usefulness among users. It is again noted that a system with poor response time cannot be taken to be useful for e-learning and blended learning especially in cases of limited bandwidth (Anderson, 2004 ). In this study, we investigate the use of Moodle and its tools as a function of potential effectiveness of blended learning.

The quality of learning management system content for learners can be a predictor of good performance in e-and blended learning environments and can lead to learner satisfaction. On the whole, poor quality technology yields no satisfaction by users and therefore the quality of technology significantly affects satisfaction (Piccoli, Ahmad, & Ives, 2001 ). Continued navigation through a learning management system increases use and is an indicator of success in blended learning (Delone & McLean, 2003 ). The efficient use of learning management system and its tools improves learning outcomes in e-learning and blended learning environments.

It is noted that learner satisfaction with a learning management system can be an antecedent factor for blended learning effectiveness. Goyal and Tambe ( 2015 ) noted that learners showed an appreciation to Moodle’s contribution in their learning. They showed positivity with it as it improved their understanding of course material (Ahmad & Al-Khanjari, 2011 ). The study by Goyal and Tambe ( 2015 ) used descriptive statistics to indicate improved learning by use of uploaded syllabus and session plans on Moodle. Improved learning is also noted through sharing study material, submitting assignments and using the calendar. Learners in the study found Moodle to be an effective educational tool.

In blended learning set ups, face-to-face experiences form part of the blend and learner positive attitudes to such sessions could mean blended learning effectiveness. A study by Marriot, Marriot, and Selwyn ( 2004 ) showed learners expressing their preference for face-to-face due to its facilitation of social interaction and communication skills acquired from classroom environment. Their preference for the online session was only in as far as it complemented the traditional face-to-face learning. Learners in a study by Osgerby ( 2013 ) had positive perceptions of blended learning but preferred face-to-face with its step-by-stem instruction. Beard, Harper and Riley ( 2004 ) shows that some learners are successful while in a personal interaction with teachers and peers thus prefer face-to-face in the blend. Beard however dealt with a comparison between online and on-campus learning while our study combines both, singling out the face-to-face part of the blend. The advantage found by Beard is all the same relevant here because learners in blended learning express attitude to both online and face-to-face for an effective blend. Researchers indicate that teacher presence in face-to-face sessions lessens psychological distance between them and the learners and leads to greater learning. This is because there are verbal aspects like giving praise, soliciting for viewpoints, humor, etc and non-verbal expressions like eye contact, facial expressions, gestures, etc which make teachers to be closer to learners psychologically (Kelley & Gorham, 2009 ).

Learner outcomes

The outcomes under scrutiny in this study include performance, motivation, satisfaction and knowledge construction. Motivation is seen here as an outcome because, much as cognitive factors such as course grades are used in measuring learning outcomes, affective factors like intrinsic motivation may also be used to indicate outcomes of learning (Kuo, Walker, Belland, & Schroder, 2013 ). Research shows that high motivation among online learners leads to persistence in their courses (Menager-Beeley, 2004 ). Sankaran and Bui ( 2001 ) indicated that less motivated learners performed poorly in knowledge tests while those with high learning motivation demonstrate high performance in academics (Green, Nelson, Martin, & Marsh, 2006 ). Lim and Kim, ( 2003 ) indicated that learner interest as a motivation factor promotes learner involvement in learning and this could lead to learning effectiveness in blended learning.

Learner satisfaction was noted as a strong factor for effectiveness of blended and online courses (Wilging & Johnson, 2009) and dissatisfaction may result from learners’ incompetence in the use of the learning management system as an effective learning tool since, as Islam ( 2014 ) puts it, users may be dissatisfied with an information system due to ease of use. A lack of prompt feedback for learners from course instructors was found to cause dissatisfaction in an online graduate course. In addition, dissatisfaction resulted from technical difficulties as well as ambiguous course instruction Hara and Kling ( 2001 ). These factors, once addressed, can lead to learner satisfaction in e-learning and blended learning and eventual effectiveness. A study by Blocker and Tucker ( 2001 ) also showed that learners had difficulties with technology and inadequate group participation by peers leading to dissatisfaction within these design features. Student-teacher interactions are known to bring satisfaction within online courses. Study results by Swan ( 2001 ) indicated that student-teacher interaction strongly related with student satisfaction and high learner-learner interaction resulted in higher levels of course satisfaction. Descriptive results by Naaj, Nachouki, and Ankit ( 2012 ) showed that learners were satisfied with technology which was a video-conferencing component of blended learning with a mean of 3.7. The same study indicated student satisfaction with instructors at a mean of 3.8. Askar and Altun, ( 2008 ) found that learners were satisfied with face-to-face sessions of the blend with t-tests and ANOVA results indicating female scores as higher than for males in the satisfaction with face-to-face environment of the blended learning.

Studies comparing blended learning with traditional face-to-face have indicated that learners perform equally well in blended learning and their performance is unaffected by the delivery method (Kwak, Menezes, & Sherwood, 2013 ). In another study, learning experience and performance are known to improve when traditional course delivery is integrated with online learning (Stacey & Gerbic, 2007 ). Such improvement as noted may be an indicator of blended learning effectiveness. Our study however, delves into improved performance but seeks to establish the potential of blended learning effectiveness by considering grades obtained in a blended learning experiment. Score 50 and above is considered a pass in this study’s setting and learners scoring this and above will be considered to have passed. This will make our conclusions about the potential of blended learning effectiveness.

Regarding knowledge construction, it has been noted that effective learning occurs where learners are actively involved (Nurmela, Palonen, Lehtinen & Hakkarainen, 2003 , cited in Zhu, 2012 ) and this may be an indicator of learning environment effectiveness. Effective blended learning would require that learners are able to initiate, discover and accomplish the processes of knowledge construction as antecedents of blended learning effectiveness. A study by Rahman, Yasin and Jusoff ( 2011 ) indicated that learners were able to use some steps to construct meaning through an online discussion process through assignments given. In the process of giving and receiving among themselves, the authors noted that learners learned by writing what they understood. From our perspective, this can be considered to be accomplishment in the knowledge construction process. Their study further shows that learners construct meaning individually from assignments and this stage is referred to as pre-construction which for our study, is an aspect of discovery in the knowledge construction process.

Predictors of blended learning effectiveness

Researchers have dealt with success factors for online learning or those for traditional face-to-face learning but little is known about factors that predict blended learning effectiveness in view of learner characteristics and blended learning design features. This part of our study seeks to establish the learner characteristics/backgrounds and design features that predict blended learning effectiveness with regard to satisfaction, outcomes, motivation and knowledge construction. Song, Singleton, Hill, and Koh ( 2004 ) examined online learning effectiveness factors and found out that time management (a self-regulatory factor) was crucial for successful online learning. Eom, Wen, and Ashill ( 2006 ) using a survey found out that interaction, among other factors, was significant for learner satisfaction. Technical problems with regard to instructional design were a challenge to online learners thus not indicating effectiveness (Song et al., 2004 ), though the authors also indicated that descriptive statistics to a tune of 75% and time management (62%) impact on success of online learning. Arbaugh ( 2000 ) and Swan ( 2001 ) indicated that high levels of learner-instructor interaction are associated with high levels of user satisfaction and learning outcomes. A study by Naaj et al. ( 2012 ) indicated that technology and learner interactions, among other factors, influenced learner satisfaction in blended learning.

Objective and research questions of the current study

The objective of the current study is to investigate the effectiveness of blended learning in view of student satisfaction, knowledge construction, performance and intrinsic motivation and how they are related to student characteristics and blended learning design features in a blended learning environment.

Research questions

What are the student characteristics and blended learning design features for an effective blended learning environment?

Which factors (among the learner characteristics and blended learning design features) predict student satisfaction, learning outcomes, intrinsic motivation and knowledge construction?

Conceptual model of the present study

The reviewed literature clearly shows learner characteristics/background and blended learning design features play a part in blended learning effectiveness and some of them are significant predictors of effectiveness. The conceptual model for our study is depicted as follows (Fig.  1 ):

Conceptual model of the current study

Research design

This research applies a quantitative design where descriptive statistics are used for the student characteristics and design features data, t-tests for the age and gender variables to determine if they are significant in blended learning effectiveness and regression for predictors of blended learning effectiveness.

This study is based on an experiment in which learners participated during their study using face-to-face sessions and an on-line session of a blended learning design. A learning management system (Moodle) was used and learner characteristics/background and blended learning design features were measured in relation to learning effectiveness. It is therefore a planning evaluation research design as noted by Guskey ( 2000 ) since the outcomes are aimed at blended learning implementation at MMU. The plan under which the various variables were tested involved face-to-face study at the beginning of a 17 week semester which was followed by online teaching and learning in the second half of the semester. The last part of the semester was for another face-to-face to review work done during the online sessions and final semester examinations. A questionnaire with items on student characteristics, design features and learning outcomes was distributed among students from three schools and one directorate of postgraduate studies.

Participants

Cluster sampling was used to select a total of 238 learners to participate in this study. Out of the whole university population of students, three schools and one directorate were used. From these, one course unit was selected from each school and all the learners following the course unit were surveyed. In the school of Education ( n  = 70) and Business and Management Studies ( n  = 133), sophomore students were involved due to the fact that they have been introduced to ICT basics during their first year of study. Students of the third year were used from the department of technology in the School of Applied Sciences and Technology ( n  = 18) since most of the year two courses had a lot of practical aspects that could not be used for the online learning part. From the Postgraduate Directorate ( n  = 17), first and second year students were selected because learners attend a face-to-face session before they are given paper modules to study away from campus.

The study population comprised of 139 male students representing 58.4% and 99 females representing 41.6% with an average age of 24 years.

Instruments

The end of semester results were used to measure learner performance. The online self-regulated learning questionnaire (Barnard, Lan, To, Paton, & Lai, 2009 ) and the intrinsic motivation inventory (Deci & Ryan, 1982 ) were applied to measure the constructs on self regulation in the student characteristics and motivation in the learning outcome constructs. Other self-developed instruments were used for the other remaining variables of attitudes, computer competence, workload management, social and family support, satisfaction, knowledge construction, technology quality, interactions, learning management system tools and resources and face-to-face support.

Instrument reliability

Cronbach’s alpha was used to test reliability and the table below gives the results. All the scales and sub-scales had acceptable internal consistency reliabilities as shown in Table  1 below:

Data analysis

First, descriptive statistics was conducted. Shapiro-Wilk test was done to test normality of the data for it to qualify for parametric tests. The test results for normality of our data before the t- test resulted into significant levels (Male = .003, female = .000) thereby violating the normality assumption. We therefore used the skewness and curtosis results which were between −1.0 and +1.0 and assumed distribution to be sufficiently normal to qualify the data for a parametric test, (Pallant, 2010 ). An independent samples t -test was done to find out the differences in male and female performance to explain the gender characteristics in blended learning effectiveness. A one-way ANOVA between subjects was conducted to establish the differences in performance between age groups. Finally, multiple regression analysis was done between student variables and design elements with learning outcomes to determine the significant predictors for blended learning effectiveness.

Student characteristics, blended learning design features and learning outcomes ( RQ1 )

A t- test was carried out to establish the performance of male and female learners in the blended learning set up. This was aimed at finding out if male and female learners do perform equally well in blended learning given their different roles and responsibilities in society. It was found that male learners performed slightly better ( M  = 62.5) than their female counterparts ( M  = 61.1). An independent t -test revealed that the difference between the performances was not statistically significant ( t  = 1.569, df = 228, p  = 0.05, one tailed). The magnitude of the differences in the means is small with effect size ( d  = 0.18). A one way between subjects ANOVA was conducted on the performance of different age groups to establish the performance of learners of young and middle aged age groups (20–30, young & and 31–39, middle aged). This revealed a significant difference in performance (F(1,236 = 8.498, p < . 001).

Average percentages of the items making up the self regulated learning scale are used to report the findings about all the sub-scales in the learner characteristics/background scale. Results show that learner self-regulation was good enough at 72.3% in all the sub-scales of goal setting, environment structuring, task strategies, time management, help-seeking and self-evaluation among learners. The least in the scoring was task strategies at 67.7% and the highest was learner environment structuring at 76.3%. Learner attitude towards blended learning environment is at 76% in the sub-scales of learner autonomy, quality of instructional materials, course structure, course interface and interactions. The least scored here is attitude to course structure at 66% and their attitudes were high on learner autonomy and course interface both at 82%. Results on the learners’ computer competences are summarized in percentages in the table below (Table  2 ):

It can be seen that learners are skilled in word processing at 91%, email at 63.5%, spreadsheets at 68%, web browsers at 70.2% and html tools at 45.4%. They are therefore good enough in word processing and web browsing. Their computer confidence levels are reported at 75.3% and specifically feel very confident when it comes to working with a computer (85.7%). Levels of family and social support for learners during blended learning experiences are at 60.5 and 75% respectively. There is however a low score on learners being assisted by family members in situations of computer setbacks (33.2%) as 53.4% of the learners reported no assistance in this regard. A higher percentage (85.3%) is reported on learners getting support from family regarding provision of essentials for learning such as tuition. A big percentage of learners spend two hours on study while at home (35.3%) followed by one hour (28.2%) while only 9.7% spend more than three hours on study at home. Peers showed great care during the blended learning experience (81%) and their experiences were appreciated by the society (66%). Workload management by learners vis-à-vis studying is good at 60%. Learners reported that their workmates stand in for them at workplaces to enable them do their study in blended learning while 61% are encouraged by their bosses to go and improve their skills through further education and training. On the time spent on other activities not related to study, majority of the learners spend three hours (35%) while 19% spend 6 hours. Sixty percent of the learners have to answer to someone when they are not attending to other activities outside study compared to the 39.9% who do not and can therefore do study or those other activities.

The usability of the online system, tools and resources was below average as shown in the table below in percentages (Table  3 ):

However, learners became skilled at navigating around the learning management system (79%) and it was easy for them to locate course content, tools and resources needed such as course works, news, discussions and journal materials. They effectively used the communication tools (60%) and to work with peers by making posts (57%). They reported that online resources were well organized, user friendly and easy to access (71%) as well as well structured in a clear and understandable manner (72%). They therefore recommended the use of online resources for other course units in future (78%) because they were satisfied with them (64.3%). On the whole, the online resources were fine for the learners (67.2%) and useful as a learning resource (80%). The learners’ perceived usefulness/satisfaction with online system, tools, and resources was at 81% as the LMS tools helped them to communicate, work with peers and reflect on their learning (74%). They reported that using moodle helped them to learn new concepts, information and gaining skills (85.3%) as well as sharing what they knew or learned (76.4%). They enjoyed the course units (78%) and improved their skills with technology (89%).

Learner interactions were seen from three angles of cognitivism, collaborative learning and student-teacher interactions. Collaborative learning was average at 50% with low percentages in learners posting challenges to colleagues’ ideas online (34%) and posting ideas for colleagues to read online (37%). They however met oftentimes online (60%) and organized how they would work together in study during the face-to-face meetings (69%). The common form of communication medium frequently used by learners during the blended learning experience was by phone (34.5%) followed by whatsapp (21.8%), face book (21%), discussion board (11.8%) and email (10.9%). At the cognitive level, learners interacted with content at 72% by reading the posted content (81%), exchanging knowledge via the LMS (58.4%), participating in discussions on the forum (62%) and got course objectives and structure introduced during the face-to-face sessions (86%). Student-teacher interaction was reported at 71% through instructors individually working with them online (57.2%) and being well guided towards learning goals (81%). They did receive suggestions from instructors about resources to use in their learning (75.3%) and instructors provided learning input for them to come up with their own answers (71%).

The technology quality during the blended learning intervention was rated at 69% with availability of 72%, quality of the resources was at 68% with learners reporting that discussion boards gave right content necessary for study (71%) and the email exchanges containing relevant and much needed information (63.4%) as well as chats comprising of essential information to aid the learning (69%). Internet reliability was rated at 66% with a speed considered averagely good to facilitate online activities (63%). They however reported that there was intermittent breakdown during online study (67%) though they could complete their internet program during connection (63.4%). Learners eventually found it easy to download necessary materials for study in their blended learning experiences (71%).

Learner extent of use of the learning management system features was as shown in the table below in percentage (Table  4 ):

From the table, very rarely used features include the blog and wiki while very often used ones include the email, forum, chat and calendar.

The effectiveness of the LMS was rated at 79% by learners reporting that they found it useful (89%) and using it makes their learning activities much easier (75.2%). Moodle has helped learners to accomplish their learning tasks more quickly (74%) and that as a LMS, it is effective in teaching and learning (88%) with overall satisfaction levels at 68%. However, learners note challenges in the use of the LMS regarding its performance as having been problematic to them (57%) and only 8% of the learners reported navigation while 16% reported access as challenges.

Learner attitudes towards Face-to-face support were reported at 88% showing that the sessions were enjoyable experiences (89%) with high quality class discussions (86%) and therefore recommended that the sessions should continue in blended learning (89%). The frequency of the face-to-face sessions is shown in the table below as preferred by learners (Table  5 ).

Learners preferred face-to-face sessions after every month in the semester (33.6%) and at the beginning of the blended learning session only (27.7%).

Learners reported high intrinsic motivation levels with interest and enjoyment of tasks at 83.7%, perceived competence at 70.2%, effort/importance sub-scale at 80%, pressure/tension reported at 54%. The pressure percentage of 54% arises from learners feeling nervous (39.2%) and a lot of anxiety (53%) while 44% felt a lot of pressure during the blended learning experiences. Learners however reported the value/usefulness of blended learning at 91% with majority believing that studying online and face-to-face had value for them (93.3%) and were therefore willing to take part in blended learning (91.2%). They showed that it is beneficial for them (94%) and that it was an important way of studying (84.3%).

Learner satisfaction was reported at 81% especially with instructors (85%) high percentage reported on encouraging learner participation during the course of study 93%, course content (83%) with the highest being satisfaction with the good relationship between the objectives of the course units and the content (90%), technology (71%) with a high percentage on the fact that the platform was adequate for the online part of the learning (76%), interactions (75%) with participation in class at 79%, and face-to-face sessions (91%) with learner satisfaction high on face-to-face sessions being good enough for interaction and giving an overview of the courses when objectives were introduced at 92%.

Learners’ knowledge construction was reported at 78% with initiation and discovery scales scoring 84% with 88% specifically for discovering the learning points in the course units. The accomplishment scale in knowledge construction scored 71% and specifically the fact that learners were able to work together with group members to accomplish learning tasks throughout the study of the course units (79%). Learners developed reports from activities (67%), submitted solutions to discussion questions (68%) and did critique peer arguments (69%). Generally, learners performed well in blended learning in the final examination with an average pass of 62% and standard deviation of 7.5.

Significant predictors of blended learning effectiveness ( RQ 2)

A standard multiple regression analysis was done taking learner characteristics/background and design features as predictor variables and learning outcomes as criterion variables. The data was first tested to check if it met the linear regression test assumptions and results showed the correlations between the independent variables and each of the dependent variables (highest 0.62 and lowest 0.22) as not being too high, which indicated that multicollinearity was not a problem in our model. From the coefficients table, the VIF values ranged from 1.0 to 2.4, well below the cut off value of 10 and indicating no possibility of multicollinearity. The normal probability plot was seen to lie as a reasonably straight diagonal from bottom left to top right indicating normality of our data. Linearity was found suitable from the scatter plot of the standardized residuals and was rectangular in distribution. Outliers were no cause for concern in our data since we had only 1% of all cases falling outside 3.0 thus proving the data as a normally distributed sample. Our R -square values was at 0.525 meaning that the independent variables explained about 53% of the variance in overall satisfaction, motivation and knowledge construction of the learners. All the models explaining the three dependent variables of learner satisfaction, intrinsic motivation and knowledge construction were significant at the 0.000 probability level (Table  6 ).

From the table above, design features (technology quality and online tools and resources), and learner characteristics (attitudes to blended learning, self-regulation) were significant predictors of learner satisfaction in blended learning. This means that good technology with the features involved and the learner positive attitudes with capacity to do blended learning with self drive led to their satisfaction. The design features (technology quality, interactions) and learner characteristics (self regulation and social support), were found to be significant predictors of learner knowledge construction. This implies that learners’ capacity to go on their work by themselves supported by peers and high levels of interaction using the quality technology led them to construct their own ideas in blended learning. Design features (technology quality, online tools and resources as well as learner interactions) and learner characteristics (self regulation), significantly predicted the learners’ intrinsic motivation in blended learning suggesting that good technology, tools and high interaction levels with independence in learning led to learners being highly motivated. Finally, none of the independent variables considered under this study were predictors of learning outcomes (grade).

In this study we have investigated learning outcomes as dependent variables to establish if particular learner characteristics/backgrounds and design features are related to the outcomes for blended learning effectiveness and if they predict learning outcomes in blended learning. We took students from three schools out of five and one directorate of post-graduate studies at a Ugandan University. The study suggests that the characteristics and design features examined are good drivers towards an effective blended learning environment though a few of them predicted learning outcomes in blended learning.

Student characteristics/background, blended learning design features and learning outcomes

The learner characteristics, design features investigated are potentially important for an effective blended learning environment. Performance by gender shows a balance with no statistical differences between male and female. There are statistically significant differences ( p  < .005) in the performance between age groups with means of 62% for age group 20–30 and 67% for age group 31 –39. The indicators of self regulation exist as well as positive attitudes towards blended learning. Learners do well with word processing, e-mail, spreadsheets and web browsers but still lag below average in html tools. They show computer confidence at 75.3%; which gives prospects for an effective blended learning environment in regard to their computer competence and confidence. The levels of family and social support for learners stand at 61 and 75% respectively, indicating potential for blended learning to be effective. The learners’ balance between study and work is a drive factor towards blended learning effectiveness since their management of their workload vis a vis study time is at 60 and 61% of the learners are encouraged to go for study by their bosses. Learner satisfaction with the online system and its tools shows prospect for blended learning effectiveness but there are challenges in regard to locating course content and assignments, submitting their work and staying on a task during online study. Average collaborative, cognitive learning as well as learner-teacher interactions exist as important factors. Technology quality for effective blended learning is a potential for effectiveness though features like the blog and wiki are rarely used by learners. Face-to-face support is satisfactory and it should be conducted every month. There is high intrinsic motivation, satisfaction and knowledge construction as well as good performance in examinations ( M  = 62%, SD = 7.5); which indicates potentiality for blended learning effectiveness.

Significant predictors of blended learning effectiveness

Among the design features, technology quality, online tools and face-to-face support are predictors of learner satisfaction while learner characteristics of self regulation and attitudes to blended learning are predictors of satisfaction. Technology quality and interactions are the only design features predicting learner knowledge construction, while social support, among the learner backgrounds, is a predictor of knowledge construction. Self regulation as a learner characteristic is a predictor of knowledge construction. Self regulation is the only learner characteristic predicting intrinsic motivation in blended learning while technology quality, online tools and interactions are the design features predicting intrinsic motivation. However, all the independent variables are not significant predictors of learning performance in blended learning.

The high computer competences and confidence is an antecedent factor for blended learning effectiveness as noted by Hadad ( 2007 ) and this study finds learners confident and competent enough for the effectiveness of blended learning. A lack in computer skills causes failure in e-learning and blended learning as noted by Shraim and Khlaif ( 2010 ). From our study findings, this is no threat for blended learning our case as noted by our results. Contrary to Cohen et al. ( 2012 ) findings that learners’ family responsibilities and hours of employment can impede their process of learning, it is not the case here since they are drivers to the blended learning process. Time conflict, as compounded by family, employment status and management support (Packham et al., 2004 ) were noted as causes of learner failure and drop out of online courses. Our results show, on the contrary, that these factors are drivers for blended learning effectiveness because learners have a good balance between work and study and are supported by bosses to study. In agreement with Selim ( 2007 ), learner positive attitudes towards e-and blended learning environments are success factors. In line with Coldwell et al. ( 2008 ), no statistically significant differences exist between age groups. We however note that Coldwel, et al dealt with young, middle-aged and old above 45 years whereas we dealt with young and middle aged only.

Learner interactions at all levels are good enough and contrary to Astleitner, ( 2000 ) that their absence makes learners withdraw, they are a drive factor here. In line with Loukis (2007) the LMS quality, reliability and ease of use lead to learning efficiency as technology quality, online tools are predictors of learner satisfaction and intrinsic motivation. Face-to-face sessions should continue on a monthly basis as noted here and is in agreement with Marriot et al. ( 2004 ) who noted learner preference for it for facilitating social interaction and communication skills. High learner intrinsic motivation leads to persistence in online courses as noted by Menager-Beeley, ( 2004 ) and is high enough in our study. This implies a possibility of an effectiveness blended learning environment. The causes of learner dissatisfaction noted by Islam ( 2014 ) such as incompetence in the use of the LMS are contrary to our results in our study, while the one noted by Hara and Kling, ( 2001 ) as resulting from technical difficulties and ambiguous course instruction are no threat from our findings. Student-teacher interaction showed a relation with satisfaction according to Swan ( 2001 ) but is not a predictor in our study. Initiating knowledge construction by learners for blended learning effectiveness is exhibited in our findings and agrees with Rahman, Yasin and Jusof ( 2011 ). Our study has not agreed with Eom et al. ( 2006 ) who found learner interactions as predictors of learner satisfaction but agrees with Naaj et al. ( 2012 ) regarding technology as a predictor of learner satisfaction.

Conclusion and recommendations

An effective blended learning environment is necessary in undertaking innovative pedagogical approaches through the use of technology in teaching and learning. An examination of learner characteristics/background, design features and learning outcomes as factors for effectiveness can help to inform the design of effective learning environments that involve face-to-face sessions and online aspects. Most of the student characteristics and blended learning design features dealt with in this study are important factors for blended learning effectiveness. None of the independent variables were identified as significant predictors of student performance. These gaps are open for further investigation in order to understand if they can be significant predictors of blended learning effectiveness in a similar or different learning setting.

In planning to design and implement blended learning, we are mindful of the implications raised by this study which is a planning evaluation research for the design and eventual implementation of blended learning. Universities should be mindful of the interplay between the learner characteristics, design features and learning outcomes which are indicators of blended learning effectiveness. From this research, learners manifest high potential to take on blended learning more especially in regard to learner self-regulation exhibited. Blended learning is meant to increase learners’ levels of knowledge construction in order to create analytical skills in them. Learner ability to assess and critically evaluate knowledge sources is hereby established in our findings. This can go a long way in producing skilled learners who can be innovative graduates enough to satisfy employment demands through creativity and innovativeness. Technology being less of a shock to students gives potential for blended learning design. Universities and other institutions of learning should continue to emphasize blended learning approaches through installation of learning management systems along with strong internet to enable effective learning through technology especially in the developing world.

Abubakar, D. & Adetimirin. (2015). Influence of computer literacy on post-graduates’ use of e-resources in Nigerian University Libraries. Library Philosophy and Practice. From http://digitalcommons.unl.edu/libphilprac/ . Retrieved 18 Aug 2015.

Ahmad, N., & Al-Khanjari, Z. (2011). Effect of Moodle on learning: An Oman perception. International Journal of Digital Information and Wireless Communications (IJDIWC), 1 (4), 746–752.

Google Scholar  

Anderson, T. (2004). Theory and Practice of Online Learning . Canada: AU Press, Athabasca University.

Arbaugh, J. B. (2000). How classroom environment and student engagement affect learning in internet-basedMBAcourses. Business Communication Quarterly, 63 (4), 9–18.

Article   Google Scholar  

Askar, P. & Altun, A. (2008). Learner satisfaction on blended learning. E-Leader Krakow , 2008.

Astleitner, H. (2000) Dropout and distance education. A review of motivational and emotional strategies to reduce dropout in web-based distance education. In Neuwe Medien in Unterricht, Aus-und Weiterbildung Waxmann Munster, New York.

Barnard, L., Lan, W. Y., To, Y. M., Paton, V. O., & Lai, S. (2009). Measuring self regulation in online and blended learning environments’. Internet and Higher Education, 12 (1), 1–6.

Beard, L. A., Harper, C., & Riley, G. (2004). Online versus on-campus instruction: student attitudes & perceptions. TechTrends, 48 (6), 29–31.

Berenson, R., Boyles, G., & Weaver, A. (2008). Emotional intelligence as a predictor for success in online learning. International Review of Research in open & Distance Learning, 9 (2), 1–16.

Blocker, J. M., & Tucker, G. (2001). Using constructivist principles in designing and integrating online collaborative interactions. In F. Fuller & R. McBride (Eds.), Distance education. Proceedings of the Society for Information Technology & Teacher Education International Conference (pp. 32–36). ERIC Document Reproduction Service No. ED 457 822.

Cohen, K. E., Stage, F. K., Hammack, F. M., & Marcus, A. (2012). Persistence of master’s students in the United States: Developing and testing of a conceptual model . USA: PhD Dissertation, New York University.

Coldwell, J., Craig, A., Paterson, T., & Mustard, J. (2008). Online students: Relationships between participation, demographics and academic performance. The Electronic Journal of e-learning, 6 (1), 19–30.

Deci, E. L., & Ryan, R. M. (1982). Intrinsic Motivation Inventory. Available from selfdeterminationtheory.org/intrinsic-motivation-inventory/ . Accessed 2 Aug 2016.

Delone, W. H., & McLean, E. R. (2003). The Delone and McLean model of information systems success: A Ten-year update. Journal of Management Information Systems, 19 (4), 9–30.

Demirkol, M., & Kazu, I. Y. (2014). Effect of blended environment model on high school students’ academic achievement. The Turkish Online Journal of Educational Technology, 13 (1), 78–87.

Eom, S., Wen, H., & Ashill, N. (2006). The determinants of students’ perceived learning outcomes and satisfaction in university online education: an empirical investigation’. Decision Sciences Journal of Innovative Education, 4 (2), 215–235.

Garrison, D. R., & Kanuka, H. (2004). Blended learning: Uncovering its transformative potential in higher education. Internet and Higher Education, 7 (2), 95–105.

Goyal, E., & Tambe, S. (2015). Effectiveness of Moodle-enabled blended learning in private Indian Business School teaching NICHE programs. The Online Journal of New Horizons in Education, 5 (2), 14–22.

Green, J., Nelson, G., Martin, A. J., & Marsh, H. (2006). The causal ordering of self-concept and academic motivation and its effect on academic achievement. International Education Journal, 7 (4), 534–546.

Guskey, T. R. (2000). Evaluating Professional Development . Thousands Oaks: Corwin Press.

Hadad, W. (2007). ICT-in-education toolkit reference handbook . InfoDev. from http://www.infodev.org/en/Publication.301.html . Retrieved 04 Aug 2015.

Hara, N. & Kling, R. (2001). Student distress in web-based distance education. Educause Quarterly. 3 (2001).

Heinich, R., Molenda, M., Russell, J. D., & Smaldino, S. E. (2001). Instructional Media and Technologies for Learning (7th ed.). Englewood Cliffs: Prentice-Hall.

Hofmann, J. (2014). Solutions to the top 10 challenges of blended learning. Top 10 challenges of blended learning. Available on cedma-europe.org .

Islam, A. K. M. N. (2014). Sources of satisfaction and dissatisfaction with a learning management system in post-adoption stage: A critical incident technique approach. Computers in Human Behaviour, 30 , 249–261.

Kelley, D. H. & Gorham, J. (2009) Effects of immediacy on recall of information. Communication Education, 37 (3), 198–207.

Kenney, J., & Newcombe, E. (2011). Adopting a blended learning approach: Challenges, encountered and lessons learned in an action research study. Journal of Asynchronous Learning Networks, 15 (1), 45–57.

Kintu, M. J., & Zhu, C. (2016). Student characteristics and learning outcomes in a blended learning environment intervention in a Ugandan University. Electronic Journal of e-Learning, 14 (3), 181–195.

Kuo, Y., Walker, A. E., Belland, B. R., & Schroder, L. E. E. (2013). A predictive study of student satisfaction in online education programs. International Review of Research in Open and Distributed Learning, 14 (1), 16–39.

Kwak, D. W., Menezes, F. M., & Sherwood, C. (2013). Assessing the impact of blended learning on student performance. Educational Technology & Society, 15 (1), 127–136.

Lim, D. H., & Kim, H. J. (2003). Motivation and learner characteristics affecting online learning and learning application. Journal of Educational Technology Systems, 31 (4), 423–439.

Lim, D. H., & Morris, M. L. (2009). Learner and instructional factors influencing learner outcomes within a blended learning environment. Educational Technology & Society, 12 (4), 282–293.

Lin, B., & Vassar, J. A. (2009). Determinants for success in online learning communities. International Journal of Web-based Communities, 5 (3), 340–350.

Loukis, E., Georgiou, S. & Pazalo, K. (2007). A value flow model for the evaluation of an e-learning service. ECIS, 2007 Proceedings, paper 175.

Lynch, R., & Dembo, M. (2004). The relationship between self regulation and online learning in a blended learning context. The International Review of Research in Open and Distributed Learning, 5 (2), 1–16.

Marriot, N., Marriot, P., & Selwyn. (2004). Accounting undergraduates’ changing use of ICT and their views on using the internet in higher education-A Research note. Accounting Education, 13 (4), 117–130.

Menager-Beeley, R. (2004). Web-based distance learning in a community college: The influence of task values on task choice, retention and commitment. (Doctoral dissertation, University of Southern California). Dissertation Abstracts International, 64 (9-A), 3191.

Naaj, M. A., Nachouki, M., & Ankit, A. (2012). Evaluating student satisfaction with blended learning in a gender-segregated environment. Journal of Information Technology Education: Research, 11 , 185–200.

Nurmela, K., Palonen, T., Lehtinen, E. & Hakkarainen, K. (2003). Developing tools for analysing CSCL process. In Wasson, B. Ludvigsen, S. & Hoppe, V. (eds), Designing for change in networked learning environments (pp 333–342). Dordrecht, The Netherlands, Kluwer.

Osgerby, J. (2013). Students’ perceptions of the introduction of a blended learning environment: An exploratory case study. Accounting Education, 22 (1), 85–99.

Oxford Group, (2013). Blended learning-current use, challenges and best practices. From http://www.kineo.com/m/0/blended-learning-report-202013.pdf . Accessed on 17 Mar 2016.

Packham, G., Jones, P., Miller, C., & Thomas, B. (2004). E-learning and retention key factors influencing student withdrawal. Education and Training, 46 (6–7), 335–342.

Pallant, J. (2010). SPSS Survival Mannual (4th ed.). Maidenhead: OUP McGraw-Hill.

Park, J.-H., & Choi, H. J. (2009). Factors influencing adult learners’ decision to drop out or persist in online learning. Educational Technology & Society, 12 (4), 207–217.

Picciano, A., & Seaman, J. (2007). K-12 online learning: A survey of U.S. school district administrators . New York, USA: Sloan-C.

Piccoli, G., Ahmad, R., & Ives, B. (2001). Web-based virtual learning environments: a research framework and a preliminary assessment of effectiveness in basic IT skill training. MIS Quarterly, 25 (4), 401–426.

Pituch, K. A., & Lee, Y. K. (2006). The influence of system characteristics on e-learning use. Computers & Education, 47 (2), 222–244.

Rahman, S. et al, (2011). Knowledge construction process in online learning. Middle East Journal of Scientific Research, 8 (2), 488–492.

Rovai, A. P. (2003). In search of higher persistence rates in distance education online programs. Computers & Education, 6 (1), 1–16.

Sankaran, S., & Bui, T. (2001). Impact of learning strategies and motivation on performance: A study in Web-based instruction. Journal of Instructional Psychology, 28 (3), 191–198.

Selim, H. M. (2007). Critical success factors for e-learning acceptance: Confirmatory factor models. Computers & Education, 49 (2), 396–413.

Shraim, K., & Khlaif, Z. N. (2010). An e-learning approach to secondary education in Palestine: opportunities and challenges. Information Technology for Development, 16 (3), 159–173.

Shrain, K. (2012). Moving towards e-learning paradigm: Readiness of higher education instructors in Palestine. International Journal on E-Learning, 11 (4), 441–463.

Song, L., Singleton, E. S., Hill, J. R., & Koh, M. H. (2004). Improving online learning: student perceptions of useful and challenging characteristics’. Internet and Higher Education, 7 (1), 59–70.

Stacey, E., & Gerbic, P. (2007). Teaching for blended learning: research perspectives from on-campus and distance students. Education and Information Technologies, 12 , 165–174.

Swan, K. (2001). Virtual interactivity: design factors affecting student satisfaction and perceived learning in asynchronous online courses. Distance Education, 22 (2), 306–331.

Article   MathSciNet   Google Scholar  

Thompson, E. (2004). Distance education drop-out: What can we do? In R. Pospisil & L. Willcoxson (Eds.), Learning Through Teaching (Proceedings of the 6th Annual Teaching Learning Forum, pp. 324–332). Perth, Australia: Murdoch University.

Tselios, N., Daskalakis, S., & Papadopoulou, M. (2011). Assessing the acceptance of a blended learning university course. Educational Technology & Society, 14 (2), 224–235.

Willging, P. A., & Johnson, S. D. (2009). Factors that influence students’ decision to drop-out of online courses. Journal of Asynchronous Learning Networks, 13 (3), 115–127.

Zhu, C. (2012). Student satisfaction, performance and knowledge construction in online collaborative learning. Educational Technology & Society, 15 (1), 127–137.

Zielinski, D. (2000). Can you keep learners online? Training, 37 (3), 64–75.

Download references

Authors’ contribution

MJK conceived the study idea, developed the conceptual framework, collected the data, analyzed it and wrote the article. CZ gave the technical advice concerning the write-up and advised on relevant corrections to be made before final submission. EK did the proof-reading of the article as well as language editing. All authors read and approved the final manuscript.

Competing interests

The authors declare that they have no competing interests.

Author information

Authors and affiliations.

Mountains of the Moon University, P.O. Box 837, Fort Portal, Uganda

Mugenyi Justice Kintu & Edmond Kagambe

Vrije Universiteit Brussel, Pleinlaan 2, Brussels, 1050, Ixelles, Belgium

Mugenyi Justice Kintu & Chang Zhu

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Mugenyi Justice Kintu .

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Cite this article.

Kintu, M.J., Zhu, C. & Kagambe, E. Blended learning effectiveness: the relationship between student characteristics, design features and outcomes. Int J Educ Technol High Educ 14 , 7 (2017). https://doi.org/10.1186/s41239-017-0043-4

Download citation

Received : 13 July 2016

Accepted : 23 November 2016

Published : 06 February 2017

DOI : https://doi.org/10.1186/s41239-017-0043-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Blended learning effectiveness
• Learner characteristics
• Design features
• Learning outcomes and significant predictors

• Open access
• Published: 21 October 2020

Uncovering trend-based research insights on teaching and learning in big data

• Young-Eun Park   ORCID: orcid.org/0000-0002-3057-7930 1  

Journal of Big Data volume  7 , Article number:  93 ( 2020 ) Cite this article

4475 Accesses

15 Citations

1 Altmetric

Metrics details

Along with the big data era, digital transformation has had a transformative effect on modern education tremendously in higher education. It transforms an institutional core value of education to better meet students’ needs by leveraging big data and digital technology. Based on this background, this study attempts to catch the principal trends, or new directions, paradigms as predictors with an association of each topic by discovering the up-to-date research trends on teaching and learning in higher education via text mining techniques. For this, 285 research articles in the area of teaching and learning in higher education were collected from several big databases (distinguishable publishers’ web platforms) through search engines for 2 years in 2018–2019. Then it was analyzed using a semantic network analysis that processes natural human language. Consequently, research results show a relatively high connection with ‘student’ or ‘student-centered/led’ rather than ‘teacher-led.’ Moreover, it exhibits that the practice and assessment in learning can be attained via diverse learning activities, containing community or outreach activities. Besides, research in academic contexts, experience-based classes, the effect of group activities, how students’ feelings or perceptions, and relationships affect learning outcomes were addressed as the main topics through topic modeling of LDA, a machine learning algorithm. This study proposes that educators, researchers, and even academic leaders can exert extraordinary power to reshape educational quality programs for future education and in a timely manner with recognizable trends or agendas in teaching and learning of higher education.

Introduction

Digital transformation in higher education covers many things from using digital tools such as LMS (Learning Management System), Interactive whiteboard, various application tools for e-learning, etc. in university classrooms to digitize university documents. However, it does not just involve in utilizing advanced tools. The change and growing demand can be more profound and deep, incorporating whole aspects of education. Although there is an initial resistance to new technologies caused by continually changing, we have no choice but to hold and follow this latest trend [ 11 , 53 ]. Moreover, no matter what has happened to us in the past and what is going on in our current lives, there is no power to keep us from having an unknown future with high uncertainty. It is the main reason we should be aware of the latest trends or changing circumstances around us. Accordingly, efforts to look at trends or predict a new paradigm based on historical big data in teaching and learning has been around continuously in the various educational sectors of higher education [ 1 , 4 , 16 , 17 , 39 , 53 ]. In particular, main issues, which have traditionally started from finding trends about students’ learning [ 17 , 20 ], expand into teaching and learning broadly [ 22 , 58 ]. Furthermore, now it amplifies into research and development [ 7 , 16 , 18 , 19 , 27 , 29 , 37 , 41 , 42 , 44 ]. Besides, previous studies favored measuring the higher education curriculum’s achievement or improvement and measuring the outcomes to fulfill the highest standards [ 2 , 6 ]. Past research trends that studied diverse factors affecting students’ learning attitudes and learning outcomes have been extended to diversified research areas. Those comprehensively connect distinct educational issues and teaching and learning methods. For example, new techniques such as flipped learning, blended learning, and interactive ways maximize education effectiveness [ 45 , 48 , 49 , 51 , 64 ]. Furthermore, there is an ever-increasing range of connections with multiple stakeholders around education, management, administration, policy, and leadership in higher education [ 8 , 35 , 44 ].

Thus, recognizing the latest issues or trends of how teaching and learning in higher education have conducted in research has a special meaning to reflect the present and previous studies and seek desirable directions in the development of future education [ 7 , 18 , 19 , 27 , 29 , 37 , 41 , 42 ]. Consequently, a number of researchers in diverse educational sectors have shown many distinguished studies related to those topics with different perspectives to keep up with the most popular and the latest educational trends. However, most studies are independent studies based on a specific situation or context rather than grasping the overall academic flow. Besides, most of them were investigated with qualitative and subjective methods through surveys and interviews, such as focus group interviews. Also, reviews of scientific publications such as articles or books, course syllabus, etc. are implemented to manually gather non-numerical data. This qualitative research approach provides obvious implications and contributions comprehensively to understand and explore an educational situation or a higher education phenomenon. However, it is a study confined to individual conditions or specific circumstances independently. Moreover, there is a potential to create subjective judgments or arguments when applying for different occasions or environments [ 41 , 42 ]. Hence, this study aims to supplement the limitations of those existing studies and grasp the general current flow of teaching and learning through big data. By identifying the latest teaching and learning trends in higher education through semantic network analysis, a text mining technique using big data of unstructured texts, this study catches the fundamental directions and main topics, including the linkage of each issue. In conclusion, this research can provide a wealth of insights to guide educators, researchers, and academic leaders in higher education with a trend-based approach in quality education. The latest educational issues identified using big data and advanced technologies will help renovate and reshape future educational programs. Also, in the face of rapidly changing educational trends and dynamic environments, it will meet the different needs of diverse stakeholders around higher education.

The rest of this research paper is organized and illustrated as follows. The next section presents reviewing existing studies extensively. Then, a proposed research framework with research questions is addressed. After that, the data and methodology are presented and analyzed in the section of results and discussion. Lastly, the implication and conclusion are included in a future research direction in the last section.

Literature review

Researches about teaching and learning in higher education.

A past or present, a critical issue to higher education lies in teaching and learning. Accordingly, different agendas from a wide variety of perspectives in higher education are progressively evolving. However, we need to revisit the basic principals and focus on the essential subjects of teaching and learning [ 9 , 45 , 48 , 49 , 51 , 60 , 64 ]. Traditionally, education has focused on finding the diverse matters surrounding the roles of students and teachers, respectively, to examine the learning effects and the factors that influence it, that is, to find cause and effect (causality) in teaching and learning [ 16 , 17 ]. It has progressed into an in-depth discussion of how psychological factors, such as students’ perceptions, feelings, or relationships with teachers, influence learning outcomes [ 24 , 36 , 62 ]. What is more, many studies examined how various demographical factors such as gender, race, and income level of families affect students’ learning or its outcomes [ 25 , 30 , 57 ]. In addition to this approach, attention was paid to the teachers’ point of view to find out more effective teaching methods and what new teaching methods were being developed and used and how those methods worked. For example, flipped learning, blended learning, online learning, or interactive learning using various technology tools or simulation game. These are the most recently adopted teaching methods [ 19 , 21 , 23 , 34 , 60 ].

Other endeavors have been made to find general trends in teaching and learning with a holistic perspective [ 13 , 16 , 19 , 20 , 22 , 27 , 37 , 41 , 42 ]. Deng et al. [ 13 ] focused on identifying trends and categorizing the study on Massive Open Online Courses in teaching and learning. Elton and Laurillard [ 16 ] sought to find research trends in students’ learning and discover new research paradigms. They analyzed the trends to uncover the determinants of how humans learn, the differences among individuals in human education, how content elements affect learning, and how contextual factors affect learning. Guri-Rozenblit [ 20 ] reviewed and analyzed four books that can use to examine trends in learning. Based on this, he defined the definitions of distance education and open education. He covered a wide range of free public/distance systems, course design, advanced technology, and delivery systems, student support and survival issues, and lastly, inter-university and inter-institutional collaboration issues. Henderson et al. [ 22 ] studied students’ perceptions of useful digital technologies in teaching and learning in the university, which has an online education or interactive education through an online system. It is attracting attention as research that captures the transforming the nature of university education. Nikitina and Lapina [ 41 ] proposed that recent business education trends were organized into three categories: a curriculum that meets the desire of society and business, partnership & networking, and a modern and flexible teaching method in their research. Besides, new forms of teaching and learning, including blended learning, interactive learning, and flipped learning, have been addressed by many scholars [ 21 , 34 , 60 ]. Besides that, a large number of studies have mainly concentrated on the numerous factors or trends affecting educational development and management [ 19 , 27 , 37 , 41 , 42 ]. For instance, Nomuoja [ 42 ] studied the current trends emerging in business schools of higher education. Consequently, career awareness, risk management, people-oriented strategy and management, and skills-based curriculum were mainly discussed. Moreover, there are interviews from international top MBA schools to discover major MBA trends such as ‘growing trend of double degrees,’ growth acceleration of online or technology-based education and blended learning in business education [ 19 , 23 , 37 ]. Accordingly, a considerable amount of research has been done with broad and varied teaching and learning perspectives. However, most of them were independent studies investigated based on a specific situation or context rather than grasping the educational flow or trends. Moreover, there is still a lack of study that looks at the global direction of such research more objectively and quantitatively using big data. Hence, this study aims to fill in the gap of these existing studies.

Semantic network analysis using big data of the unstructured text

We live in an era where all aspects of our lives are uncertain and rapidly changing [ 28 , 33 , 47 ]. The best way to cope with this uncertain and unknown future is to predict and prepare for the future based on various historical big data by reducing this prediction error. In this regard, people focus on using big data to read trends and prepare for the unknown future. This substantial phenomenon is well represented in diverse and separate research fields as well. Many scholars in a very different area are working actively to discover insights into big data using various data mining techniques [ 1 , 15 , 26 , 38 , 46 , 47 , 52 , 53 , 55 , 63 ]. We can deal with big data or data sets due to the breakthrough technology, which is too complicated or broad to be dealt with by traditional data-processing approaches. In particular, it became possible to analyze a large amount of unstructured text data through text mining, one of the data mining techniques, as linguistic techniques have developed and applied to diverse areas [ 11 , 38 , 61 ].

A morphological or semantic network analysis deals with dividing a sentence into the smallest meaningful unit of language, namely, morphemes by importing unstructured text data such as speeches, comments, or posting in social media like Twitter, Instagram, or any bibliographic information (for example, books, scholarly articles, records, interviews, etc.) [ 15 , 26 , 43 , 47 , 63 ]. It automatically extracts words in sentences, paragraphs, and documents to make it simple to construct a word-to-word network according to the degree of nearness or adjacency between words [ 3 , 26 ]. Based on that, network structures provide intuitive and beneficial illustrations for modeling semantic inference and knowledge [ 55 ]. Through this, we can comprehend the relationship among words or understand their association by combining topics through proper interpretations in a given text [ 12 , 50 , 55 ]. The more commended, the larger the size of the morpheme or word. It can then be seen at first sight, as it were, to visually stress major issues or agendas such as keywords in unstructured documents to extract critical attributes, mainly in big data that manages a large amount of information. Nodes in a semantic network mean words, and links are word-to-word adjacency relationships [ 43 ]. Until recently, network analysis required data structured by nodes and ties ahead of time. The subsequent processes were performed by individual programs, which required plenty of human efforts and time. However, with state-of-the-art technology development, natural language processing is built into data mining programs, which can directly enter unstructured text data and extract words (nodes) in morphological units and create network data encompassing words. It broadens the network analysis horizon with massive unstructured text data [ 12 , 29 ]. Accordingly, a large number of scholars has ripened into a semantic network analysis as a powerful tool of text mining in numerous ways since Rice and Danowski [ 50 ] built a basic framework for network analysis [ 15 , 40 , 50 , 56 ].

The purpose of analyzing text using text mining is very diverse. It is possible to comprehend between the lines in which the document intends to deliver by reassembling the text. Also, by visually grasping the relationship between the main concepts and other keywords in the text, it is easy to understand various networks. Through this, it is achievable to analyze the roles of words and their relationships by recognizing the word associations. One of the most significant benefits in text mining is to analyze the terms both qualitatively and quantitatively [ 11 , 38 , 39 ]. Additionally, it uses to visualize or illustrate the relationship between objects or people in text and topic modeling [ 26 , 43 ]. For this, a large amount of information can efficiently and effectively utilize to generate more comprehensive and extended knowledge, analytical reasoning, and even explorative analysis, which is the final goal of text analysis [ 11 , 12 , 14 , 38 , 39 , 59 ]. With those benefits of this approach, many scholars have discussed various topics with different perspectives using big data. Many scholars and observers have found huge opportunities and tremendous potentials of semantic network analysis with recognizing centrality indicators between words and sub-network structures of words [ 32 , 50 ]. Many of those studies exhibit the possibility of the ongoing development of the semantic networks as a powerful research tool emerging with the big data era’s advent. In particular, semantic network analysis is used in research to study teaching and learning in higher education. Shen and Ho [ 51 ] showed the importance of technology-enhanced learning (TEL) through a semantic approach as an inspired way to improve teaching and learning outcomes in high education. Kim [ 27 ] determined the study trends of music education using the semantic network analysis. Lee [ 31 ] analyzed the research trends in the area of journalism, pursuing the key words of the abstract of research articles published in 2005–2015 through semantic network analysis, then, finally established knowledge system as a result. Besides, Kim et al. [ 29 ] applied semantic network analysis to draw a significant agenda of nursing care service opinions by extracting data from online news and social media data. Recently, Park [ 47 ] took the data of news media and social media to compare the trends from the two different kinds of big data sources to predict leading Korean companies’ sustainability.

Based on those previous studies, this study aims to investigate the most recent research issues and latest trends of teaching and learning in higher education through semantic network analysis. Using a large amount of unstructured text data, we can effectively and efficiently discover trendy insights and directions of future education in teaching and learning and research [ 15 , 52 , 53 , 55 ]. Accordingly, it expects to generate subsequent development of knowledge and intuition to comprehend a new paradigm of future education in general, which is just around the corner. It would be very constructive and beneficial to educators, researchers, academic leaders, and even administrative leaders in higher education.

Proposed research framework

To pinpoint major agendas and trends in teaching and learning of higher education, semantic network analysis, a data mining technique, was used in this study. Accordingly, there is no theoretical framework with hypotheses as the data-driven approach is used in this paper. This data-driven methodology became an extraordinarily capable and promising area. A massive amount of information reserved in electronic and digital records on the internet brings tremendous opportunities and impacts remarkably for knowledge discovery, information extraction, and analytical reasoning in education fields [ 15 , 40 , 61 ]. Thus, this empowers one to extract important algorithmic properties that give powerful intuitions and insights into the structure of networks and graphs [ 38 , 53 , 55 , 65 ]. As previous literature shows, a researcher can collect big data from various sources and platforms. For example, news channels (such as BBC, CNN, ABC, etc.), social media (such as Facebook, Twitter, Instagram, YouTube, etc.), web or internet search engines (such as Google, Bing, Yahoo, AOL, Journal databases or publishers’ databases, etc.), other financial reports (such as financial statements, press releases, conference calls regarding earnings and related information, etc.), and so on [ 11 , 39 , 47 , 53 ]. In this study, the data were gathered for analysis from several distinguishable publishers (Sage Publications, Taylor & Francis, and Elsevier BV)’ web platforms through search engines. Figure  1 shows the proposed framework of this study with a holistic approach.

The proposed framework of this study

This study attempts to determine the most recent research agendas or trends of the leading higher education journals about teaching and learning in 2018 and 2019 through semantic network analysis. As the global trend is changing very fast, this study emphasizes teaching and learning in the last 2 years. For this purpose, this study establishes the following research questions.

(1) What are the main trends or agendas of teaching and learning in higher education in the last two years?

(2) What are the critical attributes of teaching and learning in higher education, and what are the implications of this?

(3) How are the specific sub-domains (topic modeling) of teaching and learning in higher education categorized as future education strategies?

Data and methodology

Data collection.

This study aims to identify the most recent educational trends and predict future directions or shifts by recognizing the main issues of teaching and learning in Higher education. For that, the data collected from 285 research articles of four international ‘top-tier’ journals ranked as ISI/SCOPUS Q1 in this field for 2 years (2018–2019) in the big databases of each publisher’s web platform. The data were selected according to the following criteria using the search engine: ISI/SCOPUS indexed, reputable publishers/open-access journals, and international peer-reviewed journals in Table  1 . Then, semantic network analysis, a powerful and compelling technique in a significant data era, is used to extract patterns or directions with uncovering data-empowered insights. Consequently, 587 unique keywords, 1743 sentences, and 285 paragraphs and documents were identified in 285 abstracts of research articles through NetMiner4’s semantic network analysis program. NetMiner is an eminent application software tool for exploratory analysis and visualization of extensive network data. It also includes several functions and features for data analytics, such as machine learning algorithms, semantic network analysis, etc. It has optimized operations and structure that can efficiently process large amounts of unstructured text data. Accordingly, the study uses this software program, particularly semantic network analysis, which is the most appropriate method to achieve the research questions.

Data cleaning with refinement

Most commonly, data encloses ambiguous and unnecessary contents. Particularly, unstructured text data include meaningless and illogical messy texts such as symbols, colon, emoticons, consonant and vowel data, and even slang and spelling errors. Thus, it is necessary to filter and purify the data through pre-processing for data cleaning and refinement [ 5 ]. In the pre-processed text, the word tab is integrated into the result area. The Word tab provides such information as a list of words, frequency of use, part of speech, which is to be extracted from the unstructured text contents according to the text process settings. After refining, the data are cleaned up, and the quality of the messy data for analysis is improved overall. In this study, messy data were tidy up using Open Refine (used to be called Google Refine), one of the best tools for data cleaning and refinement. Also, the word filtering of data cleaning software in the NetMiner was used for double-checking.

Results and discussion

This study’s first objective is to determine the most studied topics in the field of higher education in 2018 and 2019. For this, the top 20 keywords are selected through semantic network analysis among 587 keywords appearing in 285 abstracts of research papers in four top journals related to the issues of teaching and learning in 2018–2019. The result is as shown in Table  2 .

According to the results, the main ‘top 20’ keywords of teaching and learning in higher education cover the topics of ‘students-centered or student-led learning’ rather than ‘teacher-led’ and that, research, experience, group, development, practice, and engagement are identifiable. Especially, the word ‘student’ composed 7.11% of the total 797 times as a leading keyword showing the highest frequency, and it follows by study, learning, education, research, university, and experience. They are 2.97%, 2.41%, 2.04%, 1.81%, 1.81%, and 1.18%, respectively. In this study, a directional link (Directed Network) based on the ‘Binary Network,’ which does not weigh the connecting line, is created. Here, the frequency of a node is an amount defined for each node and means the number of connection lines of each node that exists as a neighbor. In-degree refers to the number of edges coming towards a vertex in a directed graph; out-degree refers to the number of arcs directed away from the vertex. Although Table  2 shows the keywords in the top 20 ranks, it observes that a few keywords in the top and the other keywords show a great difference in the number of nodes in- and out-degree. Accordingly, for the detailed view at a glance, a word cloud is created to check each keyword’s quantitative importance. Then, it considers the relationship and features of keywords in more detail through network map and topic modeling. Word Cloud is a visualization tool that illustrates the size of letters according to the importance of keywords. Based on that, we can notice the difference between relatively meaningful words, in brief, to understand how much difference is there. In this study, the word cloud node attributes are used to display information as frequency and number of words by entering the maximum number of words as 100. The result of the word cloud is as follows in the Fig.  2 .

Word Cloud of Top 100 keywords

Meanwhile, the word-to-word network is visualized with a network map to understand the data analysis results intuitively. It means that the network data is calculated and arranged according to NetMiner’s layout algorithm’s program. A layout algorithm is a method of calculating where to place nodes to visualize network data. Among the usual methods, Spring Layout is chosen as it can show the connection structure most effectively. Spring layout can be expressed by various algorithms such as Kamada & Kawai, Stress Majorization, and Clustered Eades, Fruchterman & Reingold, GEM, HDE, etc.

Among them, Kamada & Kawai, Stress Majorization, and Eades are chosen as these algorithms fit the need for data analysis and representation, then compare them as preliminary work to consider the number of various cases of subtopic extraction inherent in a given text. In this study, the isolated nodes are shown at the edge of the network map and visualized comparing the ‘Kamada & Kawai,’ ‘Stress Majorization,’ and ‘Clustered Eades’ network maps deselecting isolated nodes. The layout of the analysis results is as follows in the Fig.  3 .

Layout after deselecting isolated nodes (Kamada & Kawai vs. Stress Majorization vs. Eades)

After importing unstructured data, a 2-mode network between words and sentences/paragraphs/documents generated in the program of NetMiner transforms into the keyword–keyword 1-mode system of the research presented in the major journals about teaching and learning. Here’s a look at the detailed layout with keywords from the three core types of network maps in Fig.  4 .

The detailed layout view of network maps

In the NetMiner program, a 1-mode network generates by using word-to-word distance information. In this study, the nearness of two words is calculated, and based on that, a method of creating links between words located close together is used. The number of words includes setting the link generation range between words called ‘Window Size.’ In this study, a maximum of 3 words is included in the link generation range by entering Window Size as 3. Links create between words according to the Window Size set as above, and the two linked words are displayed as Source and Target, respectively, in Table  3 .

The term TF (Term Frequency) describes above indicates how often a particular word appears in a document. It means the higher the value, the more critical the word. However, a term commonly used (for example, do) may have a high TF value even though it is not an important word. To avoid this, we can measure how many documents appear in a text by Document Frequency (DF). In conclusion, TF-IDF (Inverse Document Frequency) provides information to determine how valuable a word is in a particular document based on word frequency and document frequency. The value of the weight, in general, marks the TF-IDF. This Eq.  1 of the TF-IDF score is calculated as follows.

tf i, j  = The number of occurrences of i in j; df i  = The number of documents containing i; N = total number of documents.

Weight is the link frequency of generated word pairs, meaning that the words of ‘student’ and ‘experience’ weight 28, and the word pair appears 28 times in the entire text. # of Sentences/Paragraphs/Documents is the number of sentences/paragraphs/documents in which the word pair appears. In this study, the word pair of ‘learning’ and ‘environment’ appear 26 times, with a weight of 26 and # of documents of 19. Gini Coefficient indicates how concentrated the word pair is intensely in a few sentences, paragraphs, and documents. It is also how evenly it appears in multiple sentences, paragraphs, and documents. A value closer to 1 demonstrates that the more focused it is on a few sentences, paragraphs, and documents, the more important the word pair is. In this case, the criterion for sentences, paragraphs, and documents is a co-occurrence unit selected when creating a 1-mode network.

Lastly, the method of LDA (Latent Dirichlet Allocation), which is a machine learning algorithm, is used to extract subtopics embedded in the text [ 10 , 54 ]. LDA is one of the most popular and influential topic models, a method for analyzing a broad set of unstructured documents. The basic idea is that unstructured documents are represented as a topic distribution where each topic is characterized by a word distribution [ 12 , 54 ].

We can denote p(z❘d i ), p(w❘z i,j ) as the topic distribution for each document i, and the word distribution for a topic allocated to jth word of document i, respectively. In the learning phase, LDA fits p(z❘d i ), p(w❘z i,j ) to a pair of documents (i.e., a document-by-word sparse matrix). Given these distributions, the LDA can generate a new document with the following generative process [ 12 ]:

for jth word in the ith document:

Choose a topic z i, j  ~ Multinomial (p(z❘d i ))

Choose a topic w i, j  ~ Multinomial (p(w❘z i,j )).

For each topic, the top nodes from the main nodes show in Table  4 . Moreover, this table shows the number of nodes and probability included for each topic when the classification labels of nodes in a Subnodeset are assigned to the topic that maximizes the topic proportion from ‘SubNode’.

MainNode (it is a Keyword) and SubNode (it is a Document) show probability information about which topic to associate. Using this method, we can figure out that the more similar the values of the response variables are, the more likely they are to contain compatible subjects. In contrast, the more significant the differences between them, the more likely they are to include different topics. The results of the top 3-topic modeling, according to the # of documents, show as follows. The main subject of the first topic can be interpreted as that practice and learning assessment can be achieved through various activities such as community activities. The second topic suggests that students’ college education can also reach through experience-based classes as staff. Third, the supervisor’s research may indicate that subjects in academic contexts can be addressed. In addition to that, several issues are revealed as uprising topics through the topic modeling of LDA. For example, a system-based education, the results or effects of group activity, how skills and knowledge can play a critical role in performance models. Moreover, how students’ perceptions or feelings can affect learning outcomes, relationships with students, and feedback could be analyzed to develop learning strategies, learning outcomes using cases, and finding teaching strategies through students’ and teachers’ practices.

Implication and conclusion

This study aspires to grasp the latest research agendas and academic trends in teaching and learning with the keywords of major international journals in higher education through semantic network analysis. Consequently, it turns out that this study obviously provides educators, researchers, and academic leaders with data-empowered insights and intuition. The identifiable future agendas, such as a trend-based approach in teaching, learning, and research in higher education, will guide future education with right direction. In this regard, this study outlines the significant implications as follows.

First of all, the world is changing faster than ever; accordingly, it is not easy to follow up on the current educational trends in higher education by mastering whole agendas. Most academic members do not adapt as quickly to keep them up-to-date in the double loop with strategic agility. Given that, this study allows us to identify the current flow of education and educational issues at a glance by leveraging advanced technology and big data. This study focuses on the most recent 2 years of big research data to best support this current situation rather than looking at old panel data such as ten or twenty-year past data. Recent big data generated by reputable publishers’ web database platforms and analytics has become an essential component for the higher education sector. In this way, this research reflects best the current educational situations in teaching and learning of higher education and contribute to preparing for the future with data-empowered intuitions and insights.

Secondly, the semantic network analysis method, a data-driven approach used in this paper, may shed light on the development of new techniques using the machine learning algorithm to get the whole picture of a new paradigm or shift in higher education. It proves that this is a very effective and useful tool to understand the key attributes of current flow and network map in teaching and learning thoroughly by extracting main issues and associated sub-topics. In this regard, this study enhances a more objective view of the actual educational reality and phenomena through big data and machine learning models in the educational sector entirely.

Lastly, traditionally, ‘teacher-centered’ or ‘teacher-led’ education in teaching and learning was the central theme in higher education. A teacher or instructor mainly focused on delivering entire contents from his or her side to students unilaterally. However, now, the paradigm is wholly changed into ‘student-centered’ or ‘student-led’ education. Then, various issues, such as learning, education, research, and experience that follow incidentally, are broken down and spread to diverse subjects. This study firmly reflects in this modern paradigm and educational trend quantitatively and qualitatively. Furthermore, this research result will play a critical role in reshaping educational environments and critical perspectives on teaching and learning in higher education placed on big data.

Although this research suggests several substantial and critical implications, there are still some limitations to be improved below. It will address in future research.

In this study, 285 scholarly articles from the top-ranked international journals related to teaching and learning in 2018–2019 were gathered and analyzed to recognize the most recent research agendas and trends, as it intended, intends to see only up-to-date data given that the trend is changing quickly. However, it is a fact that studies using massive data improve the results in terms of the level of validity and reliability of analysis in general, and it still works in the academic world. Accordingly, future research is needed to collect more data by increasing the number of articles on teaching and learning.

Next, the semantic network analysis or semantic mining technique performs statistical, logical, and rule-based semantic networks analysis quantitatively and qualitatively. However, the research method still leaves room for consideration. Traditional network analysis techniques analyzed physical world relationships simply based on distance (such as centrality), strength, direction, etc. However, recently network analysis has been developed continuously. The giant network is continually forming and flowing subsequent knowledge as large as 1: N link for real-time communication through social media such as Snapchat, Instagram, and YouTube, etc. Thus, it should notice that recent network analysis is very complicated, like the social network web, with a vast amount of data flowing through the network structure. Also, it is necessary to derive and analyze sub-networks aiming to apply them to diverse sectors by reflecting those attributes of a network into the research world.

Lastly, approaching individually or together with a holistic perspective in different education sectors such as law, engineering, business, computer science, or any other studies in higher education is worth doing for future research. The discovery and comparison of the most recent issues or keywords on teaching and learning in each field of higher education will help drive innovation and change in teaching and learning entirely. Furthermore, it encourages academic leaders, more senior or higher management teams, to have effective and proactive leadership by realizing the trend-based education and following up the future directions.

Availability of data and materials

In this study, the data supporting the findings are openly available in publishers’ web platforms of four journals, respectively. These data are derived from the articles of each journal in the public domain.

Algarni A. Data mining in education. Int J Adv Comput Sci Appl. 2016;7(6):456–61. https://doi.org/10.14569/ijacsa.2016.070659 .

Article   Google Scholar  

Angelo TA. Transforming assessment: high standards for higher learning. AAHE Bulletin. 1996;48:3–4.

Google Scholar  

Atteveldt WV. Semantic network analysis: techniques for extracting, representing, and querying media content. Charleston: BookSurge Publishers; 2008.

Ausubel DP. Educational psychology: a cognitive view. London: Holt Rhinehart & Winston; 1968.

Balan S, Rege J. Mining for social media: usage patterns of small businesses. Bus Syst Res. 2017;8(1):43–50.

Banta T, Land J, Black K, Olander F. Assessment in practice: Putting principles to work on college campuses. San Francisco: Jossey - Bass; 1996.

Barrie SC, Prosser M Aligning research on student learning with institutional policies and practices on evaluation and quality assurance, Paper presented at the 11th ISL Conference, Brussels, 4–6 September; 2002.

Bauer M, Henkel M. Responses of academe to quality reforms in higher education: a comparative study of England and Sweden. Tert Educ Manag. 1997;3(3):211–28.

Biggs J. The reflective institution: assuring and enhancing the quality of teaching and learning. High Educ. 2001;41(3):221–38.

Article   MathSciNet   Google Scholar  

Blei D, Ng A, Jordan M. Latent dirichlet allocation. J Mach Learn Res. 2003;3:993–1022.

MATH   Google Scholar  

Bose R. Advanced analytics: opportunities and challenges. Ind Manag Data Syst. 2009;109(2):155–72. https://doi.org/10.1108/02635570910930073 .

Cyram NetMiner (2019). NetMiner Semantic Network Analysis Manual. Cyram. Retrieved from https://www.NetMiner.Com .

Deng R, Benckendorff P, Gannaway D. Progress and new directions for teaching and learning in MOOCs. Comput Educ. 2019;129:48–60.

Drieger P. Semantic network analysis as a method for visual text analytics. 9th Conference on Applications of Social Network Analysis (ASNA). Procedia Soc Behav Sci. 2013;79:4–17. https://doi.org/10.1016/j.sbspro.2013.05.053 .

Doerfel ML. What constitutes semantic network analysis? A comparison of research and methodologies. Connections. 1998;21(2):16–26.

Elton LRB, Laurillard DM. Trends in research on student learning. Stud High Educ. 1979;4(1):87–102. https://doi.org/10.1080/03075077912331377131 .

Entwistle NJ . Complementary paradigms for research and development work in higher education, Conference of the European Association for Research and Development in Higher Education, Rotterdam; 1973.

Executive Core. Future Trends in Business Education. Summer 2015. Executive Core, LLC. 2015; https://www.uniconexed.org/wp-content/uploads/2017/01/Future_Trends-Revised_Final_Report_Executive_Core-August-Jan.12.2016-v2.pdf .

Foster J. Key Trends Influencing Graduate Business Education in 2019. https://www.mba.com/article-and-announcements/articles/why-business-school/key-trends-influencing-graduate-business-education-in ; 2019.

Guri-Rozenblit S. Distance/Open Learning-trends and developments as reflected in recent literature. Stud High Educ. 1991;16(1):83–90.

Heilstra TM, Siguroardottir MS. The flipped classroom: does viewing the recordings matter? Act Learn High Educ. 2017;19(3):211–23. https://doi.org/10.1177/1469787417723217 .

Henderson M, Selwyn N, Aston R. What works and why? Student perceptions of ‘useful’ digital technology in university teaching and learning. Stud High Educ. 2017;42(8):1567–79. https://doi.org/10.1080/03075079.2015.1007946 .

Iniguez, S. (2015). Major trends in business education. MBA Journal. 2015 12:16. https://www.mbajournal.de/major-trends-in-business-education/ .

Jarvenoja H, Naykki P, Tormanen T. Emotional regulation in collaborative learning: when do higher education students activate group level regulation in the face of challenges? Stud High Educ. 2019;44(10):1747–57. https://doi.org/10.1080/03075079.2019.1665318 .

Kelly E, O’Connell P, Smyth E. The economic returns to field of study and competencies among higher education graduates in Ireland. Econ Educ Rev. 2010;29(4):650–7. https://doi.org/10.1016/j.econedurev.2009.11.001 .

Kharlamov A, Gradoselskaya G, Dokuka S. Dynamic semantic network analysis of unstructured text corpora. International Conference on Analysis of Images, Social Networks and Texts (AIST 2017, p. 392–403); 2017.

Kim HM. Analysis of research trends of South Korean music education through semantic network analysis. Korean J Res Music Educ. 2015;44(4):49–68.

Kim ND, SNU Consumer Trend Analysis Center. Trends Korea 2020. Futuristic window (Milaeui Chang): Seoul, Korea; 2019.

Kim M, Choi M, Youm Y. Semantic network analysis of online news and social media text related to comprehensive nursing care service. J Korean Acad Nurs. 2017;47(6):806–16. https://doi.org/10.4040/jkan.2017.47.6.806 .

Lalley C, Houston J, Gasteen A. Gender disparities in horizontal mismatch penalties: an examination of professional’ degrees in the UK (2007–2015). Stud High Educ. 2018;44(12):2265–80. https://doi.org/10.1080/03075079.2018.1485145 .

Lee SJ. Research trends of Korean journalism and communication studies using a semantic network analysis. J Korea Contents Assoc. 2016;16(7):179–89.

Lee S, Choi JH, Kim HW. Semantic network analysis on the MIS research keywords: APJIS and MIS Quarterly 2005–2009. Asian Pac J Inf Syst. 2010;20(4):25–51.

Levine D. Trends in Technology: How we work, live and consume. USA Trends Day Keynote. 2019; https://www.daniellevine.com/ .

Lomer S, Anthony-Okeke L. Ethically engaging international students: student generated material in an active blended learning model. Teach High Educ. 2019;24(5):613–32. https://doi.org/10.1080/13562517.2019.1617264 .

Marginson S, Considine M. The enterprise university: Power, governance and reinvention in Australia. Cambridge: Cambridge University Press; 2000.

Martin F, Wang C, Sadaf A. Student perception of helpfulness of facilitation strategies that enhance instructor presence, connectedness, engagement and learning in online courses. The Internet and Higher Education. 2018;37:52–65. https://doi.org/10.1016/j.iheduc.2018.01.003 .

Maxwell S. Top 3 MBA Trends in 2019 . Hiperpool. 2019, https://hiperpool.com/blog/top-3-mba-trends .

Miner G, Elder J, Fast A, Hill T, Nisbet R. Practical text mining and statistical analysis for non-structured text data applications. 1st ed. Cambridge: Academic Press; 2012.

Mishra M, Mishra P, Somani AK. Understanding the data science behind business analytics(Chapter 5, p. 93–116). In: Big data analytics: tools and technology for effective planning. CRC Press. 2017. https://doi.org/10.1201/b21822 .

Monge PR, Eisenberg EM. Emergent communication networks. In: Jablin FM, Putnam LL, Roberts KH, Porter LW, editors. Handbook of organizational communication: An interdisciplinary perspective. Thousand Oaks: Sage Publications Inc; 1987. p. 304–42.

Nikitina T, Lapina I. Overview of Trends and Developments in Business Education, The 21st World Multi-Conference on Systemics, Cybernetics and Informatics: WMSCI 2017, Vol.2, USA, Orlando. 2017.

Nomuoja JO (2010) Current trends in business education in higher institutions. www.globalacademicgroup.com .

Nulty, P. (2017). Semantic Network Analysis of Contested Political Concepts. International Conference on Computational Semantics (IWCS 2017), Retrieved from http://www.aclweb.org/anthology/W/W17/#6800 .

OECD. Trends Shaping Education 2019 Spotlight 18 Play!. Paris: OEDC Publishing; 2019.

Book   Google Scholar  

Park SY. An analysis of the technology acceptance model in understanding university students’ behavioral intention to use e-Learning. Educ Technol Soc. 2009;12(3):150–62.

Park YE, Alenezi M. Predicting the popularity of Saudi multinational enterprises using a data mining technique. J Manag Inf Decis Sci. 2018;21(1):1–14.

Park YE. Data Empowered Insights for Sustainability of Korean MNEs. J Asian Financ Econ Bus. 2019;6(3):173–83. https://doi.org/10.13106/jafeb.2019.vol6.no3.173 .

Pitan OS, Muller C. University reputation and undergraduates’ self-perceived employability: mediating influence of experiential learning activities. High Educ Res Dev. 2019;38(6):1269–84. https://doi.org/10.1080/07294360.2019.1634678 .

Price L, Kirkwood A. Using technology for teaching and learning in higher education: a critical review of the role of evidence in informing practice. Higher Education Research and Development. 2014;33(3):549–64. https://doi.org/10.1080/07294360.2013.841643 .

Rice RE, Danowski JA. Comparing comments and semantic networks about voicemail. Proc Am Soc Inf Sci. 1991;28:134–8.

Shen CW, Ho JT. Technology-enhanced learning in higher education: a bibliometric analysis with latent semantic approach. Comput Hum Behav. 2020. https://doi.org/10.1016/j.chb.2019.106177 .

Shneiderman B, Aris A. Network visualization by semantic substrates. IEEE Trans Visual Comput Graphics. 2006;12(5):733–40.

Srivastava J, Srivastava AK. Data mining in education sector: a review. Special Conference Issue: National Conference on Cloud Computing & Big Data. 184–190. 2014. https://www.ijana.in/Special%20Issue/C33.pdf .

Steyvers M, Griffiths T. Probabilistic topic models. In: Landauer TK, McNamara DS, Dennis S, Kintsch W, editors. Handbook of latent semantic analysis. New Jersey: Lawrence Erlbaum Associates Publishers; 2007. p. 427–48.

Steyvers M, Tenenbaum JB. The large-scale structure of semantic networks: statistical analyses and a model of semantic growth. Cognit Sci. 2005;29(1):41–78.

Stohl C. European managers’ interpretations of participation: a semantic network analysis. Hum Commun Res. 1993;20(1):97–117. https://doi.org/10.1111/j.1468-2958.1993.tb00317.x

Strauss H, Maisonneuve CDL. The Wage Premium on Tertiary Education: new Estimates for 21 OECD Countries. OECD J Econ Stud. 2009;2009:183–210.

Tierney A. The scholarship of teaching and learning and pedagogic research within the disciplines: should it be included in the research excellence framework? Stud High Educ. 2020;45(1):176–86. https://doi.org/10.1080/03075079.2019.1574732 .

Venkatesh R, Balasubramanian C, Kaliappan M. Development of big data predictive analytics model for disease prediction using machine learning technique. J Med Syst. 2019;43(8):272. https://doi.org/10.1007/s10916-019-1398-y .

Wood A, Galloway RK, Sinclair C, Hardy J. Teacher-student discourse in active learning lectures: case studies from undergraduate physics. Teach High Educ. 2018;23(7):818–34. https://doi.org/10.1080/13562517.2017.1421630 .

Wright J. Methodology review, part 2: text mining and semantic network analysis. 2018. https://jaredmwr.wordpress.com/2018/03/13/methodology-review-part-2-text-mining-and-semantic-network-analysis/ .

Xing W, Tang H, Pei B. Beyond positive and negative emotions: looking into the role of achievement emotions in discussion forums of MOOCs. Internet High Educ. 2019. https://doi.org/10.1016/j.iheduc.2019.100690 .

Yun E, Park Y. Extraction of scientific semantic networks from science textbooks and comparison with science teachers’ spoken language by text network analysis. Int J Sci Educ. 2018;40(17):2118–36. https://doi.org/10.1080/09500693.2018.1521536 .

Zainuddin Z, Halili SH. Flipped classroom research and trends from different fields of study. Int Rev Res Open Distance Learn. 2016;17(3):313–40.

Zaki MJ, Meira W. Data mining and analysis: fundamental concepts and algorithms. London: Cambridge University Press; 2014.

Download references

Acknowledgements

Not applicable.

Not applicable. This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Author information

Authors and affiliations.

Management Department, College of Business Administration, Prince Sultan University, Riyadh, Saudi Arabia

Young-Eun Park

You can also search for this author in PubMed   Google Scholar

Contributions

The author confirms the sole responsibility for this manuscript fully as a sole author for the following: study conception and design, data collection, analysis and interpretation of results, and manuscript preparation. The author read and approved the final manuscript.

Corresponding author

Correspondence to Young-Eun Park .

Ethics declarations

Competing interests.

The author reports no potential conflict of interest.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Park, YE. Uncovering trend-based research insights on teaching and learning in big data. J Big Data 7 , 93 (2020). https://doi.org/10.1186/s40537-020-00368-9

Download citation

Received : 30 July 2020

Accepted : 08 October 2020

Published : 21 October 2020

DOI : https://doi.org/10.1186/s40537-020-00368-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Teaching and learning
• Research trends
• Higher education
• Semantic network analysis

REVIEW article

Shaping the foundation of the supercyberkids learning framework: a comprehensive analysis of cybersecurity education initiatives provisionally accepted.

• 1 Institute of Didactic Technologies, Department of Human and Social Sciences, Cultural Heritage, National Research Council (CNR), Italy
• 2 Institute of Informatics and Telematics, Department of Engineering, ICT and Technology for Energy and Transport, National Research Council (CNR), Italy
• 3 European Cyber Security Organisation, Belgium
• 4 School of Digital Technologies, Tallinn University, Estonia

The final, formatted version of the article will be published soon.

In the rapidly evolving digital landscape, cybersecurity education for children is paramount. The 'SuperCyberKids' project, funded under the EU Erasmus+ programme, aims to address this need by developing a comprehensive educational ecosystem for children aged 8 to 13 and their teachers. Central to this initiative is the SuperCyberKids Learning Framework (SCKLF), which incorporates a game-based approach to enhance engagement and motivation through a bespoke digital learning platform and broader ecosystem. This paper focuses on the second pillar of SCKLF, a detailed survey of 65 cybersecurity education initiatives, offering a practical perspective on the current state of cybersecurity education. The survey covered initiatives within and beyond the European Union, emphasizing the target age group. The analysis of these initiatives provides invaluable insights into the practical application of cybersecurity education and plays a crucial role in shaping the SCKLF. By highlighting the diversity of approaches and strategies in cybersecurity education, this research contributes to a more holistic and applied perspective, ensuring the framework's relevance and effectiveness in fostering digital resilience among young learners.

Keywords: Cybersecurity education, SuperCyberKids Learning Framework, Digital learning, Children's Online Safety, Cybersecurity Education Initiatives

Received: 24 Jan 2024; Accepted: 13 May 2024.

Copyright: © 2024 Manganello, Earp, Fante, Bassi, Fabbri, Matteucci, Vaccarelli, Olesen, De Vibraye, Callaghan and Gentile. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

* Correspondence: Dr. Flavio Manganello, Institute of Didactic Technologies, Department of Human and Social Sciences, Cultural Heritage, National Research Council (CNR), Genova, Italy

People also looked at

Publications — Over 100 years of publishing excellence

• Book Author Resources
• Submit a Book Proposal
• AMS Rights, Licensing, and Permissions
• Open Math Notes
• Frequently asked questions
• Member Journals
• Research Journals
• Translation Journals
• Distributed Journals
• Open Access Journals
• Guidelines and Policies
• Journal Author Resources

Librarian Resources

• eBook Collections
• COUNTER Usage Statistics
• My Subscriptions
• Subscription Information
• Licensing Information

Mathematical Reviews/MathSciNet®

• MathSciNet ®
• Reviewer Home
• MathSciNet ® Subscriptions

Membership — Welcome to your membership center

Join the ams, renew your membership, give a membership, individual membership.

• Member Benefits
• Member Directory
• Reciprocating Societies
• Members in Developing Countries

Institutional Membership

• Domestic Institutions
• International Institutions
• Two-Year Institutions
• Graduate Student Chapter Program

Other Member Types

• Corporate Memberships
• Associate Memberships

Meetings & Conferences — Engage with colleagues and the latest research

National meetings.

• Joint Mathematics Meetings
• Upcoming JMMs
• Previous JMMs
• Special Lectures
• Professional Enhancement Programs (PEPs)

Sectional Meetings

• Upcoming Sectionals
• Previous Sectionals
• Presenting Papers
• Hosting Sectionals

Other Meetings, Conferences & Workshops

• Mathematics Research Communities
• Education Mini-conference
• International Meetings
• Mathematics Calendar
• Short Courses
• Workshop for Department Chairs and Leaders

Meetings Resources

• Suggest a Speaker
• AMS Meetings Grants
• Submitting Abstracts
• Welcoming Environment Policy
• MathSafe – supporting safe meetings

News & Outreach — Explore news, images, posters, and mathematical essays

News from the ams.

• AMS News Releases
• Feature Stories
• Information for Journalists
• In Memory Of

Math Voices

• Feature Column
• Math in the Media
• Column on Teaching and Learning

Explorations

• Recognizing Diverse Mathematicians
• AMS Posters
• Mathematics & Music
• Mathematical Imagery
• Mathematical Moments

Professional Programs — Resources and opportunities to further your mathematical pursuits

Professional development.

• Employment Services
• Mathjobs.org
• BEGIN Career Initiative
• Mathprograms.org
• Mathematical Opportunities Database
• Research Seminars

Institutional Information and Data

• Annual Survey of the Mathematical and Statistical Sciences
• CBMS Survey
• Other Sources of Data
• Directory of Institutions in the Mathematical Sciences
• Professional Directory

Grants & Support

• AMS-Simons Grants for PUI Faculty
• Travel Grants
• Fellowships & Scholarships
• Epsilon Fund
• Child Care Grants

Awards & Recognition

• AMS Prizes & Awards
• Fellows of the AMS

Education — Resources to support advanced mathematics teaching and learning

For students.

• Information for Undergraduate and High School Students
• Research Experiences for Undergraduates (REUs)
• Considering Grad School
• Find Grad Programs
• Applying to Grad School
• What do Mathematicians Do?

For Teachers

• Teaching Online
• Teaching Resources
• Inclusive Classrooms
• Assessing Student Learning
• Education Webinars

For Department Leaders & Mentors

• Information for Department Leaders
• paraDIGMS (Diversity in Graduate Mathematical Sciences)

Government Relations — Advocating for the mathematical sciences

Elevating mathematics in congress.

• Our Mission
• Letters, Statements, & Legislation
• Congressional Briefings

Legislative Priorities

• Federal Issues of Concern
• Federal Budget Process

Get Involved

• Advocacy Resources
• Take Action

DC-Based Fellowships

• Congressional Fellowship
• Mass Media Fellowship
• Catalyzing Advocacy in Science & Engineering (CASE) Fellowship

Giving to the AMS — Your gifts make great things happen for mathematics   Make a Gift

What you can support.

• The 2020 Fund
• Next Generation Fund
• Birman Fellowship for Women Scholars
• JMM Child Care Grants
• MathSciNet for Developing Countries

Create a Legacy

• Make a Tribute Gift
• Create a Permanent Fund
• Establish a Prize, Award or Fellowship
• Bequests and Charitable Estate Planning

Honoring Your Gift

• Donor Stories
• Donor Wall of Honor
• Thomas S. Fiske Society
• AMS Contributors Society
• AMS Gardens

Giving Resources

• AMS Development Committee
• AMS Gift Acceptance Policy

About the AMS — Advancing research. Connecting the mathematics community.

Our organization.

• Executive Staff
• Equity, Diversity, & Inclusion
• Jobs at AMS
• Customer Service

Our Governance

• Board of Trustees
• Executive Committee

Governance Operations

• Calendar of Meetings
• Policy Statements & Guidelines

Proceedings of the American Mathematical Society Series B

Published by the American Mathematical Society since 2014, this gold open access, electronic-only journal is devoted to shorter research articles in all areas of pure and applied mathematics.

ISSN 2330-1511

The 2020 MCQ for Proceedings of the American Mathematical Society Series B is 0.95 . What is MCQ? The Mathematical Citation Quotient (MCQ) measures journal impact by looking at citations over a five-year period. Subscribers to MathSciNet may click through for more detailed information.

• All volumes

Contents of Volume 11 HTML articles powered by AMS MathViewer