Myth or added value? Four teaching-learning effects in a fact check

Experiments form an integral part of pedagogical-psychological research. As “classics”, some of these have received a great deal of attention and become widespread. They are often cited, but are rarely read in the original. As a consequence, the results may be misinterpreted or their significance overestimated. We set out in search of four frequently cited studies whose effects in the teaching-learning context are often considered to be certain.

Dr Fox effect

In an experiment from the 1970s, Naftulin et al. (1973) investigated how a lecture’s rating is influenced by lecture style. A professional actor took on the role of Dr Fox, an allegedly renowned scientist. He gave a charismatic lecture, but its content was full of contradictory statements, false conclusions and pointless references.

The audience gave the lecture a very positive rating and said it had given them food for thought. The authors interpreted this as proof that lecture style has a greater influence on students’ subjective satisfaction with courses than content. With authority and wit, students can be effectively seduced into the illusion of learning.

This realisation became known as the Dr Fox effect. It is often used to prove that student rating is not a valid measure of the quality of courses (understood as the fostering of learning). 

As is so often the case, however, the situation is not so clear-cut. In a replication study, Ware & Williams (1975) not only reviewed the students’ satisfaction but also their learning success. This time, Dr Fox gave six lectures. These differed in content (high-, medium- and low-level) and were each held in an entertaining version and a matter-of-fact version. As in the original study, students gave the stimulating lectures a better rating than the matter-of-fact ones. What was surprising, however, was that even though the level of content was comparable, students also performed better in the test of knowledge during the stimulating lecture.

Peer and Babad (2014) replicated the effects of the original study (Naftulin et al., 1973) again 40 years later: even taking varying test conditions into account (e.g. prior knowledge, supposed status of the lecturer), the students gave a consistently positive rating to Dr Fox’s lecture, which was stimulating but largely empty with regard to content. In addition, Peer and Babad (2014) asked the test subjects about the subjective learning effect. Here it was revealed that only about a third of the participants said that they had learned something in the lecture.

What do these studies show? The Dr Fox effect is not just an illusion. Students can benefit from an entertaining presentation style on the one hand whilst still being able to assess whether they have learned anything on the other. Therefore, the question of the validity of course evaluations cannot be clarified unequivocally by the Dr Fox effect.

Uncanny valley effect

Masahiro Mori, a robotics professor, coined the term “uncanny valley” in 1970. He used it to describe the phenomenon that artificial agents, such as robots, avatars or synthetic voices, can trigger feelings of unease when they appear to be almost, but not completely, human. Mori outlined a non-linear curve of familiarity as a function of human likeness (see Figure 1): with increasing similarity, familiarity initially increases constantly but reaches a critical point shortly before perfect human resemblance, at which even minor deviations, such as a deceptively lifelike prosthetic hand that appears cold when the hand is shaken, are perceived as particularly uncanny (the “uncanny valley”). Mori also suspected that this effect could be intensified by movement (Mori, MacDorman & Kageki, 2012).

In their meta-analysis of 72 individual studies (Hedges’ g = 1.01), Diel et al. (2021) confirmed the existence of the uncanny valley effect. However, the occurrence of the effect is not universal but linked to specific conditions: a particularly strong sense of unease arises as a result of inconsistent cues – when artificial-looking skin texture is combined with highly realistic eyes that look alive, for example (“perceptual mismatch”; Kätsyri et al., 2015).

The quantitative confirmation of the effect and the empirically proven significance of a perceptual mismatch provide practical information, including with regard to the design of virtual characters like those used in online learning environments, for example. With the help of AI, human-like avatars and photorealistic representations can now be created quickly and easily – this does involve the risk of inconsistency, however. So instead of aiming for a risky, almost perfect human likeness, going for a deliberately stylised approach that is not hyperrealistic often proves a safer option and leads to greater acceptance.

Testing effect

The testing effect (or test-enhanced learning) describes the characteristic that the active retrieval of knowledge fosters the long-term retention of learning content significantly better than reading it through repeatedly or simply repeating the information. Both the test format and the timing of the tests influence how strong the effect is (Rummer & Schweppe, 2022). Teachers can provide learners with different types of test, including matching tasks, cloze tests, short-answer tests, multiple-choice tests, free-response questions, creating mind maps or writing short essays.

In a meta-study, Rowland (2014) showed that test formats have varying degrees of effectiveness. Free-response questions foster learning to a greater degree than recognition tests. Feedback after a test also significantly increases learning success. That means that it is worth considering possible feedback options to support students in their learning process. The timing of a test also has an impact on learning success. Longer intervals between learning and testing increase the effect, although even short intervals lead to positive results.

According to the meta-study by Yang et al. (2021), learning success increases if several tests are given. The testing effect is universally applicable. It works at all levels of education, from primary school to university. Furthermore, it does not depend on either age or gender. Regular tests therefore form a key building block for sustainable learning.

And if the test doesn’t go so well?

No problem: forgetting something helps you remember. People who test themselves and make mistakes learn better than people who just practise repetition. The boost in learning comes from active recall, not the test. Even just trying to remember knowledge strengthens the memory, even if mistakes happen in the process.

Dunning-Kruger effect

David Dunning and Justin Kruger conducted experiments to study the self-assessment of personal skills in different social and intellectual areas (Kruger & Dunning, 1999). The level of knowledge was measured in tests on humour, grammar and logic. The participants then assessed their own performance without knowing the actual result. It was found that people who scored poorly rated their own abilities too positively. The authors came to the conclusion that people who only have limited skills in one area tend to overestimate their own abilities (Dunning-Kruger effect). Dunning and Kruger suspect that the problem lies in the metacognitive abilities of the participants. Incompetent people do not usually have the knowledge necessary to recognise their own mistakes or gaps in knowledge (Dunning, 2011). Rather like AI, when they cannot access a knowledge base with regard to a topic, they seem to “hallucinate”. With increasing knowledge and experience, however, the self-assessment of these people improves (Kruger & Dunning, 1999) and, like Socrates, they come to realise: “I know that I know nothing”. Critics of the Dunning-Kruger effect, however, note that the results came about due to the study’s statistical design and that the effect can be generated using randomly generated data (Nuhfer et al., 2016). Furthermore, no other variables are taken into account, such as personality traits or cultural influences.

Nevertheless, Dunning and Kruger’s views should encourage lecturers to foster their students’ self-reflection and metacognition skills. That way, students can identify what they do not yet know (self-reflection) and are able to close those gaps in knowledge (metacognition). In terms of formative assessment, lecturers should set their students a large number of different tasks. This gives them the opportunity to review their own skills on a continual basis. Immediately afterwards, lecturers should give feedback that points out gaps in knowledge and encourages students to consider what they can do specifically to fill those gaps. This fosters the students’ metacognitive skills. In addition, the students’ self-assessment of their own performance should always be facilitated so that realistic self-reflection is possible.

Conclusion

The four effects described demonstrate that it is worth questioning common assumptions about teaching and learning and taking a differentiated look at study results – because even “classics” can turn out to be myths.

If you know of any other effects that should be run through a fact check, please write to us, preferably by E-Mail or LinkedIn. We will then review them in a future blog post.

Birgit Hawelka

+ posts Bio

Dr. Birgit Hawelka is a research associate at the center for University and Academic Teaching at the University of Regensburg. Her research and teaching focuses on the topics of teaching quality and evaluation. She is also curious about all developments and findings in the field of university teaching.

• Birgit Hawelka

  Ward rounds, responsibility and vision: Professor Dr. Martina Müller-Schilling talks about sustainable teaching in medicine
• Birgit Hawelka

  Myth or added value? Four teaching-learning effects in a fact check
• Birgit Hawelka

  In conversation with Plato, Polybius and Cassius Dio – Constitutional debates as role plays and podcasts
• Birgit Hawelka

  Exam diversity instead of a standardised exam: Flexible assessment with specification grading

Regine Bachmaier

+ posts Bio

Dr. Regine Bachmaier is a research associate at the Centre for University and Academic Teaching (ZHW) at the University of Regensburg. She supports teachers in the field of “digital teaching”, among other things, through workshops and individual counseling. In addition, she tries to keep up to date with the latest developments in the field of “digital teaching” and pass them on.

• Regine Bachmaier

  Myth or added value? Four teaching-learning effects in a fact check
• Regine Bachmaier

  Science communication: Connection between research and society
• Regine Bachmaier

  Universities between monitoring and justice. 4 scenarios for the future of higher education

Dr. Linda Puppe

+ posts Bio

Dr. Linda Puppe is a research assistant at the Centre for University and Academic Teaching (ZHW) at the University of Regensburg. She focuses on the topics of innovation in teaching and motivation. Furthermore, she is interested in digital learning environments.

• Dr. Linda Puppe

  Myth or added value? Four teaching-learning effects in a fact check
• Dr. Linda Puppe

  The die is cast: The Rubicon model and its relevance for higher education
• Dr. Linda Puppe

  Learning as sustainable development: Why universities need to impart more than just knowledge
• Dr. Linda Puppe

  Agile – Inspired by the software industry, adapted for higher education

Stephanie Rottmeier

+ posts Bio

Dr. Stephanie Rottmeier is a research assistant at the Centre for University and Academic Teaching (ZHW) at the University of Regensburg. She supports and advises lecturers with regard to the didactic design of lectures and seminars. Her focus here is on the themes of self-regulated learning, particularly the digital organisation of self-learning phases, and students’ motivation to learn.

• Stephanie Rottmeier

  Myth or added value? Four teaching-learning effects in a fact check
• Stephanie Rottmeier

  The die is cast: The Rubicon model and its relevance for higher education
• Stephanie Rottmeier

  From curiosity to fascination: How interest stimulates learning
• Stephanie Rottmeier

  Office hours – more than a meet-and-greet