Bias of Student Evaluation of Teaching

18. April 2024

Lehrblick - Lehrportraits

00:00 / 18:20

In today’s podcast episode, we have a conversation with Bob Uttl, diving into the world of student teaching evaluations. We’ll explore what’s possible and, most importantly, discuss the boundaries of what these evaluations can really tell us.

Bob Uttl is a professor of psychology at Mount Royal University (MRU) in Calgary, Canada. He is a cognitive psychologist with main research interests in memory, intelligence, aging, assessment and psychometrics. As a (co-)author of several high impact papers, he examined the validity and biases of SET ratings. Professor Uttl’s SET-related research has been featured in a number of news outlets including Inside Higher Ed, University Herald, The Guardian and elsewhere.

Time Stamps

01:26	Bob, in your work on student evaluation of teaching, you’ve been criticizing the use of standardized instruments for measuring teaching effectiveness. Could you expand what arguments and findings led you to this conclusion?
03:58	So intelligence is the most important factor that influences the learning outcomes. Would you agree?
04:33	However, would it be possible that lecturers might influence the students’ interest in the subject they’re teaching and so they would indirectly be kind of responsible or able to influence the learning outcomes?
05:05	Are there are some other bias that could influence or distort the results of student evaluation?
07:10	That would apply to mandatory courses, I think. They should be poorer rated than courses the students can choose by themselves and by their interest, right?
07:43	In one of your texts, you mentioned a study suggesting that incentives like giving chocolate can influence student evaluation. How would you explain such strange findings? And what psychological factors could be swaying students’ opinions in this particular instance.
10:39	If I summarize for the moment, it can be said that there is no objective method for measuring teaching quality. However, SETs then perhaps measure how well teachers manage to meet students’ expectations in good quality teaching?
11:15	Do students know their grades in Canada or at your University before they are doing the SETs?
11:48	The American as well as the German Sociological Association, suggest using SETs not for summative evaluation but for formative evaluation. They recommend it as an opportunity to gather student feedback, their point of view on different aspects of a course. Would you agree to such scenarios?
13:17	Can we identify and measure teaching behaviors that clearly signal ineffectiveness?
13:38	In your latest text you summarize some of them as an indicator for ineffective teaching. That might be the basis for the deans to talk about what happens in this class, or to identify ineffective lecturers, right?
15:32	Okay, so instead of trying to measure teaching effectiveness, you would suppose to identify more ineffective teaching. So to flip it around.

References

American Sociological Association (2019). Statement on Student Evaluations of Teaching. www.asanet.org/sites/default/files/asa_statement_on_student_evaluations_of_teaching_feb132020.pdf

Deutsche Gesellschaft für Soziologie (2020, September 3). Stellungnahme der Deutschen Gesellschaft für Soziologie zum Umgang mit studentischen Lehrveranstaltungsevaluationen. https://soziologie.de/aktuell/stellungnahmen/news/stellungnahme-der-deutschen-gesellschaft-fuer-soziologie-zum-umgang-mit-studentischen-lehrveranstaltungsevaluationen

Uttl, B., White, C. A., & Gonzalez, D. W. (2017). Meta-analysis of faculty’s teaching effectiveness: Student evaluation of teaching ratings and student learning are not related. Studies in Educational Evaluation, 54, 22–42. https://doi.org/10.1016/j.stueduc.2016.08.007

Uttl, B. (2023). Student evaluation of teaching (SET): Why the emperor has no clothes and what we should do about it. Human Arenas. https://doi.org/10.1007/s42087-023-00361-7