ChatGPT and the Future of University Assessment

ChatGPT-3 became freely available on 30 November 2022, causing widespread concern among academic circles. Some have argued that students can now produce written work using artificial intelligence with very little effort, which effectively undermines the value of a university degree. In this article, Kate Lindsay explores various scenarios that illustrate how ChatGPT might influence the very nature of university assessment in the future.

In the past few weeks alone a plethora of articles and guides have emerged to advise how we can approach this new challenge to academic integrity and quality, some of them written by ChatGPT itself, just to prove the point (Cotton, Cotton & Shipway, 2023).

Historically, academia has responded tepidly to technological advancements, and now it looks like history may be repeating itself: In the 1990s the growth of ‘distance’ learning was poised to cause the eventual dissolution of the university and the profession of the university lecturer (Eamon, 1999), which was somewhat mirrored 30 years later during the online pivot of Covid19. Let’s not forget that lecture capture has been deemed a technology that would drive students out of lecture halls and replace faculty with recorded online provision (Fagan, 2021).

All of these concerns have proven unfounded, but ChatGPT might actually be the disruptive force academics have been fearing for decades. That’s because ChatGPT arguably undermines the most significant aspect of a college education: the assessment and evaluation of student achievement. Let’s look at different scenarios.

1. Ban it

Whilst this is a rather a futile exercise, it is an option that we have already seen some schools take (Yang, 2023). Blocking a tool from a network or device is something most students can get around by simply logging into other networks, using other devices. Could we develop detection software, that is trained to identify AI language algorithms? GPT-2 Output Detector Demo has made a start here, identifying patterns of irregularities in written work that might indicate chat bot assistance. 
But GPT is only going to advance, every time we prompt ChatGPT we train it. 

We may well become human-batteries fuelling the machines through constantly feeding them information and data, becoming trapped within an AI generated writing matrix! This detection race is one we will not win. 

Seriously though, plagiarism and cheating is not new, and is already a significant activity across Universities. The QAA recently estimated that one in seven (14%) graduates may have paid someone to undertake their assignment (QAA, 2020). A recent bit of polling also indicated that 16% of students had cheated in their online exams in 2022, and 52% surveyed knew someone who had cheated in their academic assessment that year. A very small percentage, only 5%, had been caught (Alpha Academic Appeals, 2022)AI technology has the ability to make this practice easier and more accessible, but banning it is simply trying to implement an analogue solution to digital problem. And it’s questionable whether it is the technology or our approach to assessment that is problematic.

2. Return to pen and paper exams

Pen and paper exams are most likely to return where there may be less appetite or resource to approach assessment in different ways. Of course exams still exist in many institutions, but increasingly it is recognised that requiring students to be placed in highly pressurised inflexible environments lacks a degree of inclusivity and manifests all kinds of attainment gaps. 

A return to pen and paper exams feels like a step back but it is one some universities are willing to take to thwart the machine (Cassidy, 2023). Sitting online exams under controlled conditions, with lock-down browser technology and the ability to detect whether students are pasting large chunks of text into their authoring interface is another option. But online proctoring software comes with financial and operational challenges, not to mention ethical considerations.

There may be something said though for encouraging more synchronous writing exercises. Johann N. Neem, talking to Inside Higher Ed, says that faculty could find ‘new ways to help students learn to read and write well and to help them make the connection between doing so and their own growth’ (D’Agostino, 2023). An example would be offering opportunities for students to write in class and learn to approach writing as a practice of learning as well as a demonstration of it.

3. Develop AI Literacies as part of Student Assessment

This is undoubtably a good step. Over the past few weeks a range of ideas for engaging, authentic and creative assignments have been shared by the higher education community on how ChatGPT can be incorporated into student assessment and develop critical AI literacies. These range from essay reflection and improvement exercises, to prompt competitions and fact checking. A growing body of guides are emerging – I particularly like “Update Your Course Syllabus for chatGPT” from Watkins (2022), which includes 10 AI-ready assignment ideas.

AI is not going away, it is technology that we are all going to be (and already are) engaging with. Developing AI literacies to enable students to use the technology responsibly and critically, is part of preparing them for the world of work. I’m not on board with a recent statement by JISC that ‘We should really regard them as simply the next step up from spelling or grammar checkers: technology that can make everyone’s life easier’ (Weale, 2023). We have to help students to understand where AI writing tools can support them, where it can enable a better outcome, and what its limitations are. Unlike a spell-checker, there is a lot more room for error – after all ChatGPT does not provide you with an answer to your prompt, it provides you with an output. That is a different thing altogether.

It’s also worth noting that AI literacy is a space where we should be mindful of looming social inequalities and environmental challenges. Improved future releases of GPT are not planned to be free to access, whilst the carbon footprint of training just a single AI model is significant (Heikkilä, 2022). As institutions usually committed at some level to social justice, Universities need to look at where they stand on these matters and support students in navigating them.

4. Assess ‘Humanness’

Like we can not ban AI tools, we can not use them for all assignments. Currently tools such as ChatGPT can not do a number of things that are generally expected from a university student – its ability to cite or critique scientific sources is limited, it can not refer to recent real world events or published material, it can’t demonstrate higher-level thinking, argue or have original ideas. 

The more detail and facts you ask for, the more it falters. Today, one of the best ways for students to prove that they are not a predictive language model is to demonstrate sophisticated thinking, which after all is the purpose of a university education. And we have to ask, if a machine can tell us what we need to know, what is the point in learning it? We need to reevaluate the contexts of what students need to know and how they need to learn.

It is possible that the rapid evolution of ChatGPT can positively increase adoption of assessment techniques that measure learners on critical thinking, problem-solving and reasoning skills. Increased use of oral and video assessment, reflective assignments that ask students to explain their thinking process, developing mind maps, engaging in group projects are other examples of assessment techniques. 

This is something many Higher Education learning professionals have been promoting for some time in a bid to better connect University education with workplace skills, offering more authentic experiences. 

5. Using ChatGPT to support assessment processes

Much that has come out of the sector on ChatGPT and assessment has very much focused on risks of plagiarism or cheating. However, on the faculty side there are opportunities to review how this technology can support the assessment burden that many institutions face. The more sophisticated we make assessment in response to AI, the more difficult it will be for AI technology to support tasks such as marking. However, whilst assessment rubric contains generic criteria around structure, referencing and content topics then there is scope to train AI to support the grading of these elements.  Another possible time saver would be Multiple-Choice Question production. Back in 2018 Donald Clark wrote on the potential for AI to reduce the burden of MCQs which are time consuming and difficult to author and quite often lack the quality needed to make them as robust as they could be. Not only can AI help to generate question banks, but it can enable more forms of open input questions as the technology makes possible the interpretation of answers such as typed words, numbers of short answers. 

In just a few months ChatGPT has laser-focused Higher Education on reviewing the essay as a form of assessment, on the practice of writing, how that will change and why. But we are already moving on. AI is also in the space of media production, not only can it produce art, videos and audio it can adopt voices and faces – our chat bot is evolving beyond written words. It would be naive to think we can stand still. This is a technology that is already two steps ahead of our attempts to contain it in some form of meaningful, solid assessment strategy.

In writing this piece I have avoided the possible option that we ‘do nothing’ in response to ChatGPT. Unless we confront the implications of AI for teaching and learning, and embrace it as a part of our policies and pedagogies to develop critical thinking in an AI world, then we really will start to the lose the value of a University education.

What impact do you think ChatGPT or AI will have on assessments in your discipline or field? Let us know what you think in the comment section below, or on Twitter or LinkedIn.

---

References

Cassidy, C. (2023, 10. Januar). Australian universities to return to ‘pen and paper’ exams after students caught using AI to write essays. The Guardian. https://www.theguardian.com/australia-news/2023/jan/10/universities-to-return-to-pen-and-paper-exams-after-students-caught-using-ai-to-write-essays

Clark, D. (2018, 22. September). Learning Designers will have to adapt or die. Here’s 10 ways they need to adapt to AI. Donald Clark Plan B. http://donaldclarkplanb.blogspot.com/2018/09/learning-designers-will-have-to-adapt.html

Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2023). Chatting and cheating: Ensuring academic integrity in the era of ChatGPT. Innovations in Education and Teaching International, 1–12. https://doi.org/10.1080/14703297.2023.2190148

D’Agostino, S. (2023, 12. Januar). ChatGPT advice academics can use now. Inside Higher Ed. https://www.insidehighered.com/news/2023/01/12/academic-experts-offer-advice-chatgpt

Eamon, D.B. (1999). Distance education: Has technology become a threat to the academy?. Behavior Research Methods, Instruments, & Computers, 31, 197–207. https://doi.org/10.3758/BF03207711

Fagan, J. (2021, 3. Oktober). University of Exeter Lecturers Threaten Industrial Action over Lecture Recordings. TP Transcription. https://www.tptranscription.co.uk/university-of-exeter-lecturers-threatening-industrial-action-over-lecture-recordings/

Heikkilä, M. (2022, 14. November). We’re getting a better idea of AI’s true carbon footprint. MIT Technology Review. https://www.technologyreview.com/2022/11/14/1063192/were-getting-a-better-idea-of-ais-true-carbon-footprint/

QAA.(2020). Contracting to Cheat in Higher Education –How to Address Contract Cheating, the Use of Third-Party Services and Essay Mills. UK:Quality Assurance Agency (QAA)

Watkins, R. (2022, 19. Dezember). Update Your Course Syllabus for ChatGPT. Medium. https://medium.com/@rwatkins_7167/updating-your-course-syllabus-for-chatgpt-965f4b57b003

Weale, S., (2023, 13. Januar). Lecturers urged to review assessments in UK amid concerns over new AI tool. The Guardian. https://www.theguardian.com/technology/2023/jan/13/end-of-the-essay-uk-lecturers-assessments-chatgpt-concerns-ai

Yang, M. (2023, 6. Januar). New York City schools ban AI chatbot that writes essays and answers prompts. The Guardian. https://www.theguardian.com/us-news/2023/jan/06/new-york-city-schools-ban-ai-chatbot-chatgpt

---

This is an abridged version of Kate Lindsay’s article “ChatGPT and the Future of University Assessment,” originally published on her blog on January 16, 2023 under the CC BY-SA license.