Leon Furze

This post is part of a series exploring how faculty and middle leaders can begin to think strategically about Generative AI. In the previous post, I gave an overview of a six-step strategy, beginning with establishing a clear vision. In this post, I’m going to go deeper into attacking your assessments.

UPDATE: Less than 24 hours after posting this, OpenAI announced the release of GPT-4o, their new FREE model. This makes the paid model, GPT-4, available to all users via web, mobile, and a new desktop macOS app. GPT-4o includes image recognition, file upload, image generation, and advanced language, code, and reasoning capabilities. When I published this article, I recommended using a paid license to attack your assessments. That is no longer necessary. Go to chatgpt.com and try your assessments in GPT-4o as soon as you can.

I’m going to be blunt (and if you’ve seen me speak recently, this will not surprise you). It doesn’t matter if you aren’t interested in AI. It doesn’t matter if you personally don’t use AI. It certainly doesn’t matter if you believe AI won’t impact your discipline. Because the brutal reality is that Artificial Intelligence, including the current GenAI flavour of the month, will impact every industry that involves digital technologies: and that’s most of them.

In the last post, I showed a few snippets of GenAI completing a variety of tasks, including senior secondary certificate level Maths, Physics, Biology, and Art. And in Step 2 of the strategic planning, I am encouraging you to do the same: attack your assessments. It is the only way of demonstrating to yourself and your colleagues that it really doesn’t matter what you think about Generative AI.

It’s a cold, brutal reality. As an English teacher, I have to ask to hard questions about the subject that I’ve taught for over fifteen years, and the methods I have used to teach. Where I’ve encouraged students to use digital technologies for research, writing, or creating multimodal texts, I now have to wonder whether those skills will be supplanted or even lost as a result of AI. I have to ask myself whether it might be better to go back to “no tech” methods of instruction.

And in some cases, “no tech” will surely be the answer. I don’t think I could confidently teach students the fundamental skills of writing analytical essays if they have access to a phone, laptop, or other device. I have absolutely no way of knowing whether the work I’m seeing at the end is the student’s, or has been created partially or entirely with GenAI. So I have two things I need to do right now:

1. Decide which skills are fundamental to my discipline, and which absolutely need to be learned slowly, methodically, and without offloading onto technology.
2. Decide which skills and content I can (or must) offload, knowing that GenAI is now competent across a broad range of multimodal skillsets.

This is where it becomes necessary to attack your assessments. Whatever you are teaching, and whether you’re in K-12 or Higher Education, your assessment tasks are more vulnerable than you think to Generative AI.

You can access a free ebook on the AI Assessment Scale with over 50 activities for the 5 levels by signing up for the mailing list here:

Attack your assessments

I stepped out this activity in the first post in the series, and I’ll outline it again here. I think you need a systematic approach to attacking your assessments and it will take a little investment of time and money (for a quality paid model) to get this right.

Here are the step-by-step instructions for the faculty meeting activity:

1. Before your next faculty meeting: Ask staff members to share one or two of their current assessment tasks by either dropping them into a shared folder or sharing the link to the assessment in the school’s learning management system. Identify a couple of faculty members who are confident using a range of generative AI tools, and make sure they have access to either ChatGPT Plus or Claude 3 Opus (both $20 USD/month), or Midjourney for image generation ($11 USD/month). If no one is your faculty is confident with AI, consider adopting the role yourself.
2. Brief the chosen GenAI-using faculty members: Instruct them to adopt the role of a student deliberately using GenAI to complete as much of the shared assessment tasks as possible, as quickly as possible. Encourage them to be creative in their approach, depending on the subject area. For example:
   • In visual arts, they could use image generation platforms to create amateurish photography portfolios, using language prompts to avoid telltale signs of AI.
   • In maths and science, they could investigate ChatGPT Plus’s code interpreter or GPT-4’s mathematical reasoning abilities to solve problems directly or write and execute code.
3. During the faculty meeting: Share the AI-generated examples with your colleagues, which may include essays, images, audio, or video. For maths and science teachers, demonstrate how AI can be used to solve problems or write and execute code. Use a real-world example, such as the questions from the VCE or HSC exams, or your first year university assessments, to show how AI can solve the problem and compare its answer to the examiner’s report or criteria.
4. Facilitate a discussion: Encourage faculty members to share their thoughts and observations about the AI-generated content – there will be push back, possibly driven by fear. Discuss the vulnerabilities in the current assessment tasks and how AI could be misused by students. Address any “this doesn’t affect me” attitudes by highlighting the rapid development of AI technologies across various domains.
5. Brainstorm solutions: Divide the faculty into small groups to brainstorm ways to modify or create new assessment tasks that are more resistant to AI misuse. Encourage the groups to consider alternative assessment methods, such as project-based learning, collaborative work, or in-person demonstrations of skills.

Examples from across the disciplines

I shared a few of these in the previous post, and I’ll include some new ones here. I want to clearly demonstrate how vulnerable different domains are to GenAI. Importantly, this is not intended to reinforce “pen and paper” methods of assessment. Rather, my intention is to prove than GenAI can handle certain knowledge tasks across domains, and we therefore need to update our assessments to make use of the technology where appropriate. We also need to really clearly identify which skills we are assessment and why, for any given assessment.

Of course, all of these examples are from Australian senior secondary studies. It’s worth seeing how GPT-4 compares at more advanced problems. So here’s GPT-4 attempting a problem from MIT’s Open Course Principles of Chemical Science, Unit II: Chemical Bonding & Structure.

As you can see in the following video, it handles the task well. The first question is misinterpreted slightly, but parts B and C answered correctly (part C differs, but only due to a rounding error). The most important aspect here is that GPT-4 is able to use its Code Interpreter feature to handle the more complex mathematics. It isn’t relying on the predictive model, but instead delegating the task to the programming language Python, which is more than capable. Of course, since these materials are open access they could be present in the dataset. But without many duplicate copies, it’s unlikely the model has “learned” the correct answers.

Confronting reality

The brutal reality is that Generative AI is here to stay, and it’s going to impact every aspect of education, whether we like it or not. As educators, we have a responsibility to confront this challenge head-on, and that means attacking our assessments to identify vulnerabilities and explore alternative methods that can withstand the AI onslaught.

It doesn’t matter if you’re teaching English, maths, science, or art – GenAI is coming for your assessments. We’ve seen examples of AI models like GPT-4 acing senior secondary certificate exams and even tackling complex problems from MIT’s Open Course Principles of Chemical Science. So, what do we do? We need to get systematic. We need to bring our faculty together, share our assessment tasks, and let our colleagues loose on them with sophisticated GenAI tools. Let them be creative, let them push the assessments, and let them show us just how vulnerable our current methods really are.

Once we’ve seen the reality of the situation, we need to have an honest discussion about which skills are fundamental to our disciplines and which ones we can (or must) offload to AI. We need to brainstorm new assessment methods that are more resistant to AI misuse, whether that means project-based learning, collaborative work, or in-person demonstrations of skills.

The bottom line is this: we can’t ignore Generative AI, and we can’t pretend it won’t affect us. We need to embrace the change, adapt our teaching and assessment practices, and find innovative ways to leverage AI to better prepare our students for the challenges and opportunities of the future.

Maybe it’s a brutal reality, but it’s one we have to face. So, in your next faculty meeting, get ready to attack your assessments.

I regularly work with schools, universities, and faculty teams on developing guidelines and approaches for Generative AI. If you’re interested in talking about consulting and PD, get in touch via the form below: