Leon Furze

This is the fourth post in a series on Teaching Writing in the Age of AI. The first post provided an overview of some of the changes we’re facing as the number of AI writing tools increases. Post two covered conversations about academic integrity, and the third post offered some practical advice on teaching students to be critical readers and writers.

In this post, I’ll be exploring the assessment of writing, and why AI is such an apparent threat to the way we currently teach and assess. In putting this article together, I leaned on a couple of resources which are well worth checking out:

• John Warner’s article from December ‘Freaking out about ChatGPT’ presents a level-headed response to writing assessment
• Phil Dawson’s AARE article on “cheating” provides an excellent discussion of why students cheat, and how to address cheating.

There’s quite a bit of discussion here about what constitutes cheating, and ways to build “anti-cheating” approaches into assessment. If you’re more interested in the applications for teaching writing, then jump straight to the practical part by clicking here.

Assessment and “cheating”

Concepts of assessment and cheating seem inextricably linked. The media has been awash with concerns about students using models like ChatGPT to cheat in online exams, bypass traditional assessment methods, and write entire essays.

However, the cheating narrative seems to be largely a media invention that hasn’t taken into account student or educator voice. While some have expressed concerns about the impact of AI on assessment, it’s still not clear whether fears of cheating are reflected by how students actually use the technology.

As for the teachers, many educators have shown a keen interest in exploring the opportunities of AI, as well as its challenges, through professional development and webinars. So where is the fear and speculation about cheating coming from? It appears to be a convenient narrative for headlines, rather than a reflection of the views of educators and students.

To shift the narrative away from concerns of cheating, it’s important to first understand why students might cheat in the first place, and what “cheating” actually means.

Why students cheat

Research into cheating tends to focus on higher education. At a tertiary level cheating is big business, with “contract cheating” identified as one of the major issues. The Tertiary Education Quality and Standards Agency, TEQSA, defines contract cheating as follows:

…when students outsource their assessments to a third party, whether that is a commercial provider, current or former student, family member or acquaintance. It includes the unauthorised use of file-sharing sites, as well as organising another person to take an examination.

https://www.teqsa.gov.au/preventing-contract-cheating/what-contract-cheating-and-methods-reduce-it

It’s such a problem that the Australian Government has passed laws to penalise contract cheating agencies which provide free or paid services such as producing essays and sitting examinations.

ChatGPT and other AI models, however, might do an even better job than the legal system of putting contract cheating agencies out of business. Why pay up to hundreds of dollars for someone to write an essay on your behalf when GPT can do it for free? The release of ChatGPT has forced us to reconsider traditional methods of assessing student writing, and to question whether relying solely on a single artefact, such as an essay or exam, is the most effective way to evaluate students’ learning.

Philip Dawson, Associate Director of the Centre for Research in Assessment and Digital Learning (CRADLE) at Deakin University is more interested in why students cheat than finding heavy-handed ways to stop cheating. In his article for AARE, Dawson suggests that there are many reasons why students cheat, including:

• Inadequate preparation or time management skills
• Pressure to achieve high grades or compete with peers
• Lack of engagement with the subject matter or task
• Perception that cheating is easy or that the risks of being caught are low
• Belief that cheating is a victimless crime or that everyone else is doing it
• Lack of understanding or respect for academic integrity
• Availability of technology and resources that make cheating easier.

Dawson notes that these reasons are not mutually exclusive and can interact with each other in complex ways. Although this research is centred on higher education, the conversation is equally important for secondary schools. In fact, several of Dawson’s points including engagement, understanding of academic integrity, and pressure to achieve high grades should be things we address in spite of ChatGPT.

Interested in other articles like this? Join the list:

The role of assessment design in addressing cheating

Sticking with Dawson’s article for a while, he suggests that there is a “tier list” of approaches to anti-cheating, ranking methods from most down to least effective:

Here’s a further explanation of Dawson’s tiers:

Tiers	Approaches
S (why “S?”)	Swiss cheese (layering multiple anti-cheating interventions), central teams (resourcing cheating experts), amnesty/self-report (students coming forward to confess), programmatic assessment (building best practice into all elements of program design, not just assessment)
A	Tasks students want to do (students less likely to cheat if they enjoy the task), vivas (discussions with students to determine if they did the work themselves), stylometry (tech that compares student assignments to see if they were written by the same person), document properties (signals found in document metadata), staff training (detecting contract cheating with training)
B	Learning outcomes (assessing higher-order outcomes), proctoring/lockdown (using lockdown browsers and other proctoring approaches), open book (exam that allows notes, eliminating one type of cheating), content-matching (text-matching tools to deter copy-paste plagiarism), better exam design (tricks to prevent cheating), face-to-face exams
C	Academic integrity modules (teaching students about integrity), honour codes (students sign document promising to be honest), reflective practice (students reflect on work), legislation (laws banning cheating)
D	Site blocking (banning access to tools like ChatGPT), authentic assessment (application in real-life settings, but with no evidence it reduces cheating)
F	Unsupervised MCQs (multiple choice), bans (banning essays entirely), reusing the same task

Unfortunately, much of the discourse around ChatGPT and education has centred on punitive or restrictive methods such as those found in Dawson’s D-F tiers. The “Swiss cheese” approach of layering methods – drawing on approaches from across the tiers – would seem to be a more productive way to deal with the potential misuse of the technology.

These approaches are useful for addressing cheating in higher education, but they could also be considered in the design of assessment tasks in secondary school. By creating tasks that are engaging, relevant, and aligned with learning outcomes, educators can encourage students to approach the task with greater integrity and motivation.

Teaching and Assessing Writing

So how do we combine the topmost, more effective, tiers of Dawson’s approaches with ways of teaching and assessing writing?

In John Warner’s article he suggested a few ways of preventing students from “cheating” at writing, many of which overlap with Dawson’s ideas:

• Give students learning experiences of intrinsic interest and extrinsic worth so they’re not tempted to cheat.
• Use methods of assessment that take into consideration the processes and experiences of learning, rather than simply relying on a single artefact like an essay or exam.
• Require students to practice metacognitive reflection, asking them to articulate what they have learned and then valuing and responding to what they tell us.
• Change the way we grade and require students to demonstrate the ability to synthesise multiple sources.
• Ask students to bring their own unique perspectives and intelligences to the questions we ask them.
• Create assignments that integrate this technology into the learning.

You can see how tasks which focus on intrinsic motivation align with the A-tier, and considering the entire process of learning overlaps with the “programmatic assessment” of the S-tier. Metacognitive reflections might form part of a viva, or a C-tier reflective response. I’m very excited about the potential for these kinds of approaches in teaching writing.

Here are some practical suggestions for what that might look like in secondary school writing education:

Approach	Example 1	Example 2	Example 3
Intrinsically motivating tasks	Writing a personal essay on a topic of the student’s choice	Creating a podcast episode on a current event of interest to the student	Composing a poem or creative piece that captures a particular emotion or experience
Tasks which value the whole process of learning	Keeping a writing journal throughout the semester to track progress and reflect on strengths and weaknesses	Engaging in peer review and revision workshops with a focus on the writing process rather than the final product	Incorporating low-stakes writing assignments that allow for experimentation and exploration of ideas
Tasks which require metacognitive reflection	Writing a reflection essay on the process of writing an assigned essay, including challenges faced and strategies employed	Recording a video reflecting on the revision process and changes made to an initial draft	Completing a self-evaluation checklist that prompts students to consider their own writing strengths and weaknesses
Assessments which require synthesis of ideas and sources	Composing a research paper that draws upon a variety of sources to develop a nuanced argument	Writing a critical analysis essay that requires students to integrate multiple perspectives on a particular topic	Creating a multimedia presentation that combines text, images, and video to explore a complex issue
Students’ own perspectives	Writing an argumentative essay that requires students to take a stance on a current event or controversial topic	Crafting a personal narrative that draws upon the student’s unique experiences and worldview	Composing a persuasive speech on an issue that the student is passionate about
Assignments that integrate LLM technology into the learning	Collaborating with a ChatGPT interface to create an outline or generate ideas for an essay	Using GPT-3 language models to create a dialogue between historical figures or characters in a novel	Experimenting with AI-generated writing prompts and using them as a starting point for creative writing exercises

Avoiding the arms race

We don’t need to try to “beat” AI, and it’s unreasonable to place the burden on teachers and educators to police the use of these technologies. They will become prolific, replacing or augmenting many of our current generation of technologies including search and word processing.

Engaging in an arms race against AI and falling back on “lower tier” approaches such as blocking, supervised examinations, and legislation may temporarily discourage cheating, but it does not address the reasons why students cheat in the first place.

To end this article, I’d like to suggest how to layer several approaches in a single unit of work (The S-Tier “Swiss Cheese” approach). These would be particularly useful in the kind of unit where a student is required to read a text and produce an analytical essay in response (such as VCE English and EAL Unit 2 OC1, or a HSC Band 6 essay). It has been one of the most talked about assessment styles in the cheating discourse, and I want to explore ways we can move beyond that narrative.

Assessment stage	Approaches
Before unit	Academic integrity modules for students, honour codes, staff training in discussing academic integrity and cheating
During unit (formative assessment)	Intrinsically motivating tasks, synthesis of multiple sources including diverse perspectives on the set text, open book assessments
After unit (summative assessment)	“Mini vivas”, reflective practice, use of AI to assist with essay outlines and drafting, opportunities for amnesty/self-report

Here’s what that looks like integrated into a typical 8-week analytical response unit of work:

Week No.	Suggested Lesson activities
Before unit	Establish the importance of academic integrity by introducing students to academic conventions and standards. Teach students how to reference sources, avoid plagiarism, and use citation tools. Discuss “honour codes” with students.
Week 1	Introduce the text and provide an overview of the unit. Set clear learning outcomes and assessment criteria for the final tasks. Assign an intrinsically motivating task related to the opening of the text, such as a personal/creative writing exercise or a discussion forum.
Week 2	Continue exploring the text through reading strategies, close reading and annotation. Conduct a formative assessment, such as a group presentation or debate, to encourage collaboration and engagement with the text. Use an open book format for the assessment (e.g., a Socratic seminar with pre-prepared responses). Provide opportunities for metacognitive reflections such as “metacognitive journal” or writing journal.
Week 3	Assign a critical analysis task, such as a close reading or a character analysis. Demonstrate text-matching and AI detection tools (e.g. Turnitin or OpenAI’s detection tool) to used detect plagiarism and ensure originality of student work: discuss the limitations of these tools and how students can use them to detect “accidental plagiarism”. Provide students with feedback and opportunities for revision.
Week 4	Continue close reading activities. Conduct a reflective practice session, where students reflect on their writing so far and its development, and receive feedback from peers and teachers. Encourage students to engage in self-reflection on their understanding of the text.
Week 5	Conduct a face-to-face or pre-recorded “mini viva“, where students discuss the writing they have produced so far with the teacher and provide evidence of their learning. Use the viva to assess the authenticity of student work and to continue to discuss academic integrity. (Re)Introduce students to the final essay task and discuss analytical essay writing skills, drawing on their prior knowledge of essay tasks.
Week 6	Assign a research task related to the text, where students must synthesise information from multiple sources and demonstrate critical thinking skills. Students should be encouraged or even required to use AI tools such as ChatGPT, perplexity.io, you.com, and new Bing. Discuss the strengths and limitations of these tools compared to tools like traditional search engines, Google Scholar, and library services.
Week 7	Begin the final assessment tasks, such as the analytical essay. Allow class time for students to plan, draft, and generate ideas. Conduct most of the draft writing during class time. Encourage honesty and integrity, and provide support for students who may be struggling with the demands of the unit.
Week 8	Conduct a final reflective practice session, where students reflect on their learning and development over the unit. Provide students with an opportunity to self-report cheating and seek help and support, for example by using a Google Form or other survey. Require students to submit a written reflection on their learning, and provide feedback and opportunities for improvement. Have students submit the final essay.

In March, I’ll be running a Professional Development for educators on Teaching Writing in the Age of AI. It will cover teaching writing with and without AI, assessment, integrity, and how to incorporate AI into writing units.