Thanks for visiting! Before you read on, please make sure you visit our new website at https://aiassessmentscale.com/. It’s the new home for all of the AIAS related materials, publications, and resources. Thanks!
It’s been over 12 months since the first blog post about the AI Assessment Scale, and a lot has changed, both with the technology and with our understandings of how it impacts assessments in K-12 and higher education across a range of disciplines. The AIAS has been adopted by schools and universities worldwide, and will feature in this year’s UNESCO digital week. It has also been promoted as a tool for assessment transparency by the Australian Tertiary Education Quality and Standards Agency, as well as adapted and translated around the world.
But we (the authors, Mike Perkins, Jasper Roe, Jason MacVaugh and me) acknowledge that there is still a lot of work to be done in understanding the implications of generative and attentional intelligence for assessment and learning. The original AIAS and its subsequent formal version (published in JUTLP) represents a moment in time where educational institutions across the world were reaching for something to help with the immediate problems of AI, such as the perceived threat to academic integrity.
Jason Lodge at University of Queensland and TEQSA refers to these as the acute problems of AI, but we recognise the need for robust frameworks that also tackle the chronic problems brought on in some ways by how we approach ideas of assessment and academic integrity in education.
So we have reflected on all of the versions of the AIAS we have seen across the world in K-12 and higher education. We have sought out critique and engaged with diverse perspectives, from school teachers to students, university lecturers, to disability activists, experts in fields including assessment security, cognitive sciences, and pedagogy.
And over the past months, we have refined and invigorated the AI Assessment Scale to bring it up to speed with our current understandings of generative AI and learning.
Timeline of the AIAS
To look at how we have arrived at this updated version, it’s useful to consider where the AIAS came from:
- It began as a discussion with teaching and learning staff at Edith Cowan University about the need for more than a binary “use or don’t use” solution for AI and assessment during a time of peak AI threat in early 2023.
- The first version, which I will call Version 0, was largely based on my understanding of assessment as a teacher of English and literacy in K-12 and higher education.
- Version 0 was adapted by Mike Perkins, Jason MacVaugh (British University Vietnam), Jasper Roe (James Cook University Singapore) and me to broaden the applicability across other disciplines and make it more relevant to the international higher education context.
- This version, which I will call Version 1, is the most popular and widely used iteration of the AIAS, and was published in JUTLP and a forthcoming peer-reviewed publication based on the pilot study at British University Vietnam. It’s also the version that has been most widely adopted around the world.
In between the academic articles on the AI Assessment Scale, the authors have been taking the work in various directions:
Mike has been presenting the work, seeking more feedback and supporting HE institutions worldwide in implementing the AIAS in various international contexts.
My focus has largely remained on K-12 with occasional forays into higher education, particularly following TEQSA’s call for action plans requiring all Australian universities to outline an approach to artificial intelligence. I produced a free ebook containing examples and activities aligned to the levels of the AIAS, and wrote various other blog posts including discussions of the AI Assessment Scale and cheating.
You can access a free ebook on the AIAS with over 50 activities for the 5 levels by signing up for the mailing list here:
Mike and Jasper have continued to refine the AIAS in different contexts, including a recent preprint on how the AIAS can be reframed for English for Academic Purposes (EAP).
- April 2023: First AIAS post
- December 2023: AIAS preprint published on arXiv
- December 2023: AI Assessment Scale Blog Post
- March 2024: AI Assessment Scale Pilot Project Blog Post
- March 2024: The AI Assessment Scale Pilot Study preprint
- April 2024: JUTLP Volume 21 Number 6: AI Assessment Scale Pilot
- May 2024: AIAS adaptations from across the world
- August 2024: AIAS for English for Academic Purposes preprint
- August 2024: The AIAS and “cheating” blog post
Refining the AIAS
There are many reasons why we chose to update the AI Assessment Scale. The conversations we have had in the past few months have been long, winding, and complex. In a forthcoming article, we will articulate our main areas for revision in detail, including discussing the theoretical underpinnings of the new levels and our decisions and the discussions we have had with other academics along the way. In this blog post, I’m going to outline some of the main changes from my perspective and why I believe it has been necessary to update the scale.
Technology Changes
Even since the publication of the peer-reviewed version of the AIAS in JUTLP in April 2024 (only four months ago), we have seen some significant advances in the technology which have impacted our thinking.
By the time of publication, GPT-4 was well established. GPT-4o and 4o Mini have now been released to the public as the base free version of ChatGPT. Similarly, Claude 3.5 Sonnet is available with limited credits for free users. That the most powerful models available on the market are now free has had huge ramifications. These tools are significantly more proficient in language, code, mathematical reasoning, and across a range of other benchmarks used to assess the quality of LLMs.
Multimodal generative AI has similarly progressed, and in just the past few weeks, we have seen the release of models like Flux, which can generate images almost undetectable to the human eye, something which I proved recently with a “real or fake” deep fake game, where out of almost 3000 plays, by the time of writing, fewer than a dozen people have hit 10 out of 10. Midjourney, another incredibly powerful and realistic photo generator announced on August 23 that it was officially moving beyond Discord to its midjourney.com platform with free credits for all users.
Although OpenAI touted their powerful video generation, Sora, in February, we still have not seen a public release, but models such as Runway and Luma are quickly pulling up in terms of quality, and have public versions available. And audio technologies, including the contentious Udio and Suno and ElevenLabs‘ voice generation models, have established themselves in the past two months.
As educators, we cannot hope to keep up with these technological changes, but we have adapted the AI Assessment Scale to account for them. Most notably, we have removed level four, formerly the “AI + human evaluation” level, and replaced it with the previous level five “Full AI”. At this level, there are no restrictions on how a student chooses to use Artificial Intelligence to approach the task.
The new Level Five is an exploratory level where students, educators, and generative artificial intelligence work together to identify new ways of meeting learning outcomes. As the most recent advice from TEQSA points out, students may well have skills in generative AI that are advanced beyond those of their instructors, and there is no reason not to leverage those skills.
“Engaging in conversations and partnering with students and student groups is critical in developing immediate action. Students bring important perspectives to the discussion about AI as key contributors. In addition, many students are already sophisticated users of these technologies and contribute expert views about how they can and should be used in learning, teaching and assessment.“
The new level five also allows for experimentation with cutting-edge or near-horizon versions of this technology, such as artificial intelligence agents and more advanced multimodal technologies, or the convergence of technologies such as AR/VR and robotics with artificial intelligence.
Pedagogical Changes
We also acknowledge that the ways that people have been teaching with (and sometimes against) AI have developed in the past 18 months in unexpected ways. Universities such as the University of Sydney have invested enormous time and resources in the creation of open access LLM-based applications such as Cogniti, which can be trained on the corpus of an instructor’s materials to create a custom chatbot similar to a GPT or a Claude.
Projects leveraging these tools offer incredible opportunities for educators. The original AIAS was seen by many as a tacit permission for educators to use and experiment with artificial intelligence in the classroom, and we have carried that forward into this new version. Again, the only level which rules out artificial intelligence use is Level One: No AI. We now provide more clarity on why educators might choose that level.
To avoid a shopping list of AI tasks, we retain some of the distinction between Levels 2 and 3, articulating Level 2 as assessments where students may use AI as part of the planning, initial composition, research, or in later years of university courses (once they have established foundational skills) through AI-powered research tools. At Level 3, the focus is instead on evaluation, feedback and the use of AI chatbots, such as custom designed course tutors provided by the education institute or the educator. Students can also use AI at Level 3 for the development of their writing, as long as they are still applying a critical eye to the output.
Whilst some approaches like University of Sydney’s two-lane framework are deliberately broad to acknowledge the reality of education and educators’ understanding of these technologies, we still believe that people need more support in exploring different applications of generative AI in their contexts.
Assessment Security
One of the most significant changes in our revised AI Assessment Scale is our perspective on assessment security and assessment validity. This has been shaped in large part by discussions in Australia through webinars and materials provided by TEQSA and Deakin University’s CRADLE, and in particular, Phillip Dawson, Margaret Bearman, Mollie Dollinger, and David Boud in their recent publication Validity matters more than cheating.
It is a potentially disruptive but absolutely necessary perspective which understands that permitting any use of AI effectively permits all use of AI, and since it is undetectable and sophisticated across domains, the distinction between previous Levels 2, 3, 4, and even 5 is somewhat arbitrary.
In the revised version, we take the stance that assessment security, in the traditional sense, is only possible at Level 1, but that assessment validity is possible throughout. We will discuss this further in a forthcoming journal article.
Stylistic Changes
A surprising amount of time and attention goes into the branding of academic frameworks for both pragmatic and theoretical reasons: research needs to be communicated clearly, made accessible, and be practical.
The first version of the AIAS was successful in part due to its simplicity and the bold use of the red to green gradient. The AIAS at times has been conflated with traffic light approaches suggesting “No AI” means stop and Green Level Five assessments “Full AI” means go. As the authors have pointed out in numerous publications, including the peer-reviewed JUTLP, the stop-slow-go of the traffic light was never our intention.
Still, the red to green colour scheme of the AI Assessment Scale, whilst being fundamental to its adoption, has troubled us for a while. In a recent LinkedIn article, Danny Liu of University of Sydney criticised traffic light approaches, including ones with “yellow-green, or yellow-red in the mix as well”.
We agree.
So, in the new version, we’ve adopted more neutral colours, still easily identifiable (e.g., the “pink level”, the “blue level”), but without the symbolic connotations of stopping or going. We also chose these colours for accessibility purposes. We rejected the idea of a gradient, since it can be difficult for partially sighted people to discern between levels of the gradient if the contrast between levels is not distinct. We further tested our ideas using online accessibility tools and talking to people with colour blindness and low sight to arrive at the five new distinct colours.
Accessibility and inclusion will be another strand of our forthcoming paper.
Although we have kept the table in the same style as the original AIAS, we also acknowledge that the five levels suggest a hierarchy to some people, perhaps in the sense that “more AI is better”, or suggesting that there is a percentage of allowable AI use at a given level (e.g., 25% at level two, 30% at level three and so on).
We understand, however, these are arbitrary distinctions. Depending on the context and the discipline and the type of assessment, different levels of AI use may be appropriate at different times. We have therefore designed a new circular version of the Scale graphic, which suggests that all levels may be treated equally in terms of their merits for assessment design.
Mike, Jasper, and Jason will have their own nuanced perspectives and positions on many of these decisions, and we’ll draw those together in the full article.
Here is the revised AI Assessment Scale:
Our forthcoming paper and the presentations at the UNESCO Digital Week will help to explain our choices further, and provide the theoretical and pedagogical grounding for the revisions. As always, we encourage K-12 and Higher Education educators to both adopt and adapt the AIAS as part of increasing transparency and clarity over the use of AI in learning and assessment.
Thank you to everyone across the world who has contributed to our ideas as we shape and refine the AIAS. We continue to seek feedback from all educators in K-12 and Higher Education!
Get in touch to discuss the AIAS, or anything related to Generative AI professional learning and consulting services:
Want to learn more about GenAI professional development and advisory services, or just have questions or comments? Get in touch:
Leave a Reply to The AI Assessment Scale – Pontydysgu EUCancel reply