Dr Leon Furze

This is the fifth in a series on generative AI strategy for faculty leaders and teaching and learning teams. If you’ve been following along, you will have:

1. Prepared yourself with local and national school guidelines on AI
2. Attacked your assessments
3. Tried some small experiments
4. Started updating your major assessments with the AI assessment scale

You should by now have collected enough data from staff and students to begin making some judgments and to communicate those findings with your colleagues and leadership.

One of the biggest unknowns of generative artificial intelligence in education (or even more broadly, technology as a whole) is whether or not it actually has a positive impact on learning. Unfortunately, there are decades of studies which suggest that as far as edtech is concerned, the opposite is true.

AI might have a neutral or even negative impact on learning due to the risks posed by digital technologies and devices on student’s attention spans. Anyone who has been in a classroom knows how easy it is for students (and ourselves) to become distracted. Despite all of the hype and rhetoric around AI revolutionising education and the glossy sales pitches from companies like OpenAI and Google, I will not commit to saying that AI will have a positive impact on learning until I see some evidence.

I think it’s vitally important that educators on the ground carry out these small experiments and evaluate the efficacy of generative AI for themselves, so that we’re not simply being told by technology companies that AI will “revolutionise education”.

You can access a free ebook on the AIAS with over 50 activities for the 5 levels by signing up for the mailing list here:

Evaluate Your Findings

In the previous steps in this strategic planning, you should have carried out some small experiments (firing bullets, then cannonballs) and began to address your major assessments, perhaps aligning them with the AI Assessment Scale. I encouraged you to collect feedback from students and/or teachers, depending on the kinds of experiments you carried out. And it’s now time to evaluate all of that feedback.

Since this is not a peer-reviewed study, we can perhaps be a bit fast and loose with the methods and use generative artificial intelligence to speed the process along. If, like I suggested in your assessments, you subscribed to Claude from Anthropic, this would be my recommended tool for going through your data. However, by the time you’re reading this, you may well have access to GPT-4o for free, in which case, you can certainly try that.

The intention is to gather all of your data in one place, and then crunch it with AI. If you sent out student surveys or teacher evaluations, put all the open-ended question notes into a single CSV or Excel file. (If you used Microsoft Forms or Google Forms, you could also export them as PDFs and merge them together but de-identify the results first) Because Claude can take a large amount of text in its context window, there shouldn’t be any issues even if you have sampled a large group for feedback.

If you followed my advice in the previous posts, your feedback will likely have long-form responses from staff or students. And we’re going to use the AI to synthesise those responses. For the numerical responses, you can obviously just use the built-in graphs in Microsoft or Google Forms.

In the video below, I’ll demonstrate the process. I’m using feedback from participants in one of my courses because obviously, I’m not evaluating my faculty’s use of AI. And importantly, I have de-identified the results before using them with AI, deleting the names and email addresses collected during the survey. I’ve demonstrated this in both GPT-4o (MacOS desktop version) and Claude for comparison:

Watch the video and follow the process with your own data. If you have collected information from students and teachers, then do it separately for each. Remember to de-identify all of your data before using any GenAI platform. Prepare this evaluation for your next faculty meeting for discussion.

Discuss Your Findings

Make no assumptions about whether your findings will be positive or negative. As I said earlier, I can’t absolutely promote the use of AI in education is beneficial until I’ve seen that it has a positive impact on things like student learning or teacher workload. Everything up until that point is just conjecture and my own opinion.

In the spirit of confronting the brutal reality, discussed in the first post in this series, don’t try to gloss or tidy up the results. If your results are mostly negative (for example, teachers talking about the learning curve being too steep, not having time to use or learn to use the technologies, or finding them a distraction in the classroom), then address those issues up front. If student feedback indicates that they don’t enjoy interacting with chatbots, that they feel they are not learning, or it raises any other issues, confront these as well. Do so with an open mind. Ask questions like the following:

• Did the way we conducted these experiments contribute to the negative outcomes?
• Have our perceptions or discussions of these technologies coloured the output?
• Is my personal bias as a faculty leader contributing to these results?
• Are these results due to technological or pedagogical issues?
• Could we try another experiment? Might it have different results?

If you get overwhelmingly positive results, I would still encourage you to ask the same kinds of questions. Reflecting on your processes whilst carrying out your small experiments and gathering that data is an important step before we move on to communicating your findings.

Whether your responses are positive or negative, it’s worth acknowledging that the broader encroachment of these technologies into education will not be slowed. You might unearth some serious problems with the ways that students are interacting with chatbots, but that will not stop them from interacting with chatbots. The next step is to clarify and update your earlier draft vision and decide how you will communicate.

Clarify Your Vision

In the first step, you set out to draft a vision for the faculty based on school, local, national, and international guidance on incorporating generative AI. Now, having been through the process of testing it out in your discipline with your staff and students, you should be able to refine that. Perhaps you found that despite your best efforts, using generative AI really didn’t improve the quality of students’ work or their understanding. Or maybe you found that encouraging teachers to use generative AI reduced their workload significantly. Incorporate these findings into your original vision. For example, if your vision started off as something like this:

For students in our faculty to ethically and appropriately use generative AI tools

Sharpen that vision up to something like this:

Guide students through ethical and appropriate use of generative AI technologies whilst prioritising technology-free instruction for fundamental skills

Or if your vision began like this:

To explore ways that generative artificial intelligence can reduce teacher workload in our faculty

You might make it more specific by incorporating some of your findings, like this:

Incorporate ethical and appropriate automations into planning and communication processes for all staff in our faculty to reduce workload

Once you have your updated vision, it’s time to pull everything together and get ready to communicate your strategy.

Communicate

As the faculty leader, you’re now going to develop a generative AI strategy one-pager. This needs to be simple and succinct because this might be communicated to school leadership (including assistant principals and principals) who have not been part of this strategic planning process. Or you may be communicating it to other faculties who have not yet been through the process themselves. You might also be communicating it to parents or external stakeholders like the school or university board.

Again, you can use generative AI to help out at this stage. But I would encourage you to put all of the materials on the table for thorough discussion with your human colleagues before handing the synthesis over to the AI. Here’s how I would suggest you write your one-pager:

1. Gather up all of your materials, starting with the initial documentation that you reviewed (international, local, and school guidelines) and your draft vision. Quickly revisit these ideas and see if there’s anything you’ve missed, or drifted away from. For example, if you’re in an Australian K-12 context, are you still adhering to the core principles of the Australian Framework?
2. Gather your reflections from attacking your assessments and updating your assessments. Write a short statement describing how you identified vulnerabilities in your existing assessments and how you have experimented with updating your assessments accordingly.
3. Write a brief statement about your small experiments, whether they were for staff or students. Try to summarise your findings as concisely as possible in a paragraph or dot points.
4. Write out your updated vision and make sure everybody is comfortable with it. (Although please, for the sake of everyone in the room, don’t spend an hour debating synonyms.)
5. Synthesise your observations about assessment and your findings from your small experiments into a set of guiding principles or strategic directions which support your vision.
6. Create a one-pager with the vision, the strategic directions, and (if you feel it necessary) a brief explanation of the process by which you reached these conclusions. See the example below.
7. Last of all, communicate this with your colleagues, community, and students. You’ll probably go up the line first to the executive team to get their approval, and then perhaps communicate horizontally to other faculties, and finally to students, parents, and the broader community.

I’ve run parent information sessions with dozens of schools and thousands of parents, and I can safely say that there is an incredibly high level of interest from parents who want to know if schools are addressing these technologies and their implications for teaching and learning. By going through this faculty strategic planning process, you’ve demonstrated your commitment to doing so.

Example English Faculty GenAI Strategy One-Pager

English Faculty AI Vision and Strategy

Vision: To understand the potential of generative AI to enhance English teaching and learning in ways that are safe, ethical, inclusive and aligned with the Australian Framework for Generative AI in Schools.

Key Principles and Directions:

1. Educate students at appropriate levels

• Educate students about how generative AI works, its capabilities and limitations
• Prioritise the instruction of critical thinking to evaluate AI-generated content and identify potential biases
• Integrate AI tools into the English curriculum in age-appropriate ways, guided by national regulations

2. Use AI to support teacher expertise

• Explore using AI to assist with curriculum development and differentiated instruction
• Maintain teacher control over instructional decisions and student evaluation, particularly for high stakes assessments
• Provide ongoing, subject specific training and support for staff on effective AI use in English

3. Adapt assessments for generative AI

• Review all existing assessments and identify ways students could misuse AI
• Experiment with varied assessment types that measure higher-order skills to which AI does not apply
• Clearly communicate AI use policies in assessments and evaluate the impact of AI use through ongoing measurement of student data

4. Ensure accessibility, fairness and student wellbeing

• Provide equitable student access to AI technologies used in English
• Monitor AI use and impacts to prevent harm, discrimination or privacy breaches
• Educate students on responsible AI use, including copyright and attribution

5. Open experimentation, evaluation and dialogue

• Continue to conduct small trials of AI in English to guide faculty understanding
• Actively monitor developments in generative AI and iterate on our approach, whether that means increasing or decreasing AI use
• Engage staff, students and parents to communicate vision and incorporate feedback

This strategy was informed by an extensive review of AI in education guidelines, consultation with staff and students, and trials of AI technologies in English classes. While still a work in progress, it provides an initial roadmap for putting our AI vision into practice in a thoughtful, measured way. We will continue refining our approach through ongoing experimentation, evaluation and stakeholder engagement.

Reflect and Review

This is the final stage of the strategic planning, but it doesn’t really warrant an entire separate article because you’ve already done most of the hard work. You now need to set some regular checkpoints for reviewing the strategy and continually evaluating its effect. The moment you identify that something is not having a positive impact on teaching and learning, stop doing it. There are more than enough ineffective processes in the education system without us pouring AI special sauce over the top of them.

I would encourage you to continue in the spirit of experimentation, occasionally bringing in new small experiments (firing a few more bullets). And when you identify your targets, whether it’s student engagement, learning, teacher workload, or something else, lean in and see if you can sharpen your faculty strategy.

Most importantly, share what you are learning with your colleagues and the community. Share it with me through the contact form on this website, by email, or on LinkedIn. Nobody in education knows the answer to whether these technologies will have positive, neutral, negative, or mixed impacts on education. The more of us that carry out these small experiments and share what we find, the better off everyone will be.

Over the coming weeks, I’ll be turning this series into an online course to guide faculty leaders through the process. Make sure you join the mailing list to stay up to date on those announcements.

I regularly work with schools, universities, and faculty teams on developing guidelines and approaches for Generative AI. If you’re interested in talking about consulting and PD, get in touch via the form below: