A few weeks back I spoke with a Victorian secondary principal who was keen to explore artificial intelligence; not to replace teachers or “revolutionise learning”, but to see whether it could make some of the administrative work of running a school less tedious.
Despite not wanting to do anything too spicy, or remotely concerning student data, they described restrictions, a lack of support, and a feeling that professional learning around AI was being actively discouraged. They felt hamstrung: expected to shepherd the school through a world increasingly shaped by AI, but given little room or resourcing to actually understand it.
From a policy standpoint, Victoria’s published rules permit some AI use, but within significant constraints. And in the background of all these conversations, the principal felt they were in a holding pattern waiting for the state to follow other education departments in building an in-house “approved” AI tool.
Across Australia, there are already many examples of these approved, education-flavoured chatbots. New South Wales has NSWEduChat. South Australia has EdChat, developed with Microsoft. Queensland has Corella. Western Australia has ClassmAIte, with another $4.6 million committed to extending its pilot through 2027. In Victoria, we already have VicGov Chat for public servants. The education sector here will certainly follow suit.
There are legitimate reasons for these kinds of tools in an enterprise setting, and many organisations are building in-house to deal with privacy concerns, on-shore data processing, and data storage. In education, there’s an additional justification in providing equitable access, particularly for families who can’t afford commercial subscriptions to AI platforms. And young students don’t need extensive AI exposure before they’ve developed foundational knowledge and skills.
But a centrally provided, department branded, chatbot is not an AI strategy. It’s a deliverable: something a department can launch, point to, and declare progress on. Building teacher (and student) capacity with AI is much harder and more complex than just building an education-friendly chatbot.
In 2024, I wrote Alcohol Free AI, arguing that we should stop pretending these technologies become “safe” when wrapped in education guardrails. We don’t prepare teenagers for the risks of alcohol by handing them alcohol-free beer in class. We build social norms, set boundaries, regulate access, and educate them about the risks. Safeguards are important, but they don’t make the underlying technology harmless.
A secure, private, “safe” chatbot can become a sort of get out of jail free card: a performance of safety in the interests of looking as though we’re doing the right thing, while avoiding the real work.
Are bespoke chatbots the right choice for education departments?
What evidence do we have that these bespoke educational chatbots are the right choice? Very little. South Australia’s external EdChat review in 2024 told us some initial statistics about uptake and interactions, but nothing about improved learning outcomes. The same report presented interactions showing students’ “appreciation” for the chatbot, including statements like “I love you so much” and “Can I talk to u like a friend”.
Given the growing concerns over the use of social or companion chatbots by young people, I don’t see this as a positive.

I’ve also heard many complaints from school leaders across various states that departmental chatbots compare poorly with freely available commercial applications. I haven’t independently tested these closed systems, but any custom built platform performs behind both frontier and mainstream commercial models. I’ve heard directly in schools that students don’t enjoy using them, and teachers prefer to use their own personal accounts on other platforms.
The international evidence is no stronger. Khanmigo was one of the first education chatbots, and its first major randomised trial, run across 18 Tennessee middle schools over two years, found that almost every student tried it and then mostly ignored it. The median student messaged it on about a third of the days they practised, and the modest gains looked the same as Khan Academy without the AI. Sal Khan has called it a “non-event” for most students.
As University of Queensland’s Jason Lodge argues, “the evidence is mixed, and thinner than the marketing suggests”. Closer to home, the public record for NSWEduChat, Corella and ClassmAIte consists of self-reported time savings, principal testimonials and ministerial media releases.
The design and use of AI in education is still important, but I don’t think anyone has cracked it yet. In a large Turkish field experiment, students given unrestricted GPT-4 did worse on later exams, while a “hint-based” version largely avoided the harm. A carefully built tutor at Harvard outperformed an active learning class. Khosravi and colleagues argue that general-purpose tools are built to minimise effort, which is the opposite of what learning requires. But these studies show that pedagogical design is important when it is embedded in teaching. They don’t show that a department-branded chatbot gets used more, or works better, than the tools teachers and students already have.
Shadow AI and surveillance
An approved tool can quickly become the only tool people feel permitted to discuss. NSW says other free generative AI tools are “neither endorsed nor recommended”. Staff can still access most of them, subject to safety requirements; student access on department networks is more restricted. That’s not a blanket ban, but formal permission and professional permission aren’t always experienced in the same way.
That’s how you create shadow AI. Departments lose sight of poor practice, but they may also lose sight of good practice. If teachers can’t openly share their experiments, we shut down one of the best forms of professional learning: informal conversations between colleagues. The University of Melbourne’s TRAINE study, drawing on 665 teacher responses, already describes educators using multiple AI platforms. My ongoing surveys in schools suggest the same, with the majority platform still ChatGPT (and increasingly Claude).
Then there’s surveillance. South Australia’s EdChat can notify schools when student prompts trigger wellbeing filters, as can commercial platforms like school-issued student licenses for Microsoft Copilot. There’s a legitimate safeguarding case, and the department says it doesn’t monitor conversations in real time. But a young person seeking advice about loneliness or mental health is likely to use a platform outside the school’s view. We’ve created an audit trail for the part of the problem we can monitor, but the actual use is very much in the shadows, on personal devices and at home.
The wrong technology
As I argued in the previous article in this series, chatbots are a very narrow interface for technology becoming part of our digital infrastructure. Imagine a teacher securely asking an AI embedded in a learning management system for a report drawing on authorised information from several systems. Or AI connecting timetabling, administration and accounts software, with proper permissions and oversight. That requires infrastructural planning, procurement standards, cybersecurity and professional judgement. So-called agentic AI brings new risks, but offers much greater potential than chatbots to reduce teachers’ professional workload.
I don’t doubt the expertise or intentions of the people building these tools – I know many of the individuals building these platforms personally, and we’re all trying to support students and schools with a complex, constantly changing technology. But every bespoke product needs ongoing maintenance, evaluation and security as the technology changes. These are incredibly costly technologies, and building chatbots that are already out of date by the time they’re released is an expensive experiment for a sector not known for its cash reserves…
I’d invest that money instead in strategic and infrastructural planning, and in professional learning that strengthens teachers as experts in their existing disciplines. After all, we’re not all teachers of AI: we’re English teachers, Chemistry teachers, PE teachers…
Some of the best teaching has always happened without technology. The best use of technology facilitates good teaching; good teaching rarely happens because of technology.
We don’t need to keep trying to build better chatbots.
Want to learn more about GenAI professional development and advisory services, or just have questions or comments? Get in touch:

Leave a Reply