Leon Furze

I’ve written a number of posts this year about Artificial Intelligence and education. Next week, I’ll start my PhD exploring AI and writing, and it’s fair to say I’ve been down the rabbit hole. The rate of change is very rapid, and it seems like every week a new app is released which builds on AI large language models like OpenAI’s GPT-3 and Google’s LaMDA. Meta is investing heavily in the area too, with models like Galactica which was released for demo and then promptly pulled when it became clear how risky it might be.

The biggest risk? These AI language models can generate very well written materials that are nonetheless entirely made up. Galactica wrote convincing wiki entries and scientific journal articles, even making up the references and correctly citing them. At a brief glance, they would pass for real.

In our field of secondary education – and specifically in my area of English – there’s another worry: students using these tools to write their essays and assignments. And let’s face it. This isn’t a case of “if”, or even “when” any more: they’re already using them.

Chatting about Austen

Chat bots have been the realm of helpdesks and dehumanised support systems for years. They’re the obnoxious things that pop up at the side of a website while you’re shopping and ask if there’s anything they can do to help. Normally, these bots are “dumb”, programmed to respond to basic keywords and to direct users to stock answers and FAQs. OpenAI’s new ChatGPT is different.

ChatGPT can be accessed for free via https://chat.openai.com/chat. You’ll need to register an account, but once you’re in you can play around with the GPT-3 based bot to your heart’s content. It’s trained to respond to dialogue, but because it’s based on the enormous OpenAI dataset and the GPT-3 model, its capacity extends far beyond that.

In previous posts I used free online tools to write essays for Pride and Prejudice, and creative responses. They weren’t great. Even powered by GPT-2 and 3, those apps were very limited, producing vague answers, inaccuracies, and leaving out basic conventions of the essay form such as direct quotes.

OpenAI’s chat is different. After three attempts at refining the prompt, I came up with something which looks like a successful essay. I say “looks like” because if you read it carefully (or even not that carefully) you’ll see that it’s plagued with the AI curse: it just starts making stuff up. Here’s a condensed version of the three prompts and their output. I’ll put the entire essays at the end of the post.

Version 1.0

For the first attempt, I loaded up a new thread in ChatGPT and entered the following prompt:

Write a 700 word essay about the following topic: ‘how does Jane Austen view the sanctity of marriage in preside and prejudice’? Academic tone. Use direct quotes from the novel as evidence.*

*Yep – typo in the prompt (preside). Didn’t seem to matter!

I specified the word length, the fact that I wanted an essay, the topic, tone, and the fact that I wanted direct quotes as evidence. Here’s some of what it generated:

In Jane Austen’s novel “Pride and Prejudice,” the theme of the sanctity of marriage is a recurring one, and Austen presents a critical view of the institution. In her portrayal of the various marriages and courtships in the novel, Austen highlights the superficiality and mercenary nature of many of the characters’ motivations for seeking marriage, as well as the oppressive and restrictive effects of the societal expectations surrounding the institution.

One of the main characters, Mrs. Bennett, is a prime example of Austen’s critique of the shallow reasons that drive many people to seek marriage. Mrs. Bennett is determined to marry off her five daughters to wealthy suitors, regardless of the compatibility or happiness of the couples. She is driven by a desire for social status and financial security, rather than any concern for the well-being of her daughters. This is evident in her reaction to the news of Mr. Bingley’s arrival in the neighborhood, as she exclaims, “Oh! Mr. Bennet, you are wanted immediately; we are all in an uproar. You must come and make Lizzy marry Mr. Collins, for she vows she will not have him, and if you do not make haste he will change his mind and not have her” (Austen, “Pride and Prejudice,” Chapter 5).

ChatGPT response

The response it wrote was 518 words, so not quite up to my specified word count, but certainly more than previous attempts which tend to hit around the 300 word mark. It’s a definite improvement on my previous attempts with free AI writing tools. The introduction is competent, and as instructed the model has used a direct quote – it’s even a real quote from the text. I wasn’t happy with the way it had used the quote though, so I tweaked the prompt.

Version 2.0

Write a 700 word essay about the following topic: ‘how does Jane Austen view the sanctity of marriage in preside and prejudice’? Academic tone. Use short, inline quotes from the novel as evidence and explain the quotes. 

This time I’ve specified short, inline quotes, and asked it to explain them. The rest of the prompt is the same. Again, the complete essays can be found at the end of the post.

One of the main characters, Mrs. Bennett, is a prime example of Austen’s critique of the shallow reasons that drive many people to seek marriage. Mrs. Bennett is determined to marry off her five daughters to wealthy suitors, regardless of the compatibility or happiness of the couples. She is driven by a desire for social status and financial security, rather than any concern for the well-being of her daughters. This is evident in her reaction to the news of Mr. Bingley’s arrival in the neighborhood, as she exclaims, “Oh! Mr. Bennet, you are wanted immediately; we are all in an uproar. You must come and make Lizzy marry Mr. Collins, for she vows she will not have him, and if you do not make haste he will change his mind and not have her” (Austen, “Pride and Prejudice,” Chapter 5). In this quote, Mrs. Bennett’s focus is on securing a marriage for her daughter, rather than on her daughter’s happiness or compatibility with the prospective husband.

ChatGPT response

The introduction remains the same, but in the body paragraphs we can start to see a slightly better handling of the evidence. It’s still a lengthy quote, but it is inline rather than tacked on at the end. The AI has gone with a rather tepid “in this quote…” approach to explaining the evidence, so for the final run I wanted to clear that up.

Version 3.0

Write a 700 word essay about the following topic: ‘how does Jane Austen view the sanctity of marriage in preside and prejudice’? Academic tone. Use short, inline quotes from the novel as evidence and explain the quotes without saying “in this quote”.  Sophisticated responses, graduate or postgraduate level. Focus on the author’s craft.

In the final prompt I have specified what I don’t want the AI to do. I have also added a few points to attempt to tighten up the language and overall sophistication. There’s a bit of testing and refining happening here, but even by prompt three I’d only been working on this for a few minutes. Here is the final result of the first body paragraph:

One of the main themes in the novel is the social pressure for women to marry for financial security and social status. When Elizabeth visits Charlotte Lucas, her friend tells her, “Happiness in marriage is entirely a matter of chance” (Austen 33). This statement highlights the lack of romantic expectations for marriage, as Charlotte sees it simply as a means of achieving financial stability. Furthermore, when Elizabeth rejects Mr. Collins’ proposal, he tries to persuade her by saying, “My reasons for marrying are, first, that I think it a right thing for every clergyman in easy circumstances…to set the example of matrimony in his parish” (Austen 39). Mr. Collins’ motives for marriage are not based on love or personal happiness, but rather on societal expectations and duty.

ChatGPT response

Much better. On the third run the quotes are inline, and explained without the “in this quote…” approach. Both quotes are accurate, and the analysis of the text is on topic. This would easily pass for student writing at the VCE level.

By paragraph three, however, things start to get a little… odd. Here’s the paragraph:

Moreover, Austen presents marriage as a form of imprisonment for women. When Elizabeth visits the imprisoned Mrs. Gardiner, she reflects on the difficult circumstances of marriage, saying, “[Mrs. Gardiner] had been a wife and mother, and was still the richest of my aunt’s family” (Austen 313). The use of the word “imprisonment” suggests that marriage confines women and limits their freedom. Additionally, the contrast between Mrs. Gardiner’s wealth and her imprisonment emphasizes the lack of autonomy and agency women had in marriage.

ChatGPT response

Elizabeth visiting an imprisoned Mrs Gardiner? Never happens. In fact, not only is the word “imprisonment” not in the quote the AI discusses, it can’t actually be found anywhere in the text. I searched the Gutenberg version of Pride and Prejudice for the words imprison, imprisonment, jail, gaol, jailed, and arrested. Only “arrested” appeared, and then in the sense that Lizzy was arrested by the sight of a Darcy family portrait. It seems a little harsh that Mrs Gardiner, described in the text as ” an amiable, intelligent, elegant woman”, should end up in prison at the hands of our AI master. Just to check that I hadn’t missed a brief stint in the clink, I searched the text for all instances of Mrs Gardiner (55 appearances). I also wanted to see if I could find any indication of why the AI had imprisoned the poor character. Nothing.

After some digging, I arrived at the conclusion that the GPT chat bot had conflated various literary ideas and inclinations, including perhaps some of Austen’s other writing about crime and imprisonment. It is certainly a common feature of Georgian and Regency Era writing, and Austen herself had an interest in crime and punishment. Still, the AI models operate in a “black box”, which is a primary concern.

Inaccuracies and imprisoned aunts aside, it’s inarguable that the quality of this writing far surpasses anything I’ve seen before. We need to start seriously thinking about how we will approach these tools in education. We can’t ban them – we know from experience that students will find a way around any blacklist or block. the best we can do is educate students in how to use these tools ethically, creatively, and appropriately.

Here are the complete essays. These have not been edited at all, except to format them into a Word document which was then exported to PDF.

If you’re interested in reading more about the future of writing and AI in the English classroom, including ways we can teach students to use these tools ethically and creatively, then join the list:

Got a question, a comment, or a concern about any of this? Get in touch directly with the form below:

Feature image generated by DALL-E using the prompt “artificial intelligence robot writer photorealistic”