This is an updated post in the series exploring AI ethics, building on the original 2023 discussion of privacy concerns. As generative AI has become embedded in our daily digital lives from chatbots to smart glasses the privacy implications have grown more complex and immediate. This post explores how GenAI has transformed privacy risks and provides updated resources for teaching these issues across subject areas.
Two years ago, I discussed how AI systems perpetuate biases, scrape data indiscriminately, and operate as “black boxes” that obscure their decision-making processes. Since then, generative AI tools like ChatGPT, Meta AI, and Google Gemini have become mainstream, and with that widespread adoption has come a wave of privacy incidents that illustrate the real-world consequences of these concerns.
The illusion of privacy
One of the most significant shifts in the GenAI era is how these tools create an illusion of privacy. When users type into a chatbot interface, the experience can feel like a private conversation: intimate, confidential, even therapeutic. But this perception is pretty far from the truth.
In August 2024, OpenAI disclosed that it scans ChatGPT conversations for harmful content and, in cases deemed threatening, shares them with law enforcement. The announcement sparked immediate backlash, particularly because it contradicted CEO Sam Altman’s earlier statements suggesting ChatGPT interactions should have privacy protections similar to conversations with “a therapist or a lawyer.” As The Hacker News reported, the lack of clarity about which conversations trigger human review, and the precedent for such surveillance to expand, is incredibly problematic.
Of course, this kind of surveillance isn’t limited to OpenAI. Google’s Gemini warns users not to share confidential information because conversations may be reviewed by human reviewers and retained for up to three years, even after users delete their activity. Unless you’re running a local model on your own machine, whatever you tell a GenAI chatbot is never truly private.
Sharing isn’t caring
The reality of Big Tech’s attitude towards privacy reached a new low in June 2024 when Meta AI users discovered their “private” conversations were being shared with other users. What Meta called a “Discover” feed – obfuscated as usual by their almost unreadable terms and conditions – became what TechCrunch dubbed “a privacy disaster”.
Users had been asking Meta AI for help with deeply personal matters, including medical questions, legal advice, job disputes, even tax evasion strategies. Many believed they were having private conversations or, at most, sharing with friends. Instead, a confusingly labeled “Share” button was broadcasting these conversations to a public feed visible to anyone on the platform.
Malwarebytes also documented some disturbing examples of users’ private thoughts going public. A teacher shared an email thread about arbitration for an unjust job termination, complete with identifying details. Other users discussed medical symptoms, personal confessions, and financial information. For users logged into Facebook, Instagram, or WhatsApp while using Meta AI, these conversations were linked directly to their real identities, creating risks of doxxing and harassment.
Fast Company noted this represented “a slow-motion privacy disaster,” as users unintentionally shared “raw, unfiltered pieces of their lives – far from the curated, polished image we’ve grown used to displaying on social media.”
The problem was a deliberate design choice from a company notorious for its callous attitude towards user privacy. Meta’s privacy policy technically disclosed the feature, but the user interface made it nearly impossible to understand that “sharing” meant public broadcasting.
Privacy as a product feature
The Meta AI incident illustrates a broader issue: companies treat privacy as a feature to be enabled or disabled, not a fundamental right. By October 2024, Meta doubled down on this approach, announcing that conversations with Meta AI across its platforms – including the partnership with Ray-Ban smart glasses – would be used for ad targeting. There is no opt-out; users must simply avoid the service entirely.
University of Washington linguist Emily Bender, co-author of the influential “Stochastic Parrots” paper, told Fortune that Meta is “capitalizing on what she calls the ‘illusion of privacy.’ People often confide in chatbots about things they’d never post publicly, lulled into a sense the AI is a neutral listener. ‘There’s this illusion of privacy, when in fact what you’re doing is you’re serving up this data to a company.’”
Cross-border data and national security
If privacy concerns with American AI companies are troubling, the arrival of DeepSeek – a Chinese AI startup – in early 2025 elevated those concerns to matters of international privacy. When DeepSeek’s R1 model briefly became the most-downloaded app in the United States, regulators immediately raised alarms. The core issue was that DeepSeek stores all user data on servers in China, where national security laws require companies to hand over data to the government upon request. Unlike the simultaneously headline-grabbing debate over TikTok, which centred on potential data access, DeepSeek’s infrastructure made the connection explicit.
Cybersecurity researchers soon established the situation was even worse than geopolitical fear mongering. Feroot Security discovered hidden code in DeepSeek’s web application that directly connects to CMPassport.com, the online registry for China Mobile, a state-owned telecommunications company that the FCC banned from U.S. operations in 2019 due to national security concerns. The code creates a digital “fingerprint” for each user that could potentially track their activity across the web.
European regulators responded swiftly. Italy blocked the app outright while investigating GDPR violations. Ireland, Belgium, the Netherlands, and France launched formal inquiries. South Korea’s Personal Information Protection Commission found that DeepSeek transferred user data, including AI prompts, device information, and network data, to multiple companies in China and the U.S. without obtaining consent or disclosing the practice. The Secretary of the Department of Home Affairs in Australia issued a mandatory security notice forbidding government employees from using DeepSeek.
The incident highlighted a fundamental challenge: in the absence of comprehensive privacy legislations with regards to online traffic and in particular the use of GenAI chatbots, many countries have no systematic way to protect the public from data collection or privacy breaches by foreign AI companies, or, for that matter, domestic ones.
Bystander privacy and ambient AI
Privacy violations also extend beyond the individual user to affect bystanders who never consented to data collection. Meta’s Ray-Ban smart glasses exemplify this new frontier of privacy concerns.
These glasses, equipped with cameras and AI capabilities, capture photos and videos that are sent to Meta’s cloud for processing. The wearer may consent to this data collection, but what about everyone else in the frame? In an article for The Conversation, Associate Professor, School of Journalism and Communication, Carleton University Vicky McArthur said, “what remains unclear [despite Meta publishing guidelines on appropriate use] is the issue of bystander consent and how people who appear unintentionally in the background of someone else’s photos will be used by Meta for AI training purposes.”
The situation worsened in April 2025 when Meta updated its privacy policy to remove the option to prevent voice recordings from being stored. Voice data is now retained for up to a year for AI training unless users completely disable the “Hey Meta” feature, which is essentially a core feature of the glasses. This shift from opt-in to opt-out-or-nothing represents a concerning trend in how companies handle sensitive biometric data.
And for “bad actors” deliberately misusing the hardware, Harvard students demonstrated how easily the glasses could be modified to perform real-time facial recognition, matching faces to names and personal information scraped from public databases. While Meta doesn’t officially support this feature, the potential for such surveillance raises profound questions about privacy in public spaces.
Retroactive consent and training data
Finally, another defining privacy issue of the past few years with GenAI has been the ways in which companies use existing user data to train AI models – often without explicit permission. The legal and ethical framework remains contested, with companies claiming “legitimate interest” while privacy advocates demand explicit consent. I wrote about the copyright implications of this issue in the previous Teaching AI Ethics article, but it also extends to concerns about user privacy.
In May 2024, Meta announced plans to train its AI models on public posts from Facebook and Instagram users in the European Union. Rather than seeking opt-in consent, Meta relied on GDPR’s “legitimate interest” provision and provided only an opt-out mechanism. Privacy advocacy group NOYB filed complaints with 11 European data protection authorities, arguing this approach violated users’ fundamental rights.
The European Data Protection Board responded in December 2024 with guidance clarifying that using personal data for AI training requires a proper legal basis under GDPR. Companies must demonstrate a legitimate interest that is “real, legal, clearly defined and sufficiently concrete”, not just a hypothetical business benefit.
In the United States, where comprehensive federal privacy legislation remains elusive, the FTC has also stepped into the gap. In February 2024, the agency reminded companies that retroactively changing privacy policies to permit AI training without consent may constitute an unfair or deceptive practice.
Case Study: Italy Fines OpenAI for GDPR Violations
On December 20, 2024, Italy’s data protection authority imposed a €15 million fine on OpenAI for multiple violations of GDPR in its operation of ChatGPT. The decision provides a concrete example of how regulators are applying existing privacy law to generative AI.
The Garante’s investigation found that OpenAI:
- Failed to notify authorities of a data breach: When a bug in March 2023 exposed chat history titles and payment information of 1.2% of ChatGPT Plus subscribers, OpenAI notified Ireland’s Data Protection Commission, believing it would inform other authorities. However, since OpenAI hadn’t yet established its European headquarters in Ireland at the time of the breach, Italy considered the notification inadequate.
- Processed personal data without legal basis: OpenAI used personal data to train ChatGPT without establishing an adequate legal justification under GDPR, violating principles of lawful processing and transparency.
- Failed to provide adequate age verification: The company didn’t implement sufficient systems to prevent users under 13 from accessing potentially inappropriate AI-generated content.
The fine represents approximately 20 times OpenAI’s revenue in Italy during the period in question: a calculation that drew criticism from OpenAI, which called the decision “disproportionate” and announced plans to appeal.
Beyond the financial penalty, the Garante ordered OpenAI to launch a six-month public awareness campaign in Italian media explaining how the company collects personal data and how users can exercise their rights under GDPR, including objecting to the use of their data for AI training.
This case demonstrates several key principles for teaching privacy in the AI context:
- Transparency requirements: Companies must clearly explain how they collect and use personal data, especially for novel applications like AI training
- Legal basis matters: Simply having a business interest in using data isn’t sufficient: companies must establish a lawful basis under applicable regulations
- Breach notification obligations: Companies must understand and comply with reporting requirements across multiple jurisdictions
- Special protections for minors: AI systems accessible to children require additional safeguards
The Italian decision also illustrates the uneven global regulatory landscape. While GDPR provides Europeans with clear rights and enforcement mechanisms, users in other jurisdictions often lack similar protections.
Teaching AI Ethics
Each of these articles offers opportunities to explore privacy ethics across the curriculum. Here are some updated questions and resources for various subject areas based on the 2024-2025 articles:
Legal Studies: How does GDPR’s approach to AI training data compare with the emerging U.S. framework? Examine Italy’s €15 million fine against OpenAI and discuss what “legitimate interest” means in the context of AI development.
Computer Science: What technical measures can developers implement to protect privacy in AI systems? Research the vulnerabilities discovered in ChatGPT that allowed data exfiltration and discuss how to build more secure AI applications.
Philosophy/Ethics: When does the societal benefit of AI advancement outweigh individual privacy rights? Debate Meta’s decision to use public user posts for AI training and whether “legitimate interest” provides sufficient ethical justification.
Business/Economics: How do privacy concerns affect the AI market? Analyse DeepSeek’s reception in Western markets and discuss whether privacy protections create competitive advantages or barriers to innovation.
Media Studies/Communications: How do user interface design choices affect privacy? Examine Meta AI’s “Share” button controversy and discuss the ethics of design patterns that obscure privacy implications.
Psychology: What is the “illusion of privacy” and why are users vulnerable to it? Read about how Meta’s ad targeting uses AI conversations and explore how perceived confidentiality affects user behaviour.
Health and PE: How can AI be used in healthcare while maintaining patient privacy? Discuss why Google Gemini warns against sharing medical information and what HIPAA protections (or similar local regulations and industry standards) mean in the AI context.
International Relations/Social Studies: How do different nations approach AI and data sovereignty? Compare responses to DeepSeek across Europe, Asia, and the U.S. and discuss implications for international technology policy.
Want to learn more about GenAI professional development and advisory services, or just have questions or comments? Get in touch:
Leave a Reply