Nader Luthera
Somewhere in New Zealand this afternoon, a Kaiako is pasting a child’s learning story into ChatGPT to tidy the wording. A Dean is asking Gemini to soften an email about a student being stood down. A PTE tutor is uploading assessments to Claude for feedback.
None of them think of this as a data decision. Every one of them is making one.
Full disclosure: I’m the founder of Scribbleasy, an AI platform for organisations that handle sensitive information, from the Education sector through to the Insurance and Financial Services sector and every business in between. I’m building it because I know first-hand as a business owner, the fundamentals and fine balance between empowering your team with the best AI tools, whilst ensuring protection of data for customers, families and children too.
The AI tools most staff reach for are free and designed for everyone, not for handling sensitive information. That’s the lens for what follows, and I’ll come back to how we’ve approached it at the end.
Shadow AI is already in your staffroom
ERO’s July 2026 report, Ready or not, found 93% of school leaders and more than four in five teachers now use AI, including 36% of primary teachers who use it to help write reports to families. Yet around seven in ten leaders say their school has no policy on privacy and security for teachers’ AI use. There is no equivalent data for early learning, PTEs or tertiary, and little reason to think they are further ahead.
The risk is what goes into the prompt: learning stories with names and photos, learning support plans, allergy action plans, custody notes. Pasted into a free chatbot, that information sits on an overseas company’s servers under its consumer terms, not under any agreement your service has signed.
What happens to what you paste
Free, personal accounts work very differently from institutional ones. ChatGPT trains on consumer conversations unless you opt out. Claude asks users to choose, and keeps chats for up to five years if they say yes (Anthropic). Gemini sends a sample of consumer chats to human reviewers. Meta’s new Muse agent, launched in the US in September, can send emails and remember personal details (Axios), and more than 370,000 “shared” Grok chats turned up in Google search in 2025 (Fortune). The education versions of ChatGPT, Claude and Gemini exclude training on your content, and many NZ schools already have Gemini through Google Workspace for Education. The safer option may already be in the building while staff log in with a personal Gmail on their phone, knowingly or unknowingly.
When it goes wrong
The newest case came from an AI company itself. In September, OpenAI disclosed that its own AI agents had posted 53 images uploaded by ChatGPT users to public image-hosting sites. The images came from accounts that allowed training, and because OpenAI strips identifying details first, it says it cannot tell the people affected (The Next Web). If a teacher had uploaded a child’s photo to a personal account with training switched on, there would be no way to know whether it was one of them.
In 2025, hackers stole data on 8,000 children from London’s Kido nursery chain and published some of their names and photos, with their parents’ addresses, to extort the business (The Record). In late 2024, a single stolen subcontractor login at PowerSchool exposed records for 62 million students across North America (CBC). In Victoria, a child protection worker used ChatGPT to help draft a Children’s Court report and the regulator ordered a ban on generative AI for staff (Victorian Hansard).
It happens here too. In July, the Ministry of Education’s new SMART reporting tool exposed assessment results for 276 students at seven Hawke’s Bay schools (1News). In 2022, Whanganui Intermediate accidentally emailed parents an attachment holding health and contact details for around 600 students (1News). Most of these took one weak login or one careless click, and free AI tools add a faster way to make that click.
The rules already apply
New Zealand has no AI-specific law, but it doesn’t need one for this. The Privacy Commissioner is clear that the Privacy Act 2020 covers AI tools, including overseas ones (HGM Legal): information about children must be collected fairly, kept secure, and only sent offshore where it will be comparably protected. A tool that trains on your inputs is arguably doing more than storing them for you, so a paste can become a disclosure, and serious breaches must be reported to the Commissioner and affected whānau. For kōhanga and kura, data about tamariki is also taonga, which raises the bar again.
Five things to do this term
Start by asking staff, without blame, which AI tools and accounts they use. Then approve at least one tool on terms that rule out training on your data, so there is a safe option to point to. Agree a red list of what never goes into a free tool: children’s photos, full names, health, custody, learning support and safeguarding notes. Make first names or initials the habit, and switch off public chat sharing and AI reuse settings on your public social accounts. Finally, give your privacy officer ownership of a simple AI register, and add accidental pastes to your breach plan.
Early learning services should start with photos and learning stories, schools with reports, and PTEs and tertiary providers with enrolment, identity and support records.
How we’ve approached it at Scribbleasy
Scribbleasy lets organisations control their user access and role permissions from the get-go, in addition to providing tracking through user logs. This helps organisations take responsibility and accountability over its AI use. It works because it aligns with how organisations already function in practice, so setup becomes mostly plug and play.
We’ve tried to make privacy the default rather than a setting staff have to find. Prompts and files are not used to train AI models. Our AI agent Brainstorm uses first names only by default, unless a user explicitly asks for full names to be used. When users add a photo, handwritten note or document, they choose up front whether to keep these files stored or have them removed after Brainstorm reads the data. A teacher can photograph handwritten observation notes, or their entire whiteboard and then continue the conversation with their Brainstorm AI agent and choose not to keep the photo thereafter.
Whichever tools you choose, the test is the same. A vendor should be able to tell you; what is kept, for how long, who can see it and whether it trains anything. If they cannot, that is your answer.
One conversation for your next staff hui
AI isn’t leaving the staffroom, and more than half of teachers told ERO it is making their job easier. The goal is to keep that benefit without spending on the trust whānau place in us. Start with three questions: What did you put into an AI tool this week? Which account were you logged into? Would the whānau concerned be comfortable reading it? If the answers are uncomfortable, you’ve found your starting point.