NZ is criminalising sexualised deepfakes – banning apps that make them should be next

New Zealand is changing the law to make sexualised deepfakes a crime. But this alone may not be enough to counter the rise in AI-generated fake sexual material.

This week the Deepfake Digital Harm and Exploitation Bill is set to go through its first reading, with support across the political spectrum. The amendment will make creating, sharing or selling sexually explicit deepfakes without consent a criminal offence.

It comes in response to the rapid spread of such material, including the rollout of Elon Musk’s Grok AI chatbot on X, which people used to digitally undress women and girls and generate potentially three million sexualised images.

New Zealand is not alone in confronting the problem. The United Kingdom, Australia, South Korea and the United States have already introduced or expanded laws to criminalise creating and sharing of non-consensual deepfakes.

Criminalisation is an important first step and brings New Zealand in line with developments elsewhere. But stemming the tide of sexualised deepfakes will also require regulation of the technology itself.

Deepfakes mostly target women

Deepfakes are AI-generated images, audio or video designed to make it appear that a real person said or did something that never actually happened.

With image-based sexual abuse, this usually means using AI to create convincing but fake sexual material of a person without their consent.

Sometimes, technology is used to manipulate ordinary photos pulled from social media into explicit imagery. Other AI systems can generate entirely fabricated sexual content from a text prompt alone.

The abuse is overwhelmingly directed at women. One widely cited study found 98% of deepfake videos online were pornographic and mostly targeted women.

For victim-survivors, the harms are significant regardless of whether the image is created from a real original or completely fabricated. People report humiliation, fear, anxiety, loss of control and violation of sexual autonomy.

Gaps in New Zealand law

The new bill is designed to fill gaps in the current law.

At present, New Zealand does not have a criminal offence specifically directed at sexualised deepfakes. Existing laws may apply, but they were not designed for AI-generated abuse.

The Harmful Digital Communications Act 2015 criminalises posting harmful digital communications and has already been used in at least one prosecution involving a sexualised deepfake. However, the offence requires proof that the defendant intended to cause serious emotional distress and that this actually resulted.

Those requirements can be difficult hurdles for victim-survivors of image-based sexual abuse.

A new offence introduced into the act in 2022 sought to address this for so-called “revenge porn”, making it a crime to share an intimate visual recording without consent.

This sat alongside existing offences in the Crimes Act. The concept of an “intimate visual recording” was introduced in the early 2000s to deal with covert filming with hidden cameras.

Parliament was responding to conduct known as “upskirting” and “downblousing” – forms of abuse overwhelmingly targeting women and girls in spaces where they had a reasonable expectation of privacy, such as bathrooms or changing rooms.

The law therefore focused on whether a person had been secretly recorded. But deepfakes complicate that framework because no recording may have occurred at all. This means current law may be clearer when an intimate image is real than when it is entirely fabricated.

Beyond criminalisation

The deepfakes bill attempts to remove that uncertainty by expanding the legal definition of an “intimate visual recording” to include images that are “created, synthesised or altered”.

But the bill also reflects a broader pattern in New Zealand’s response to image-based sexual abuse: the law tends to evolve only after new technologies expose gaps in existing protections.

First, it was hidden cameras and covert recordings. Then it was revenge porn. Now generative AI. Criminal law has been reactive and is not future-proofed enough to cover new technology-facilitated sexual harm.

The rapid development of new technologies means criminalisation alone is unlikely to stop the spread of sexualised deepfakes.

The tools to create deepfakes are cheap, fast and increasingly easy to access. A recent investigation by the Tech Transparency Project identified dozens of “nudify” and face-swap apps available through both the Apple and Google app stores. These apps can generate sexualised images from ordinary photographs within seconds.

Despite app store policies prohibiting sexually explicit or degrading content, many of these tools remain readily accessible, often disguised as image-editing apps.

Addressing regulation of high-risk apps

Generative AI systems rely on enormous datasets scraped from the internet, frequently including images of women and girls used without their knowledge or consent.

The result is that women’s bodies are increasingly becoming both the raw material for AI systems and the targets of abuse generated by them. That is why New Zealand needs to think beyond criminal law and address regulation.

Australia is moving to ban nudification apps and websites. So is the United Kingdom and the European Union. New Zealand should follow suit.

More broadly, New Zealand should consider a regulatory framework for high-risk AI systems, particularly for technologies capable of generating non-consensual sexual content.

This could include mandatory safety guardrails in image-generation systems, stronger obligations on app stores and platforms distributing these tools, and transparency requirements around AI training data.

The Deepfake Digital Harm and Exploitation Bill is an important step forward and parliament should pass it. But if New Zealand wants to meaningfully address image-based sexual abuse in the age of generative AI, criminal law cannot be the end of the conversation.