🔗 Share this article

Tech firms and child safety organizations will receive permission to assess whether AI systems can produce child abuse material under recently introduced UK legislation.

Significant Rise in AI-Generated Harmful Material

The announcement came as findings from a protection watchdog showing that reports of AI-generated child sexual abuse material have increased dramatically in the last twelve months, growing from 199 in 2024 to 426 in 2025.

New Regulatory Framework

Under the changes, the authorities will permit approved AI developers and child protection organizations to examine AI models – the foundational systems for chatbots and image generators – and verify they have sufficient protective measures to prevent them from producing depictions of child sexual abuse.

"Ultimately about stopping abuse before it happens," stated the minister for AI and online safety, noting: "Specialists, under strict protocols, can now identify the risk in AI models promptly."

Addressing Regulatory Challenges

The changes have been implemented because it is against the law to produce and own CSAM, meaning that AI creators and other parties cannot generate such content as part of a testing regime. Previously, authorities had to wait until AI-generated CSAM was published online before addressing it.

This law is designed to averting that problem by helping to halt the creation of those materials at their origin.

Legal Structure

The amendments are being introduced by the authorities as modifications to the criminal justice legislation, which is also establishing a ban on possessing, producing or sharing AI systems developed to generate child sexual abuse material.

Practical Consequences

This recently, the minister visited the London headquarters of Childline and heard a mock-up call to counsellors involving a report of AI-based exploitation. The call portrayed a adolescent requesting help after facing extortion using a sexualised AI-generated image of themselves, constructed using AI.

"When I learn about children facing extortion online, it is a source of intense frustration in me and rightful concern amongst families," he said.

Alarming Statistics

A prominent online safety organization reported that instances of AI-generated exploitation content – such as online pages that may contain multiple images – had more than doubled so far this year.

Cases of category A material – the gravest form of exploitation – increased from 2,621 images or videos to 3,086.

• Female children were predominantly targeted, accounting for 94% of prohibited AI images in 2025
• Portrayals of infants to toddlers increased from five in 2024 to 92 in 2025

Sector Reaction

The law change could "represent a crucial step to guarantee AI products are secure before they are released," stated the chief executive of the online safety organization.

"AI tools have made it so victims can be victimised repeatedly with just a few clicks, giving offenders the capability to create possibly endless amounts of advanced, lifelike child sexual abuse material," she continued. "Content which additionally commodifies victims' suffering, and renders children, particularly girls, more vulnerable both online and offline."

Support Interaction Information

The children's helpline also published details of counselling sessions where AI has been mentioned. AI-related harms discussed in the conversations comprise:

• Employing AI to rate weight, physique and appearance
• AI assistants dissuading young people from talking to safe adults about harm
• Facing harassment online with AI-generated material
• Online extortion using AI-faked pictures

During April and September this year, Childline conducted 367 support interactions where AI, chatbots and associated terms were mentioned, significantly more as many as in the same period last year.

Half of the mentions of AI in the 2025 sessions were related to psychological wellbeing and wellness, including using chatbots for assistance and AI therapy apps.