Last August, the office of eSafety received their first reports of sexually explicit content generated by students using generative AI to bully other students. And the number has only been growing since.
Last week, deepfakes depicting about 50 female students from a private school in regional Victoria were circulated online. Just weeks earlier, a male student in Melbourne was expelled after he created fake sexual images of a female teacher which were spread around the school. These incidents have raised concerns about the use of AI amongst students to bully others in the Australian school system.
"It is estimated around 90 per cent of all deepfake content is explicit."
said Australia's eSafety Commissioner, Julie Inman Grant.
As deepfakes can be shared with the victim’s social circles, classmates, and family, it can damage reputations and cause serious harm even if there's no blackmail involved. Victims of deepfakes often experience emotional distress, anxiety, depression, and social isolation.
They also face challenges in defending themselves because the manipulated content can be highly convincing and spread rapidly across the internet, making it difficult to mitigate its impact.