‘No one verified the evidence’: Woman says AI-generated deepfake text sent her to jail
By Chad Pradelli and Cheryl Mettendorf
Click here for updates on this story
PHILADELPHIA (WPVI) — Courts are now facing a growing threat: AI-generated deepfakes.
Melissa Sims said her ex-boyfriend created fake AI-generated texts that put her behind bars.
“It was horrific,” she said.
Sims said she spent two days of hell in a Florida jail.
“It’s like you see in the movies ‘Orange is the New Black’,” she said. “I got put into like basically a general population.”
Her story made headlines in Florida.
Sims and her boyfriend had recently moved there from Delaware County, Pennsylvania.
She said her nightmare began in November 2024 after she called the police during an argument with her boyfriend, when she said he allegedly ransacked her home.
“Next thing I know, I’m looking at him and he’s slapping himself in the face,” she said.
She said he also allegedly scratched himself. When police arrived, they arrested her for battery.
As part of her bond, the judge ordered Sims to stay away from her boyfriend and not speak to him.
Fast forward several months, and she said her boyfriend created an AI-generated text that called him names and made disparaging comments.
“I end up getting arrested for violating my bond,” she said. “No one verified the evidence.”
Judge Herbert Dixon says Sims’ ordeal is one of increasing frequency.
“Several years ago, it started out with just fake audio recordings,” he said. “And now it’s gotten to a point of fake video and fake images being produced.”
Dixon is a senior judge on the Superior Court of the District of Columbia. He also serves as a member of the Council on Criminal Justice alongside former Philadelphia Police Commissioner Charles Ramsey. Its mission is to advance policy in the criminal justice system.
“One of the things we’re trying to do is to develop a framework for the responsible use of artificial intelligence,” he said.
He believes prosecutors and police need to do a bit more due diligence in this age of AI before they bring charges.
Drexel University professor Rob D’Ovidio teaches AI forensics.
“This is scary to say, but we’re no longer going to be able to trust what we see in front of us,” he said.
He said AI-generated video, texts and other evidence can be difficult to spot. AI is simply getting too good.
“The challenge is the detection tools are not keeping up with those capabilities,” he said.
As an example, he created an AI-generated photo to show us. He input it into three different well-known AI detection software programs. All three spit back different results that ranged from 1% to 62% probability of the photo being synthetic or AI-generated.
“The standard nowadays is we trust unless proven otherwise, right? I think we have to flip the script and distrust until we verify,” he said.
As for Sims, her story has a happy ending.
After eight months of legal wrangling by her attorney, prosecutors dropped the bond violation charge against her. And, last month, she went to trial on the battery charge and was acquitted.
Sims shared her story on her journey to advocate for a new law to create AI evidence standards and penalties.
“If this can happen to me, it can happen to anyone,” she added.
In July, Pennsylvania Governor Josh Shapiro signed a new digital forgery law that makes it a felony to create AI deepfakes that injure, exploit or scam in the state.
WPVI-TV reached out to Sims’ ex-boyfriend but has not heard back.
Please note: This story was provided to CNN Wire by an affiliate and does not contain original CNN reporting. This content carries a strict local market embargo. If you share the same market as the contributor of this article, you may not use it on any platform.