Crooks are hijacking AI avatars to use for cybercrime

BY KERRY TOMLINSON, AMPERE NEWS

You can hire an AI avatar to make an ad for you, or a training video, or to host a video on your website. But criminals can hire them, too.

Here's how some AI companies are fighting to keep their avatars from tricking you out of money.

Watch here:

NOAH & FRIENDS

Noah seems agreeable enough.

"I'm looking forward to helping you create videos," he says from his home on AI company Synthesia's website.

But he also served as a fake newscaster spreading propaganda in support of Venezuela's authoritarian regime last year.

"We wanted to find out if Venezuela is really as destroyed as the media has been saying over the last few years," he speaks into the camera, surrounded by news graphics. "So, how true is it that Venezuela is such a poor country?"

Noah's coworker, Daren, also hosted a Venezuelan influence campaign.

AI colleagues Alex and Jason did stints as propagandists for China.

"There have been more than 600 mass shootings in the United States this year," Jason said in a news-style report for Chinese interests.

An entire AI crew was enlisted to support the coup leader in the African country of Burkina Faso, once again with Alex and several others.

"Homeland or death, we shall overcome," some of them say.

What went wrong?

Synthesia told us how it happened.

"One of the things that we hadn't accounted for was people creating, in some cases, factual content that was then used to polarize communities or polarize a particular country," said Alexandru Voica, Synthesia's head of corporate affairs and policy, said in an interview with Ampere News.

The company said it already had measures in place to moderate content, but after these avatar hijackings last year, they tightened them even further.

For example, in the past, you could use the stable of stock avatars to make news-like reports, as long as they were factual. Now you can't use them for any news-like videos at all because of misuse.

Political content is out for stock avatars as well.

"The majority of our customers are Fortune 500 type companies," Voica said. "Those are the companies and the people we want to do business with. We're not interested in sort of monetizing our platform for harmful purposes."

Faces of crime?

Criminals snagged avatars for more than just propaganda. They restyled avatar Gary as a fake CEO of a cryptocurrency company, according to investigators with the state of California.

Other platforms have had problems as well.

A female video host from AI company D-ID showed up in YouTube videos to trick people into downloading malware, stealing passwords and money, researchers said.

She's also hosting a channel offering up fake bank statements.

"Can I make a fake bank statement to get a loan?" she asks in a video. "Get a fake bank statement PDF from us today.""

A male D-ID avatar also promotes shady deals like a get-rich-quick scheme in Portuguese and a fake money giveaway on Tiktok.

"Cash App is giving away $750 and you are just a click away from claiming it," he says.

BATTLING BACK

D-ID did not respond to our request for information about how they try to prevent criminals from using their avatars.

The D-ID site's ethics page said it prohibits dishonest and illegal content, conducts random audits of content as well, and has a watermark that customers are not allowed to remove without permission.

Synthesia provided in-depth information about its policies. The company said it added stricter moderation, tools designed to uncover criminal activity, and they added more people to moderate.

The tools look for more than just keywords in scripts, Voica said. He declined to explain how the tools work, saying he did not want to tip off criminals.

The Synthesia use guidelines lay out a detailed system of what videos you can make with which kinds of avatars.

The company also looks for videos on the web to see if they are later harnessed for unacceptable content.

If you break the rules, your account could get banned with no refunds of your money.

BATTLE READY?

Synthesia hired a team to test its new defenses, Voica said. They set out to make videos that cross the line, including gambling, bullying, misinformation, and conspiracy theories.

"Just because you catch them today doesn't mean they're going to stop tomorrow," Voica said.

The results, according to Synthesia? Out of many attempts, one problematic video got through. It was flagged by the automatic moderator, but the human moderator approved it in error, Voica explained.

"Doing content moderation, especially doing content moderation at scale, is quite complicated. It's not easy," he said.

Next steps

Synthesia has now tightened its moderation even further. It found evidence that some criminals on the dark web took notice, possibly moving on to other platforms where they can get their money back, according to Voica.

"If you've made a genuine mistake, we will obviously evaluate the case to make a decision," he said.

"But for people who are coming specifically with bad intentions and they're there to make harmful content, this financial deterrent of not refunding them the money if they break the rules, not refunding them the money if they get an account ban, it acts as a very strong motivating factor for them to avoid our platform," he added.

Work together

Voica wants other AI companies to join in, to keep the crooks and propagandists at bay and keep the faith in AI --- and AI avatars.

"We're never going to be able to do this alone," Voica said. "It takes the entire industry to to get better at preventing this type of content from being created."

Can you spot the fake?

How can you tell the difference between a real person and an AI avatar, especially in the hands of criminals?

There are some clues you can look for in videos, audio and still images. But the technology is getting better and better, so the most important step is to fall back on the basics: