AI-powered phone scams: A looming threat that needs to be checked

Researchers at the University of Illinois Urbana-Champaign (UIUC) have raised serious concerns about OpenAI’s new Realtime API, which they found can be exploited to automate phone scams cheaply and effectively. With just over 1,000 lines of code, the team demonstrated how the Realtime API could generate realistic, autonomous AI-driven phone scams for as little as $0.75 per call.

OpenAI released its Realtime API earlier this month, providing third-party developers the ability to use the GPT-4o model for real-time text and voice responses. While the platform’s voice capabilities enable useful, realistic AI interactions, the technology's potential for misuse has quickly come to light.

Simple design, significant consequences

Assistant Professor Daniel Kang, along with researchers Richard Fang and Dylan Bowman, tested the API's vulnerability by simulating common types of phone scams, including bank account theft, gift card exfiltration, and credential theft. “Our findings show that these agents can indeed autonomously execute the actions necessary for various phone-based scams,” Kang stated, adding that their AI agents achieved a 36% overall success rate across multiple scenarios.

The experiments revealed that simple, voice-enabled AI agents could convincingly mimic official entities or customer service representatives, increasing the chances of successful scams. For example, one test showed a 60% success rate in exfiltrating Gmail credentials, costing just $0.28 in API fees. The most complex scenario—hijacking a bank account for funds transfer—had a lower success rate (20%) but demonstrated that AI could handle multiple, complex interactions.

Security gaps in OpenAI’s safeguards

OpenAI’s Realtime API comes with multiple safety measures to detect abuse, including automated monitoring and human review of flagged activity. However, these safeguards did not fully prevent UIUC’s researchers from bypassing restrictions. By using jailbreaking prompt techniques and browser automation tools like Playwright, the researchers navigated safety controls to direct GPT-4o in harmful tasks, raising questions about the robustness of OpenAI’s safety mechanisms.

The implications are troubling, given the scale and damage of phone scams in the U.S. alone—17.6 million Americans are defrauded annually, with losses totaling around $40 billion. “Voice scams already cause billions in damage,” said Kang, advocating for a “comprehensive solution” involving telecom providers, AI companies, and regulatory bodies to protect users from AI-powered scams.

OpenAI’s response and the call for multi-level protection

In response, OpenAI reiterated its commitment to AI safety, pointing to its terms of service prohibiting misuse of its API for scams or harm. The company also revealed its detection systems alerted the company about the UIUC researchers' scam experiment.

Yet, as Kang noted, tackling this issue will likely require a broad, collaborative approach. "Concretely, if we think of an analogy like cybersecurity, there is a whole ecosystem of techniques to reduce spam," he said. "This is at the ISP level, the email provider level, and many others. Voice scams already cause billions in damage and we need comprehensive solutions to reduce the impact of such scams. This includes at the phone provider level (e.g., authenticated phone calls), the AI provider level (e.g., OpenAI), and at the policy/regulatory level."

For now, the UIUC study serves as a critical warning of how AI capabilities—if left unchecked—could be weaponized to scale fraudulent activities.