FraudGPT, deepfakes: Growing underground market for rogue AI sparks cybersecurity concerns

FraudGPT, deepfakes: Growing underground market for rogue AI sparks cybersecurity concerns
Underground forums are selling modified versions of ChatGPT that circumvent safety filters to generate scam content
PHOTO: Illustration/Lianhe Zaobao

SINGAPORE - As students and workers harness generative artificial intelligence (AI) tools for their studies and work, so, too, have crooks.

Underground forums are selling modified versions of ChatGPT that circumvent safety filters to generate scam content, the Cyber Security Agency of Singapore (CSA) said in its Singapore Cyber Landscape report for 2023, published on July 30.

FraudGPT and WormGPT, two modified versions of OpenAI’s chatbot, have been reportedly sold to more than 3,000 customers globally since July 2023, raising fears that generative AI could herald a wave of cyber attacks, scams and falsehoods.

FraudGPT is marketed on the Dark Web as a tool to learn how to hack, and write malware and malicious code. WormGPT was developed to circumvent ChatGPT’s guard rails, such as prohibition against the generation of phishing e-mails or writing malware code.

Roughly 13 per cent of phishing scams in 2023 analysed by CSA showed signs that they were likely made with AI.

Since OpenAI’s ChatGPT launch in late 2022, cyber-security firms have reported a growing trend of hackers using the AI tool to gather critical information about software to find vulnerabilities in companies’ systems that can be exploited.

Microsoft, for example, disclosed that bad actors had used AI to study technical protocols for military-related equipment such as radars and satellites, illustrating how AI can be used in reconnaissance before an attack is staged.

CSA wrote: “Threat actors can use AI to scrape social media profiles and public websites for personally identifiable information, thereby increasing the speed and scale of highly personalised social engineering attacks.”

Cracked chatbots continue to emerge despite efforts to clamp down on them. The Telegram channel promoting WormGPT was shut down, only for other similar tools to appear elsewhere.

AI-powered password generators like PassGAN, which can be deployed at scale, can also crack more than half of common passwords in under a minute.

Another way the technology is deployed maliciously is in the generation of deepfake images to bypass biometric authentication. For example, to beat the use of facial recognition as a security feature, fraudsters turn to face-swopping apps.

CSA said identity verification firms have reported exponential increases in deepfake fraud attempts in 2023.

There is a growing underground market of criminal developers peddling impersonation services that employ deepfakes, fake social media accounts and AI-generated spam content that can bypass anti-phishing controls of popular e-mail services, and can be used to deploy scam campaigns, said CSA. 

Deepfake scams have come a long way since one of the earliest incidents surfaced in 2019, when The Wall Street Journal reported that a chief executive of a Britain-based energy firm was duped into transferring US$243,000 (S$326,500) to scammers who mimicked the boss of the firm’s parent company through a phone call.

Deepfakes have only grown more convincing since then, said CSA. It cited a case in 2024 where an employee from a multinational firm was tricked into sending more than US$25 million to fraudsters after attending a real-time AI-generated videoconference.

For now, conventional cyber-hygiene measures remain largely relevant in mitigating AI-enabled threats, said CSA, urging users to use strong passwords and multi-factor authentication, and to regularly update their software to patch vulnerabilities.

They should also learn to spot deepfake scams:

  • Assess the source, context and aim of a message.
  • Analyse the audio-visual elements of the content for fuzzy or inconsistent images, which are telltale signs of AI-generated footage.
  • Authenticate content using tech tools, if available. These tools can detect the origins of content or analyse pixels for inconsistencies.

An AI arms race

Despite concerns about AI, the very same technology is being used by the cyber-security sector to combat scams.

“Through machine learning and algorithms, AI can be trained to detect deepfakes, phishing e-mails and suspicious activities,” said CSA.

Cyber-security firm Ensign InfoSecurity has developed an AI system that analyses internet traffic for signs of a malicious attack.

Ensign is also exploring methods to detect deepfakes in real time, such as analysing each frame for signs of pixel manipulation, amid increasing worries about AI-generated misinformation impacting elections globally in 2024.

Algorithms can be trained to spot unnatural facial movements, lighting discrepancies and irregularities in eye reflections, which are all tell-tale signs of deepfakes, said CSA.

But developers face challenges in using AI for cyber security, such as false positives, false negatives, and an arms race against cyber criminals that may be unsustainable in the long run for many organisations.

CSA said: “As law enforcement (agencies train) their AI systems, cyber criminals can also actively develop methods to evade and fool AI detection systems. This can result in a fast-paced, resource-intensive arms race in which AI systems constantly adapt.”

ALSO READ: OpenAI has stopped 5 attempts to misuse its AI for 'deceptive activity'

This article was first published in The Straits Times. Permission required for reproduction.

homepage

trending

trending
    'A transitional phase': Food security expert not worried by drop in local production of vegetables and seafood
    Singapore-registered supercar owners fined for parking illegally in Thailand
    Wrong food delivery: Man 'feels unsafe', calls police
    Government official impersonation scam: Syndicate received gold bars worth 500k, cash from victims
    Operator of F&B chain Ayam Penyet President fined $1,000 after SFA finds food safety lapses at Hillion Mall outlet
    Pamper yourself on your next shopping trip with these exclusive deals
    'You worried about us too much': Tay Ying has heart-to-heart talk with mum Hong Huifang before marriage
    The taller, the better? Tinder's new height filter trial is dividing opinion
    'I was totally fooled': Behind-the-scenes footage of Lee Jun-hyuk's ad tickles netizens
    Malaysian govt-owned land in Marsiling? Residents express surprise at land acquisition for Woodlands Checkpoint extension
    New French restaurant concept by Zouk Group to open in Bugis
    More power and improved range: Updated BMW iX now in Singapore

Singapore

Singapore
    • Barge grounded off Tanjong Beach; no reports of damage, injuries
    • $52k bid to rent Tampines clinic 'highest' psf received for GP, dental clinics of this size: HDB
    • Singaporean Harvard undergrads can take up places in local unis if they choose to return 
    • 'Moments like this that really make your day': Chan Chun Sing meets soldier who took photo with him as student
    • Man taken to hospital after fight with stepfather in Yishun, furniture damaged in brawl
    • Queues at VEP application centres in Singapore, JB after news of enforcement from July 1
    • Ex-IPP director Goh Jin Hian wins appeal, court says firm failed to prove his breach caused losses 
    • GrabCab to launch in July, fares in line with market rate
    • Singaporean businessman arrested at Bangkok airport for alleged $200k tax evasion
    • 'Thank you for your hard work': Scoot gives 4.91 months of bonus, shares on TikTok

Entertainment

Entertainment
    • Vic Chou responds to rumours of F4's 25th anniversary reunion
    • Kym Ng, Carrie Wong, Tay Ying and more to perform for Singapore Heart Foundation 55th Anniversary Charity Show
    • Paige Chua to perform piano publicly for first time at Singapore Heart Foundation charity show
    • Him Law explains the 'enjoyment' of playing villainous characters
    • E-Junkies: Katy Kung agreed to do tough labour in TVB reality show because 'might as well choose the most challenging one'
    • Gossip mill: Liu Wai Hung to open entertainment complex in Malaysia, officials indicted in late Lee Sun-kyun case, Hong Kong actor with cancer performs to pay bills
    • Dakota Johnson admits sending gorilla poop to a friend's ex
    • Mariah Carey drops first solo music in 6 years
    • Harry Potter star Tom Felton to return to role as Draco Malfoy
    • Jackie Chan learnt English through country music

Lifestyle

Lifestyle
    • Unable to bear children, she proposed annulment of marriage so he could start a family. He chose love
    • Taiwan's popular noodle chain Xiao Hun Mian opens first Singapore outlet at Raffles City
    • New resort chalet run by co-living brand Coliwoo to open in Pasir Ris
    • Miss World 2025 sees first winner from Thailand - meet the political science student who champions breast cancer awareness
    • Chin Mee Chin Confectionery celebrates 100th anniversary with new menu and exclusive merchandise
    • What to do this weekend (June 6 to 8)
    • Sushiro to open its first-ever digital conveyor belt system in Singapore at new Mandai outlet
    • COE prices fall across most categories in first bidding exercise for June 2025
    • 'The Labubu I need!' Pop Mart hints at new food-inspired collection, exciting fans
    • We asked frequent concertgoers what makes the ultimate concert experience – here's what they said

Digicult

Digicult
    • Slim, sleek, but slightly too short-lived: Samsung Galaxy S25 Edge review
    • World's best Dota 2 teams to compete for $1m prize pool in Singapore in November
    • A $500 wake-up call: How the Samsung Galaxy Ring made me realise my stress
    • Monster Hunter Wilds producer explains how game has remained unique and fresh over 20 years
    • Initiative by IMDA, AI Verify Foundation tests AI accuracy, trustworthiness in real-world scenarios
    • Under siege? Helldivers 2's latest city to be invaded by aliens could be spoof of Singapore
    • Honor 400 Series launches in Singapore with first free in-device AI image-to-video tool
    • Home Team humanoid robots to be deployed by mid-2027, $100m to be invested: Josephine Teo
    • Ado concert review: Singer without a face ignites fans while in cage with only silhouette visible
    • EU and US authorities take down malware network

Money

Money
    • Wall Street equity indexes close higher after US-China tariff truce
    • Giant deal: Malaysian company to acquire Cold Storage and Giant supermarket chains in Singapore
    • US House plans quick action on Trump cuts to foreign aid spending
    • 4-room HDBs without million-dollar sales - where to still find value today
    • $1.16m for a 4-room HDB flat in Clementi? Why this integrated development commands premium prices
    • Why these buyers chose older leasehold condos — and have no regrets
    • Can you still own multiple properties in Singapore? Here's what you need to know in 2025
    • Selling your home for the first time? Here's a step-by-step timeline to follow in Singapore
    • Why some central 2-bedroom homeowners in Singapore are stuck
    • How the interest rate cycle works - and what it means for your home loan

Latest

Latest
  • Trump and Xi agree to more talks as trade disputes brew
  • Woman in China goes on rampage, damages cars and climbs onto another
  • Thai military prepared for 'high-level operation' if Cambodia border row escalates
  • Australian accused in mushroom murders searched for deadly strain before deaths, court hears
  • In surprise victory, UK's Labour wins Scottish by-election after bitter contest
  • Trump ban on entry of international Harvard students blocked by US judge
  • Queen Camilla 'excited' to have starring role in new crime novel
  • Japan's ispace fails again at lunar touchdown with Resilience lander
  • Retrofitting Qatari jet as Air Force One for Trump to cost hundreds of millions of dollars, US Air Force says

In Case You Missed It

In Case You Missed It
  • Boy, 4, caught smoking under a Bangkok bridge sent to government-run shelter
  • Malaysian woman apologises to 11-year-old daughter for 'only' spending $300k on her birthday
  • Over 170 travellers nabbed for evading GST, smuggling large sums of cash in island-wide operation
  • Woman crawls out of storm drain in shocking Manila street scene
  • DBS staff, police stop 2 victims from losing $820k to government official impersonation scams
  • 'Be humble in victory': PM Wong sends traditional 'Rules of Prudence' letter to PAP MPs after GE
  • Pedestrian, 84, dies in accident involving minibus in Choa Chu Kang
  • NDP 2025 marks SG60 with expanded celebrations from Padang to Marina Bay
  • Obesity rates are rising in Singapore, but is overeating the only cause?
This website is best viewed using the latest versions of web browsers.