How Cybercriminals Are Weaponising AI Chatbots
WormGPT, FraudGPT and their successors remove the guardrails from AI to enable attacks at scale. Understand the threat and how defenders are responding.
Offensive AI Tools in the Criminal Ecosystem
The emergence of AI chatbots optimised for cybercriminal use represents a qualitative shift in the democratisation of advanced attack capabilities. Where traditional tools required technical skill to use effectively, AI chatbot interfaces reduce the expertise barrier to sophisticated attacks.
WormGPT gained public attention in mid-2023 as the first widely documented example of an LLM designed for malicious use — specifically phishing email generation and business email compromise. Based on an open-source LLM fine-tuned on malware data without safety guardrails, WormGPT provided criminal actors with a tool generating convincing, grammatically perfect phishing content without the refusals that safety-guardrailed models produce.
The Landscape of Offensive AI Tools
WormGPT established a template that spawned numerous successors. FraudGPT followed with fraud-specific applications — writing scam websites, generating fake invoices, creating fraudulent banking communications. A succession of branded variants have appeared on criminal marketplaces, with subscription pricing making sophisticated AI capabilities accessible to commodity cybercriminals.
Applications in Criminal Operations
Phishing and Social Engineering at Scale
The primary application is content generation for social engineering. AI-generated phishing content defeats the historical indicator of poor quality writing. Grammatically perfect, culturally appropriate, and contextually relevant phishing emails can be generated in any language in seconds, at unlimited volume.
A criminal operator who could previously send a few hundred manually crafted phishing emails per day can now send thousands of highly personalised variants, each tailored to the target's organisational context and role.
Vulnerability Research Assistance
Offensive AI tools assist in vulnerability research — explaining vulnerability classes, suggesting exploitation approaches for specific software versions, helping less experienced attackers understand and apply published exploit code.
Malware Development
AI assistance in malware development reduces development time and suggests evasion techniques. The expertise requirement for sophisticated offensive operations is declining.
Defensive Implications
Assume phishing quality will be high. Security awareness training focusing on grammatical quality as a phishing indicator is increasingly ineffective. Training must focus on context, urgency, requests for action, and procedure-following.
Invest in technical email authentication. DMARC, DKIM, and SPF don't detect high-quality phishing content but do block domain spoofing. Full DMARC enforcement eliminates spoofed domain-based attacks.
Implement phishing-resistant MFA. FIDO2 authentication defeats credential phishing regardless of phishing email quality, making stolen credentials useless to attackers who can't authenticate from the legitimate origin.