AI Arms Race: State-Sponsored Hackers Weaponize Google's Gemini for Cyberattacks

Google has revealed that state-backed hacking groups are increasingly leveraging its Gemini AI model to enhance various stages of cyberattacks. These sophisticated actors are employing the technology for everything from initial reconnaissance and target profiling to crafting sophisticated phishing campaigns and even developing new malware. This trend highlights the growing integration of artificial intelligence into the cyber warfare landscape.

Key Takeaways

• State-sponsored groups from North Korea, China, Iran, and Russia are actively using Gemini AI.

• AI is being used for reconnaissance, target profiling, social engineering, code generation, and malware development.

• Hackers are employing techniques to bypass AI safety measures and extract model information.

• Google is continuously improving its AI safety systems to counter these evolving threats.

Reconnaissance And Target Profiling

Google Threat Intelligence Group (GTIG) observed the North Korea-linked threat actor UNC2970 using Gemini to synthesize open-source intelligence (OSINT) and profile high-value targets. This included searching for information on major cybersecurity and defense companies, as well as mapping specific technical job roles and salary data. This activity blurs the lines between professional research and malicious reconnaissance, enabling the creation of tailored phishing personas and the identification of vulnerable targets.

Social Engineering And Phishing

AI models like Gemini are being used to craft more convincing and culturally accurate phishing messages, removing common red flags such as poor grammar. Threat actors are also engaging in "rapport-building" phishing, conducting multi-step conversations to gain trust before deploying malware. Groups like APT42 from Iran have used generative AI to boost reconnaissance and targeted social engineering, searching for official email addresses and researching organizations to build believable pretexts.

Malware Development And Attack Automation

Google detected malware named HONESTCUE that leverages Gemini's API to outsource functionality generation for later stages of an attack. The malware sends prompts to Gemini and receives C# source code, which is then compiled and executed in memory, leaving minimal artifacts on disk. Additionally, AI-generated phishing kits, like COINBAIT, are being developed to masquerade as legitimate services for credential harvesting.

Model Extraction And Evasion Tactics

Threat actors are also attempting "model extraction" attacks, systematically querying AI models to replicate their behavior and build substitute models. In one instance, Gemini was targeted with over 100,000 prompts in an effort to extract its reasoning abilities. Hackers also reframe prompts, posing as security researchers or CTF participants, to trick AI systems into generating unexpected or malicious responses. Google is actively working to improve its safety mechanisms, including detection classifiers and mitigations, to combat these evolving misuse tactics.

The Evolving AI Threat Landscape

While AI has not yet led to dramatic leaps in offensive capabilities, it is becoming embedded in the daily workflows of cyber espionage groups, enhancing efficiency and automating routine tasks. Google emphasizes that defenders must also invest in AI-enabled capabilities to operate at machine speed and counter the increasing quality, quantity, and speed of AI-enabled attacks. The company is committed to strengthening safeguards and developing AI-enabled defensive capabilities.

Sources

• Google Reports State-Backed Hackers Using Gemini AI for Recon and Attack Support, The Hacker News.

• Google: state-backed hackers exploit Gemini AI for cyber recon and attacks, Security Affairs.

• Google: State-backed hackers using Gemini AI at every stage of attacks, Computing UK.

• Nation-State Hackers Embrace Gemini AI for Malicious Campaigns, Infosecurity Magazine.

• Google finds state-sponsored hackers use AI at 'all stages' of attack cycle, CyberScoop.