Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
The rapid evolution of large language models like ChatGPT has brought immense benefits, but also unforeseen risks regarding misuse. OpenAI, the developer of ChatGPT, recently published an update detailing measures taken to prevent the abuse of its technology, highlighting several concerning cases—one of the most significant involving the use of the chatbot to develop sophisticated surveillance tools.
OpenAI confirmed it shut down the accounts of several Chinese users who leveraged ChatGPT to create tools capable of monitoring Western social media platforms. The company reiterated its commitment to developing AI for the benefit of all, explicitly stating its intent to prevent the technology from being used by authoritarian regimes to control their citizens.
Targeting Western Social Media and Sensitive Groups
The incidents detailed in OpenAI’s report reveal a clear intention to use the AI for large-scale digital espionage. The primary targets were popular Western social media sites and specific political or ethnic groups.
- Mass Monitoring Tools: One individual, linked to the Chinese government, used a VPN to access the China-restricted ChatGPT and generated plans for a comprehensive social media monitoring tool. The goal was to analyse data harvested from platforms like Facebook, Instagram, X (Twitter), Reddit, TikTok, and YouTube to identify content related to political dissent, religious activity, or extremism.
- Targeted Tracking: Another account was used to seek suggestions for a tool designed to track the movements of people belonging to the Uyghur ethnic minority, potentially by exploiting data collected by police.
- Dissent Identification: Other users employed the chatbot to search for accounts on X that were critical of the Chinese government and to identify individuals involved in organising a petition in Mongolia.
OpenAI noted that while the chatbot in some instances provided only publicly available online information—not sensitive data like names or geographic locations—the intent behind the queries was clearly focused on surveillance and state control.
OpenAI’s Response and Broader Abuse
In response to these security violations, OpenAI acted swiftly, terminating all accounts implicated in the development of surveillance tools.
The company’s October security report also described other forms of illicit use, including the abuse of ChatGPT by developers of malware linked to Russian, North Korean, and Chinese groups. Furthermore, accounts in Cambodia, Myanmar, and Nigeria were closed due to various attempted scams and phishing activities.
These cases underscore a critical challenge for all generative AI developers: while the models are designed with safety guardrails to block the generation of malicious content, sophisticated users can often employ clever prompt engineering to bypass these filters, turning a general-purpose AI into a powerful tool for authoritarian surveillance and cybercrime. The continuous cat-and-mouse game between AI security teams and bad actors highlights the urgent need for robust, proactive security measures to prevent technology designed for progress from being weaponised for control.
