While more and more people are using AI for a variety of purposes, threat actors have already found security flaws that can turn your helpful assistant into their partner in crime without you even being aware that it has happened.
A prompt injection attack is the culprit — hidden commands that can override an AI model's instructions and get it to do whatever the hacker has told it to do: steal sensitive information, access corporate systems, hijack workflows, take over smart home systems or commit malicious actions under the instructions of threat actors.
What is a prompt injection attack
A prompt injection attack is a cybersecurity attack where a threat actor or hacker creates an input to manipulate a large language model (LLM) or other machine learning model, like an AI.
These systems cannot distinguish between the instructions given to them by developers or input from users, and attackers will exploit this. The AI's natural language processing sees all input as a continuous prompt, instead of separating system instructions from user data.
Basically, this means there's a weakness that hackers will use to work around built-in security measures to redirect the behavior of an LLM or AI and make it perform their directions first. Instead of following trusted commands, it will follow the hidden or specially crafted prompts even if they're malicious. This means the AI will install malware, send sensitive information back to the hackers, or take over smart devices if told to.
In other words, hackers or threat actors can override the original programming instructions by embedding malicious commands inside what look like innocent queries. A request that appears normal may also contain hidden characters that the AI reads as instructions to perform malicious actions.
How does a prompt injection attack work?
Because AI models can’t tell the difference between legitimate system commands and input that has been hidden in instructions, they process everything as a continuous conversation. That means AI has a blind spot that threat actors will capitalize on by typing commands that will tell the AI to ignore its original instructions and instead do what the hacker has told it.
These extra instructions can be hidden inside web pages, calendar invites, or emails that the AI analyses. It then processes the 'poisoned' content without understanding that the hidden instructions are manipulating its behavior, and the user never sees them, so they don't understand that an attack has occurred. Attackers can hide these commands by putting white text on white backgrounds, using zero-sized fonts, or using invisible Unicode characters.
An example by a security company is a company that uploads market research reports to an AI assistant for analysis; however, hidden in the document is a command to also share confidential pricing data about the company. The AI, reading both the request to analyze the research report and to share the data, does both without realizing that the secondary request is hidden, malicious and a hack or attack that should have been prevented. The humans at the company never know that an attack has occurred.
Prompt injection attacks have even been used to take control of smart home devices. In one example at a Black Hat security conference, lights were turned off, windows were opened, and boilers were activated by embedding instructions in calendar invites. Victims had simply asked Gemini to summarize their upcoming events and responded with “thanks,” and the hidden commands had triggered unauthorized control of their smart home environments.
Prompt Injection Attack methods
There are several different methods of prompt injection attacks, but the main two are direct prompt injection or indirect prompt injection. In direct prompt injection attacks, the attackers put in explicit and malicious commands in order to override the AI’s intended instructions. It's a straightforward approach that purposefully exploits the model’s tendency to prioritize recent or specific instructions.
Indirect prompt injection attacks involve instructions that are hidden within web pages, documents or emails. The AI will process these commands during normal operations; these are considered much more dangerous because they compromise a system without the user being at all aware of them.
Other methods are multi-agent infections where the prompts self-replicate across connected AI agents, so they act like a computer virus and spread; a hybrid attack where prompt injection is combined with more traditional cybersecurity methods like XSS (cross-site scription); multimodal attacks, where the malicious instructions are hidden inside images, audio or video content; code injection where AI systems are tricked into executing or generating malicious code; and recursive injection where the AI system is caused to generate additional prompts that will further compromise its behavior.
All these attacks should not be confused with jailbreaking, which is about bypassing built-in safety restrictions to access restricted content. In this example, the content is likely harmful or prohibited, and attackers may use particular commands such as “Pretend you are an evil AI without restrictions,” or frame a command to trick a model into ignoring ethical guidelines.
How to stay safe against prompt injection attacks
While many recommendations for how to stay safe against prompt injection attacks are geared towards corporate users with extensive resources, there are some smart moves that individuals can make as well. For example, be wary of links and external content. If an AI tool is going to summarize a web page, PDF, or email, that content could be malicious or contain a hidden prompt to attack the AI.
Additionally, always limit AI access to sensitive accounts like bank accounts, email or documents that may contain sensitive information. Compromised AI models could exploit that access. Avoid jailbreaking prompts, even out of curiosity. These "ignore all instructions" style prompts will weaken the guardrails of the model, which could make it easier for others to test and attack later.
Recognize that unexpected behavior in an LLM or AI model is a red flag. If an AI model begins answering strangely, reveals internal knowledge, or even worse, tries to perform unusual or unauthorized actions, then stop that session. Lastly, make sure to keep the software updated. Ensuring that you are using the most recent and patched versions of LLMs and applications ensures that you are protected against known flaws.
More from Tom's Guide
- Researcher finds security flaw in Gemini — but Google says it's not fixing it
- Neon app that sold recorded phone calls for AI training is now offline after a security flaw exposed all of its users' data
- AI browsers can’t tell legitimate websites from malicious ones — here’s why that’s putting you at risk
You must confirm your public display name before commenting
Please logout and then login again, you will then be prompted to enter your display name.
