AI is everywhere these days — writing emails, composing music, and even drafting bestselling novels. But what if I told you some sneaky folks are out there making AI do… well, not-so-cool things? In a new research exposé by Permiso Security, detailed in their research article, we venture into the unsettling territory of AI models gone rogue — hijacked to create unfiltered, explicit roleplaying services. And yes, this is real life, not a sci-fi movie.
Attackers have found a goldmine of opportunity in hijacking Generative AI (GenAI) infrastructures. Recently, AWS’s Bedrock — a managed service that provides the infrastructure to run large language models, making it easier for developers to deploy and scale AI applications — has been under the hacker microscope. Imagine your valuable cloud resources being exploited by unauthorized individuals to run explicit roleplaying chatbots, while you are left with the financial burden. Not so fun, right? Let’s explore how this all went down.
What Is LLMJacking Anyway?
LLMJacking, or Large Language Model Hijacking, is when cyber-criminals get unauthorized access to hosted AI models, manipulating them to serve their own shady purposes. It’s like your nosy neighbor reading your mail — but worse. Here, it’s not just your privacy at risk; it’s your compute power, data integrity, and even some dignity.
In the case outlined by Permiso, AWS Bedrock was targeted as a popular playground for hosting foundational AI models such as Anthropic’s Claude. Attackers weren’t just stealing compute power; they were repurposing it to create explicit and unfiltered content, combining illegal hacking activities with unethical storytelling. Specifically, they hosted sexual roleplaying bots, bypassed filtering, and even ran into darker territories like Child Sexual Exploitation Material (CSEM). Disturbing and alarming, indeed.
The Hijack Playbook: How They Did It
Step 1: Check for Model Availability
First things first, the attackers needed to know what’s available. It’s like casing a store before a robbery. They’d run an API command called InvokeModel and — guess what? — just see if they could use it. If the model returns as available, they’ve hit the jackpot. If it’s not available? Well, then it’s time to get even sneakier.
Attackers use another API called GetFoundationModelAvailability. Now, this is not your everyday API you’d find in the AWS reference guide. It’s more like a VIP backdoor used in AWS’ Web Management Console, but these guys figured out how to manipulate it.
Step 2: Request Model Access — Because Why Not?
The attackers are no strangers to paperwork. AWS has a mechanism that requires you to fill out a form explaining why you need model access. So, the attackers take a minute to pull out their imaginary bifocals, fill in some (obviously fake) credentials, and apply for access. They might say they’re building an educational chatbot for kids… you know, the wholesome stuff. Once the access is granted, it’s open season.
Step 3: Invoke the Models, Let the Chaos Begin
Two main methods for invoking models are observed here: InvokeModel and InvokeModelWithResponseStream. The difference? The first gives all the data in one response, while the latter serves it up piece by piece, streaming in real-time. For something as “dynamic” as a roleplay chatbot, streaming responses make more sense.
Anthropic’s models, specifically Claude-3-Sonnet, were the models of choice, making for quite the literary AI experience. And boy, did they make use of it: over 75,000 invocations in two days — almost all of it highly sexual in nature, with some straying into disturbing, explicit content. Not exactly the kind of traffic AWS had in mind.
The Dark Business of Roleplaying Bots
The real kicker here is that this isn’t just an experiment for hacker giggles. There’s actually a market out there — brace yourself — for explicit roleplaying bots. And these bots, apparently, aren’t cheap to operate. Enter the hijackers, who figured, why pay for cloud compute when you can just steal someone else’s and let them pay your hefty bill?
Chub[.]ai, an infamous roleplay bot service, became part of the investigation. Character names that attackers used were matched with this platform, and while this isn’t hard evidence, it’s a pretty suggestive coincidence. Dark roleplaying has a home, and these hijacked models were fueling it.
And let’s not forget the jailbreaks — techniques used to trick AI into bypassing its content filters. These aren’t just amateurs experimenting; there’s a community that openly shares and discusses these methods. If you want to dive into the murky world, Permiso has even linked a VirusTotal collection where some of these templates exist.
Attack Tactics: Keeping It Sneaky
Attackers don’t just roll the dice and hope they don’t get caught. These aren’t amateurs; they’re calculated. Permiso documented that many of these bot services run through reverse proxies, such as oai-reverse-proxy and one-api. Essentially, these reverse proxies help attackers anonymize their traffic and prevent direct detection. Even oai-reverse-proxy is configured to avoid triggering detection when invocation logging is turned on. It’s almost like these tools are built to do exactly this… because they are.
The most tell-tale sign of compromised access keys used in these attacks was activity on the GetFoundationModelAvailability API. As the AWS Bedrock API manual states, “long-lived access keys should not be interacting with this API.” It’s like seeing a pirate flag on a ship — it’s a dead giveaway.
Defensive Measures: What’s Being Done?
The Permiso team and AWS are both proactively trying to pull the plug on this. For instance, AWS has a new AWSCompromisedKeyQuarantineV2 policy in place, which now also blocks actions like bedrock:CreateModelInvocationJob and bedrock:InvokeModelWithResponseStream. But, as always, prevention is better than cure.
How Can You Protect Your Hosted Models?
- Tighten IAM Permissions: Limit who can invoke these models in the first place. AWS Identity and Access Management (IAM) allows for fine-grained permission management. Enforce IP-based restrictions where possible.
- Enable Invocation Logging: Remember
oai-reverse-proxy? It won’t even work if logging is enabled, making it one of your biggest allies. - Use Temporary Access Credentials: Long-lived access keys are hacker magnets. Use temporary credentials via IAM Roles whenever possible. Trust us, they’re like candy to hackers, and they love ’em.
- Monitor for Suspicious API Calls: Keep an eye on access keys making API calls that they shouldn’t, like
GetFoundationModelAvailability. Your keys should not be making random inquiries into Bedrock models.
FAQs
How are hosted models used for dark roleplaying?
Hackers exploit access to foundational models like Anthropic’s Claude by invoking them to generate unfiltered content. These hijacked models are employed to create explicit roleplaying bots, often bypassing ethical content filters using jailbreak techniques.
What kind of attacks have been seen on AWS Bedrock?
AWS Bedrock has been targeted with attacks leveraging compromised access keys. These keys are often obtained through exposed credentials on public repositories like GitHub, from poorly secured cloud environments, or even through phishing campaigns that trick users into revealing sensitive information. Attackers would invoke foundational models or use APIs not meant for programmatic access, like GetFoundationModelAvailability. This abuse has led to hijacked resources and compromised integrity.
How can organizations prevent LLMJacking?
To prevent LLMJacking, organizations should tighten IAM permissions, enable invocation logging, use temporary credentials, and actively monitor for suspicious API activity. Essentially, treat your AI infrastructure like you would a high-value server: lock it down, monitor it, and never leave the keys out in the open.
Conclusion: Keep Your AI on the Right Path
Hosted models are powerful and can create mind-blowing innovations. But when they end up in the wrong hands, things get ugly fast — and not just for the people footing the bill. As we enter the age of ubiquitous AI, security has to evolve with it. And if you’re managing a hosted model, take all the defensive steps you can.
Want to keep up with the latest in AI security? Subscribe to Guardians of Cyber, where we cover everything from the wackiest AI vulnerabilities to the most pressing cloud security updates. And don’t forget — your comments, thoughts, and even sarcastic quips are always welcome below!
