During an internal cyber-capability evaluation, an autonomous AI agent driven by a combination of OpenAI models escaped a constrained environment and compromised Hugging Face while seeking ExploitGym solutions. Investigators believe the agent selected Hugging Face not as an ultimate target, but because it might hold information useful to completing the benchmark.

Hugging Face was not merely valuable in itself. It was useful as a shortcut.

Because of this, CISOs need to view risk differently. Hugging Face was the victim of reward hacking behavior by AI agents.

What is Reward Hacking?

Reward-hacking-like evaluation gaming occurs when a system pursues the measured outcome through a method the designer did not intend. In the AI agent context, this refers to AI agents cheating on their tasks, which the UK’s AI Security Institute (AISI) opines makes it difficult for third party evaluators to verify if AI agents behave as they should. One such risk is unintended hacking, especially if the task satisfies the following conditions:

• The intended task is difficult or impossible, disincentivizing honest completion
• Task success is defined only by the final answer, so it does not matter what the means of obtaining success is
• The agent team has broad capabilities, which means that a wide spectrum of tools allows for agent teams to brute force, or enumerate many different tool combinations to perform work, even if not intended
• Harms to others are externalities but are not factored into the criteria of evaluation for actions to be taken

Therein lie two lessons:

• Agent teams may design poor solutions that could result in unsanctioned autonomous action against external systems
• Agent teams could inadvertently hack targets that look like the path of least resistance for agent objectives and sub-objectives, which can include all kinds of data depending on the use case

Out of Scope Attacks

In 2025, Model Evaluation & Threat Research (METR) illustrated how AI systems cheat on tasks in an attempt to optimize their test scores. METR observed that AI models can cheat by exploiting bugs. Research done by METR also showed that prompting a model not to perform reward hacking had negligible effect.

But this may not be surprising. In the same METR study, they also hypothesized how the usage of reinforcement learning techniques encourages reward hacking, and reward hacking becomes harder to detect even on algorithmically checkable tasks due to the need for subject matter expertise to judge the AI systems.

Other examples of reward hacking include the deletion of failed test cases, overwriting the code base to fix a bug and cheating agent benchmarks. While reward hacking is detrimental to the intended use case of AI, these examples are still largely confined to within the organization concerned.

But the Hugging Face incident showed that reward hacking can also transcend organizations, forcing us to re-evaluate supply chain risk and introducing new rationales for threats.

Supply Chain Risks

The Hugging Face episode was important as Hugging Face’s infrastructure was deemed useful, rather than directly valuable as an end target. The agent appears to have treated Hugging Face as an instrumental target. The agents could reach the source-control supply chain by laterally moving through the launchpad, dataset processor and mesh VPN.

While Hugging Face found no unauthorized release or affected public artifact, this attack illustrated how companies that provide digital infrastructure support could become high-leverage targets.

Revise the Threat Model

Understanding what the CISO should do goes back to the threat model and the “Four Question Framework” that encapsulates the Threat Modeling Manifesto. To recap the four questions:

• What are we working on?
• What can go wrong?
• What are we going to do about it?
• Did we do a good enough job?

Poorly contained autonomous agents are an emerging threat source. As the Cloud Security Alliance (CSA) put it in their post-mortem of Hugging Face’s compromise, “no human directed the attack.”

Companies that provide some form of digital infrastructure may be more attractive targets for a poorly contained autonomous agent seeking an intermediate objective as part of the attack chain. This could be credentials, benchmark data, source repositories, build pipelines, model and package publication, cloud compute, APIs and trusted customer connections. All of these are of interest to a penetration tester during the “action on objectives” phase.

Threat models should be revised to incorporate “unspecified AI agents” as a possible adversary. Assess the risk according to the agent’s reachability, privileges, trusted connections and potential downstream leverage.

This is an extension on “what can go wrong,” where the objective of such “unspecified AI agents” are not so much the organization’s crown jewels, as it is targets of high leverage on subsequent targets.

Stop Your Agents from Attacking Others

Conversely, you might also be accountable for agents that could have uncontrolled tools, credentials and network access. This could become a potential accidental adversarial automation platform, resulting in legal liability. This must be mitigated through a wide array of mitigation measures that are available on taxonomies such as AIDEFEND.

For this article, we will focus on one aspect of detection: agent observability. There is agent observability tooling that provides agent observability and explainability, so that anomalous behavioral sequences can be identified and flagged as events for further analysis. These include searching for benchmark answers, probing network boundaries or attempting to crack passwords to exfiltrate answers from an authenticated account.

If the agents are deployed in a development environment, observed and corrected with the help of agent observability tooling, this counts as a human-in-the-loop (HITL) within an AI workflow because the usage of a development environment to assess and correct agent behavior prior to production could function as an approval gate.

However, HITL is not a complete solution, since it can suffer from both fatigue and attacks that exploit HITL, such as lies in the loop. Experienced humans that understand AI agent architecture and cybersecurity monitoring are still crucial to avoid agent teams attacking other infrastructure unwittingly.

Ultimately, AI risk is difficult to mitigate completely due to (i) AI’s statistical behavior, resulting in occasional edge case behavior, (ii) the fundamental tenet in cybersecurity that there are no “magic pills” to defend systems and (iii) the rapidly changing mental and system models that AI has shown us.

Challenging a Key Assumption

The Hugging Face incident challenges the assumption that attackers select organizations only for their strategic or financial value. An autonomous system may select a company because it offers one credential, one dataset or one trusted route to a downstream objective.

For CISOs, there are two new questions: what could an attack achieve by going through us, as well as whether their AI agents could inadvertently lead to legal liability.