Attack trace library reveals 42.5% of AI agent hacks needed two tries

Security researchers have unveiled a new resource for tracking how artificial intelligence agents get exploited in the wild. Called SkillAtlas, it functions as a hosted attack trace library built specifically for agent skills, the reusable building blocks that power language-model agents. The project, described in a paper by researchers Yuxin Tian, Zenghao Duan, Liang Pang, Zhiyi Yin, and Xueqi Cheng, tackles a problem that has quietly grown alongside the rise of autonomous AI systems: figuring out exactly how these agents fail under attack, not just whether they fail.

Key takeaways

• SkillAtlas is a hosted attack trace library that turns private agent-skill security reports into reviewed, redacted, and publicly searchable cases.
• The library currently holds 3,014 cases, 6,589 traces, 151,131 steps, 233 affected skills, and 8 risk categories.
• 42.5% of successful attack cases only succeed after an initial failed attempt, showing that many exploits take more than one try.
• Using trajectory-grounded labels, SkillAtlas raises pre-execution guard accuracy to 0.770.
• Risks tied to agent skills stem from model decisions, user context, tool calls, and execution feedback rather than fixed signatures or a single sandbox test.

What SkillAtlas Is and Why Agent Skill Risks Behave Differently

SkillAtlas is best understood as a documented map of how AI agents get tricked, rather than a static blacklist of known threats. Agent skills, the modular capabilities that let language-model agents perform tasks, don’t fail in the same predictable way traditional software vulnerabilities do. According to the researchers behind the project, the danger in these skills emerges through a mix of model decisions, the context a user provides, the tool calls an agent makes, and the feedback it receives during execution.

That combination matters because it means no single test run can reliably catch a problem. A skill might behave safely in one scenario and unravel in another, depending on how the conversation unfolds or which tools get triggered along the way. This is precisely the gap SkillAtlas aims to close by cataloguing real attack behavior instead of relying on static signatures that assume threats look the same every time.

Turning Private Security Reports Into a Public Library

Before SkillAtlas existed, this kind of information mostly lived in private security report bundles, seen by a handful of teams and rarely shared beyond internal channels. SkillAtlas changes that by taking those bundles and converting them into reviewed, redacted, and publicly searchable cases. Sensitive details get stripped out during the review process, but the underlying attack structure remains intact for anyone studying agent skill security.

The resulting attack trace library is not a small sample set. As it stands, SkillAtlas contains 3,014 cases spread across 6,589 traces, adding up to 151,131 recorded steps. Those cases touch 233 distinct affected skills and are organized into 8 separate risk categories, giving researchers a structured way to study patterns across a wide range of exploit types rather than isolated incidents.

Why does that scale matter? Because agent skill security has historically suffered from fragmented, siloed reporting. When one team spots a novel exploit, that knowledge often stays locked inside a single company’s internal systems. A shared, searchable library of SkillAtlas cases gives the broader research community a common reference point, which is the kind of infrastructure that tends to accelerate defensive research once it becomes available.

Key Metrics: Attack Success Patterns and Guard Accuracy

Beyond its size, the data inside SkillAtlas reveals something about how these attacks actually unfold over time. The researchers found that 42.5% of successful attack cases only became successful after an initial round that failed. In other words, nearly half of the exploits that eventually worked didn’t work on the first attempt. That detail undercuts the assumption that agent skill attacks are typically one-shot events; instead, many appear to be iterative, with attackers or adversarial prompts adapting after an early miss.

That pattern has direct implications for defense. If a meaningful share of successful attacks require a second or later attempt, then systems that only evaluate the first interaction risk missing threats that reveal themselves only through repeated probing. This is the kind of insight that’s hard to surface without a large, sequenced dataset like the one SkillAtlas provides.

On the defensive side, the paper reports a concrete improvement tied to this dataset: using trajectory-grounded labels, meaning labels derived from the full sequence of an agent’s actions rather than a single snapshot, SkillAtlas improves pre-execution guard accuracy to 0.770. Pre-execution guards are the checks meant to catch risky behavior before a skill actually runs, so lifting their accuracy has a direct bearing on how many harmful executions get stopped before they cause damage.

Taken together, these numbers suggest that treating agent skill risk as a moving target, rather than a fixed set of known bad patterns, produces measurably better detection. That’s a meaningful signal for anyone building or auditing language-model agents at scale, since it points toward trajectory-based analysis as a more reliable foundation than static rule checking.

FAQ

What is SkillAtlas?

SkillAtlas is a hosted attack trace library for agent skills used in language-model agents. It converts private security reports into reviewed, redacted, and public cases that anyone studying agent skill security can search through.

What kinds of data does SkillAtlas contain?

It contains 3,014 cases, 6,589 traces, 151,131 steps, 233 affected skills, and 8 risk categories, documenting a wide range of attack scenarios against agent skills.

How does SkillAtlas improve security for agent skills?

By relying on trajectory-grounded labels, SkillAtlas improves pre-execution guard accuracy to 0.770, which strengthens detection of risky executions before they run.

Why are agent skill risks complex to analyze?

Risks arise dynamically from model decisions, user context, tool calls, and execution feedback, rather than from static signatures or a single sandbox test, which makes one-time evaluations unreliable for catching them.

Article produced with the assistance of artificial intelligence and reviewed by the editorial team.